A question and answer method and system based on a large language model

Abstract

The application provides a question and answer method and system based on a large language model, which comprises the following steps: determining a current question of a user and a natural language understanding prompt word; inputting the current question and the natural language understanding prompt word into a pre-trained large language model to obtain a question understanding result and a question to be answered output by the large language model according to a natural language understanding strategy; in the case that the question understanding result is a factual question, retrieving an answer from a knowledge graph based on semantic analysis; inputting the retrieved answer and an answer verification prompt word into the pre-trained large language model to obtain an answer verification result; in the case that the verification is reasonable, taking the retrieved answer as the final answer to the current question; and in the case that the verification is unreasonable, generating an answer by the large language model. In the multi-round question and answer task, the application can deeply understand the question of the user based on the context, effectively retrieve and integrate information from different knowledge sources, and thus accurately and efficiently provide an answer to the current question.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a question-answering method and system based on a large language model. Background Technology

[0002] Intelligent question answering has become an important way for people to solve problems and quickly obtain relevant information. A question answering system is a program that can answer questions posed by users in natural language as quickly and accurately as possible. Early knowledge question answering tasks were generally set up as single-turn dialogues, with methods based on retrieval or semantic parsing. Retrieval-based knowledge question answering methods first convert the user's natural language question into an embedded form and perform similarity matching with the embedded forms in a knowledge graph to extract relevant information from the knowledge graph and finally form an answer. Semantic parsing-based methods parse the user's natural language question into a set of structured programs. Once these expressions or programs are executed, they can be directly queried on the knowledge base to finally provide an answer. However, existing question answering methods often lack the ability to understand context and cannot provide accurate answers based on contextual information.

[0003] With the development of Artificial Intelligence (AI) technology, Large Language Models (LLMs) are increasingly being used. LLMs are complex AI models capable of understanding and generating human language. Their defining characteristic is their massive scale, ranging from billions to tens of billions of parameters. LLMs are frequently applied in scenarios such as text summarization, question answering, and translation. For example, in question answering scenarios, LLMs can perceive and understand context through context modeling and semantic understanding, thereby providing more accurate and coherent answers. However, existing LLMs face the illusion problem and cannot provide accurate answers in knowledge-intensive, multi-turn question answering tasks. Summary of the Invention

[0004] This invention provides a question-answering method and system based on a large language model, addressing the shortcomings of existing technologies where large language models suffer from illusion problems and are unable to provide accurate answers in knowledge-intensive, multi-turn question-answering tasks. In multi-turn question-answering tasks, the invention can deeply understand the user's question based on context, effectively retrieve and integrate information from different knowledge sources, thereby providing accurate and efficient answers to the current question.

[0005] This invention provides a question-answering method based on a large language model, comprising: determining a user's current question and natural language understanding prompts; the natural language understanding prompts include dialogue history tracking information; inputting the current question and the natural language understanding prompts into a pre-trained large language model to obtain a question understanding result and an unanswered question output by the large language model according to a natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; if the question understanding result is a factual question, retrieving an answer from a knowledge graph based on semantic parsing; the semantic parsing is parsing the unanswered question into a program that can be operated on the knowledge graph according to a neural symbol knowledge retrieval strategy; inputting the retrieved answer and answer verification prompts into the pre-trained large language model to obtain an answer verification result output by the large language model; if the answer verification result is valid, using the retrieved answer as the final answer to the current question; if the answer verification result is invalid, using the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question.

[0006] According to the question-answering method based on a large language model provided by the present invention, the method further includes: displaying the generation process of the final answer to the current question in an interpretable chat interface; setting a program editing option in the interpretable chat interface; the program editing option is used to jump to the program editing interface; the program editing interface is used to modify the program that parses errors.

[0007] According to the question answering method based on a large language model provided by the present invention, the question attribute further includes a random question; after inputting the current question and the natural language understanding prompt words into the pre-trained large language model to obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: if the question understanding result is the random question, taking the answer to the question to be answered generated by the large language model as the final answer to the current question.

[0008] According to a question-answering method based on a large language model provided by the present invention, the question understanding result further includes completeness; the completeness includes question completeness and question incompleteness; if the question understanding result is that the question is complete, the current question is taken as the question to be answered; if the question understanding result is that the question is incomplete, the current question is rewritten based on the large language model according to the natural language understanding prompts, and the rewritten question is taken as the question to be answered.

[0009] According to a question-answering method based on a large language model provided by the present invention, the question understanding result further includes clarification; the clarification includes whether the question needs clarification or not; when the question understanding result indicates that the question does not need clarification, the current question is rewritten based on the large language model according to the natural language understanding prompts, and the rewritten question is used as the question to be answered; when the question understanding result indicates that the question needs clarification, a clarification response is given to the current question based on the large language model, and the clarified question is used as the question to be answered.

[0010] According to a question-answering method based on a large language model provided by the present invention, after inputting the current question and the natural language understanding prompts into a pre-trained large language model to obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: if the question understanding result is the factual question and the answer cannot be retrieved from the knowledge graph based on the semantic parsing, the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge is taken as the final answer to the current question; the dialogue history tracking information includes decision history tracking information, backend chat history information, and frontend chat history information.

[0011] According to a question-answering method based on a large language model provided by the present invention, the neural symbol knowledge retrieval strategy includes: inputting the question to be answered into a pre-trained semantic parsing model, and obtaining the output of the semantic parsing model that can perform reasoning and searching on the knowledge graph; wherein, the semantic parsing model is trained based on the KQA Pro dataset.

[0012] According to the question-answering method based on a large language model provided by the present invention, the program that can operate on the knowledge graph is a KoPL program; the step of retrieving answers from the knowledge graph based on semantic parsing includes: using the VisKoP engine to perform reasoning and searching on the knowledge graph using the KoPL program to obtain the retrieval answer of the KoPL program.

[0013] This invention also provides a question-answering system based on a large language model, comprising: a determination module, used to determine the user's current question and natural language understanding prompts; the natural language understanding prompts include dialogue history tracking information; a comprehension module, used to input the current question and the natural language understanding prompts into a pre-trained large language model, and obtain a question comprehension result and an unanswered question output by the large language model according to a natural language understanding strategy; the question comprehension result includes question attributes; the question attributes include factual questions; and a retrieval module, used to retrieve answers from a knowledge graph based on semantic parsing when the question comprehension result is the factual question; the comprehension module further comprises ... the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts include dialogue history tracking information; the natural language understanding prompts Semantic parsing is used to parse the question to be answered into a program that can be operated on a knowledge graph based on a neural symbol knowledge retrieval strategy; a verification module is used to input the search answer and answer verification prompts into the pre-trained large language model to obtain the answer verification result output by the large language model; a first result module is used to take the search answer as the final answer to the current question if the answer verification result is reasonable; a second result module is used to take the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question if the answer verification result is unreasonable.

[0014] According to the present invention, a question-answering system based on a large language model further includes: an interactive front-end module, used to display the generation process of the final answer to the current question in an interpretable chat interface; setting program editing options in the interpretable chat interface; the program editing options are used to jump to the program editing interface; the program editing interface is used to modify the program with parsing errors.

[0015] This invention provides a question-answering method and system based on a large language model. The method includes: determining the user's current question and natural language understanding prompts; inputting the current question and natural language understanding prompts into a pre-trained large language model to obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; if the question understanding result is a factual question, retrieving the answer from a knowledge graph based on semantic parsing; inputting the retrieved answer and answer verification prompts into the pre-trained large language model to obtain the answer verification result; if the verification is reasonable, using the retrieved answer as the final answer to the current question; if the verification is unreasonable, the large language model generates the answer. This invention can deeply understand the user's question in multi-turn question-answering tasks, effectively retrieve and integrate information from different knowledge sources, thereby providing an accurate and efficient answer to the current question. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a question-answering method based on a large language model provided by the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the principle of a question-answering method based on a large language model provided by the present invention.

[0019] Figure 3 This is a schematic diagram illustrating the principle of the chat interface provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the principle of the program editing interface provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the principle of the dialogue history tracker provided by the present invention.

[0022] Figure 6 This is a schematic diagram illustrating the principle of the dialogue strategy and knowledge source provided by this invention.

[0023] Figure 7 This is a schematic diagram of the structure of a question-answering system based on a large language model provided by the present invention.

[0024] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Knowledge-intensive question answering refers to the process where users input natural language questions into a question-answering system, and the system uses knowledge sources to retrieve relevant information and return accurate factual answers. In the era of internet information retrieval, users generally rely on search engines such as Baidu, Google, and Bing, solving knowledge-intensive questions by inputting keywords. In recent years, artificial intelligence has made tremendous progress, especially with the rise of language models with powerful language capabilities. A new interactive method has emerged: finding answers through multi-turn dialogues. This change has altered people's habits of acquiring knowledge. Common large language models, such as the GPT series, LLaMA series, GLM, Baichuan, and Bloom, are models capable of dialogue with humans, possessing strong semantic understanding and multi-turn dialogue capabilities. However, using only large language models still faces the illusion problem, often resulting in irrelevant answers and an inability to provide accurate answers in knowledge-intensive question answering tasks, especially when faced with complex multi-hop problems. Current research and work are mostly based on the assumption of single-turn dialogue. However, when users need to decompose complex problems into multiple simple sub-problems, the single-turn dialogue setting cannot meet the actual needs of users. In addition, in multi-turn dialogue scenarios, the questions asked by users often contain references or omissions, which poses an additional challenge for knowledge-based question-answering systems.

[0027] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a question-answering method based on a large language model provided by the present invention.

[0028] To address the technical problems existing in the prior art, this invention provides a question-answering method based on a large language model, comprising:

[0029] 101: Determine the user's current question and natural language understanding prompts; natural language understanding prompts include dialogue history tracking information;

[0030] 102: Input the current question and natural language understanding prompts into the pre-trained large language model, and obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions;

[0031] 103: When the question understanding result is a factual question, the answer is retrieved from the knowledge graph based on semantic parsing; semantic parsing is the process of parsing the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy;

[0032] 104: Input the search answer and answer verification hints into the pre-trained large language model, and obtain the answer verification results output by the large language model;

[0033] 105: If the answer verification result is deemed valid, the retrieved answer will be taken as the final answer to the current question;

[0034] 106: If the answer verification result is deemed unreasonable, the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge shall be taken as the final answer to the current question.

[0035] For multi-turn knowledge-based question answering backends, understanding the user's context is crucial in multi-turn dialogues. In this embodiment, the user's current question and natural language understanding (NLP) prompts are first determined. These NLP prompts include dialogue history tracking information from a dialogue history tracker, assisting the question rewriting operation in the large language model's dialogue decision-making. The large language model Llama 3-70B is used as the underlying model for the controller in the dialogue strategy. NLP prompts serve as contextual information prompts to guide the large language model in making corresponding decisions regarding the user's current question. The NLP strategy defines the decision-making process for understanding the user's question. The large language model determines whether the user's current question is factual or arbitrary based on the NLP strategy, outputting the question understanding result and the question to be answered. If the question understanding result is factual, to provide an accurate factual answer, a fusion neural network and a symbolic structured program (neural symbolic knowledge retrieval strategy) are used to retrieve relevant knowledge. The neural symbolic knowledge retrieval strategy checks if the knowledge graph has an answer. Knowledge graph retrieval produces two results: an answer or no answer. Semantic parsing technology uses a trained neural network to convert natural language into a set of structured programs that can operate on the knowledge graph. This technology, due to its failure to fully generalize natural language input, cannot guarantee the accuracy of search answers and carries the potential risk of incorrect responses. Even when an answer is available, semantic parsing does not always guarantee accurate parsing, leading to potentially inaccurate results. Therefore, a pre-trained large language model verifies the reasonableness of the retrieved answer based on answer verification prompts and outputs the verification result. If the verification result is reasonable, the retrieved answer is used as the final answer to the current question. If the verification result is unreasonable, the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question. This invention addresses the challenges of complexity and continuity in multi-turn knowledge-intensive question answering by proposing a dialogue history tracker and dialogue strategies (natural language understanding strategy and neural symbol knowledge retrieval strategy). These strategies effectively handle topic shifts and follow-up questions from users, accurately retrieve answers from knowledge sources, and ensure the system understands user intent and provides accurate responses.

[0036] It should be noted that a knowledge graph is a data model with a graph structure that organizes and manages real-world knowledge by modeling entities and their relationships.

[0037] The dialogue history tracking information is input into the large language model. The large language model can provide the same format for the next input or perform the same task as the dialogue history tracking information, according to the format or task of the dialogue history tracking information.

[0038] Natural language understanding prompts refer to sentences that provide instructions to a large language model. For example, to instruct the model to determine whether a user's question is a factual question, the prompt could be, "Given a question, your task is to determine whether the question is a factual question. Please choose between 'yes' and 'no'." Once the large language model receives the prompt, it can complete the task according to the instructions.

[0039] The knowledge of the parameters of a large language model refers to the knowledge of language and the world stored in its weights and biases after the large language model has been trained with a large amount of text data.

[0040] In dialogue tasks, a response refers to the answer or reply that the system gives in response to the user's input question.

[0041] To provide users with factual answers, the multi-turn conversational question-answering system of this invention combines three knowledge sources: knowledge graphs, the parametric knowledge of large language models, and dialogue history tracking information. Specifically, it uses Wikidata as the source of the knowledge graph and employs parametric knowledge from Llama 3-70B to provide accurate factual answers.

[0042] In summary, the question-answering method based on a large language model of the present invention can deeply understand the current question asked by the user and effectively retrieve and integrate information from different knowledge sources, thereby providing accurate and efficient factual answers.

[0043] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of a question-answering method based on a large language model provided by the present invention.

[0044] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of the chat interface provided by the present invention.

[0045] Please refer to Figure 4 , Figure 4 A schematic diagram illustrating the principle of the program editing interface provided by this invention.

[0046] As a preferred embodiment, the method further includes: displaying the generation process of the final answer to the current question in an interpretable chat interface; setting program editing options in the interpretable chat interface; the program editing options being used to jump to the program editing interface; and the program editing interface being used to modify programs that parse errors.

[0047] Large language models, being black boxes, suffer from poor interpretability, making it difficult for users to fully trust the answers they provide. In this embodiment, the dialogue-based interpretable interactive front-end includes a chat interface and a program editing interface.

[0048] The chat interface design of the question-and-answer system emphasizes explainability and interactivity. The front-end interface is equipped with interpretable components that demonstrate the decision-making process the system takes when answering questions, facilitating user understanding. Furthermore, a component is designed to ensure effective user interaction to correct parsing errors. To this end, several specific functions and design elements are integrated into the chat interface. First, a custom-designed collapsible decision-making process box displays the system's decision status and the knowledge sources used to generate answers, enhancing the system's explainability. Second, an "Edit Program" button (program editing option) is added, allowing users to switch to the program editing interface and edit the program by dragging and dropping operation boxes. These functions significantly improve the system's interactivity, enabling users to accurately correct errors and effectively retrieve information from the knowledge base.

[0049] The program editing interface allows users to modify programs that malfunction through intuitive drag-and-drop operations. Users can access this function simply by clicking the "Edit Program" button in the lower right corner of the chat interface.

[0050] This invention innovatively designs a user intervention mechanism to address potential errors during program parsing. When a user discovers an error in the semantic parsing result during a dialogue, this mechanism allows the user to intervene directly and correct the error in real time, thereby ensuring the accuracy of the question-and-answer system and the smoothness of user interaction.

[0051] Please refer to Figure 5 , Figure 5 A schematic diagram illustrating the principle of the dialogue history tracker provided by this invention.

[0052] Please refer to Figure 6 , Figure 6 A schematic diagram illustrating the principle of the dialogue strategy and knowledge source provided by this invention.

[0053] As a preferred embodiment, the question attribute also includes a random question; after inputting the current question and natural language understanding prompts into the pre-trained large language model and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: if the question understanding result is a random question, taking the answer to the question to be answered generated by the large language model as the final answer to the current question.

[0054] In this embodiment, when the problem understanding result is a random question, that is, when the user's current question is casual conversation or does not require retrieval of knowledge sources, the large language model can answer based on the model's own parameter knowledge (casual conversation response).

[0055] As a preferred embodiment, the question understanding result also includes completeness; completeness includes both complete and incomplete questions; if the question understanding result indicates that the question is complete, the current question is taken as a question to be answered; if the question understanding result indicates that the question is incomplete, the current question is rewritten based on the natural language understanding prompts and the large language model, and the rewritten question is taken as a question to be answered.

[0056] Given the limitations of single-turn question-answering systems in handling complex questions, and considering the challenges that follow-up questions (such as those involving ellipsis and pronouns) pose to existing semantic parsers in multi-turn dialogue question answering, this embodiment, after confirming that the current input question requires a factual answer, evaluates whether the current question is complete or incomplete. If the question is understood as complete, it is treated as an unanswered question, and the aforementioned knowledge graph retrieval steps and large-model answer verification steps are then performed. If the question is understood as incomplete, the large-language model needs to rewrite the current question, and the rewritten question is treated as an unanswered question, followed by the knowledge graph retrieval steps and large-model answer verification steps.

[0057] As a preferred embodiment, the problem understanding result also includes clarification; clarification includes whether the problem needs clarification or not; if the problem understanding result indicates that the problem does not need clarification, the current problem is rewritten based on the large language model according to the natural language understanding prompts, and the rewritten problem is used as the question to be answered; if the problem understanding result indicates that the problem needs clarification, the current problem is clarified based on the large language model, and the clarified problem is used as the question to be answered.

[0058] In this embodiment, for incomplete questions, the large language model checks the contextual information from the dialogue history tracking information to determine whether clarification is needed. If the current question does not require clarification, the large language model will continue to rewrite the question, using the rewritten question as the question to be answered, and then perform the knowledge graph retrieval steps and the large model answer verification steps described above. If the current question requires clarification (the user's input question is incomplete and unrelated to previous dialogue history), the large language model will interact with the user to clarify the current question, thus using the clarified response question as the question to be answered, and then perform the knowledge graph retrieval steps and the large model answer verification steps described above. This invention supports users in asking follow-up questions containing omissions and referential relationships, effectively improving the interactivity of the system and the depth of question answering.

[0059] As a preferred embodiment, after inputting the current question and natural language understanding prompts into the pre-trained large language model and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: if the question understanding result is a factual question and the answer cannot be retrieved from the knowledge graph based on semantic parsing, the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge is taken as the final answer to the current question; the dialogue history tracking information includes decision history tracking information, backend chat history information, and frontend chat history information.

[0060] In this embodiment, if the question is understood to be a factual question and an answer cannot be retrieved from the knowledge graph based on semantic parsing, the large language model will check if the answer can be inferred from the dialogue history tracking information (a factual response based on dialogue history). If no useful information can be retrieved from the first two knowledge sources, namely the knowledge graph and the dialogue history tracking information, the large language model will use its own parameter knowledge to answer the user question (a factual response based on the large language model's parameter knowledge). If no usable answer is found in the knowledge sources, the large language model will explicitly state that it cannot answer the user question (no answer response).

[0061] The dialogue history tracking information includes decision history tracking information, backend chat history information, and frontend chat history information.

[0062] Historical decision-making information is used to track past decision-making processes, which can also serve as contextual examples for large language models to enhance their decision-making capabilities.

[0063] Backend chat history tracking information is used to track short answers retrieved from the knowledge graph, avoiding the tracking of lengthy answers generated by large language models. This history can be used as contextual information for the dialogue decision module.

[0064] The front-end chat history tracking information is used to track the complete answers given by the question-and-answer system and to display the answers given by the question-and-answer system to the user.

[0065] As a preferred embodiment, the neural symbol knowledge retrieval strategy includes: inputting a question to be answered into a pre-trained semantic parsing model, and obtaining a program from which the semantic parsing model can perform reasoning and searching on a knowledge graph; wherein the semantic parsing model is trained based on the KQA Pro dataset.

[0066] As a preferred embodiment, the program that can operate on the knowledge graph is a KoPL program; retrieving answers from the knowledge graph based on semantic parsing includes: using the VisKoP engine to perform reasoning and searching on the knowledge graph using the KoPL program to obtain the retrieval answer from the KoPL program.

[0067] In this embodiment, a BART model is trained on the KQA Pro dataset as the semantic parsing model. The trained semantic parsing model can translate the question to be answered into a program called KoPL. Then, the VisKoP engine is used to perform reasoning and searching on the knowledge graph using the KoPL program to obtain the retrieval answer of the KoPL program.

[0068] It's important to note that KoPL is a programming language used to manipulate various knowledge elements (such as concepts, entities, relations, attributes, and modifiers) within the knowledge graph. It's a programming language specifically designed for complex reasoning and question answering, composed of multiple basic functions. Running these function combinations retrieves answers from the knowledge graph.

[0069] The KQA Pro dataset contains 90,000 'question-KoPL' pairs, and this invention does not impose any particular limitations on it.

[0070] The question-answering system based on a large language model provided by this invention will be described below. The question-answering system based on a large language model described below can be referred to in correspondence with the question-answering method based on a large language model described above.

[0071] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a question-answering system based on a large language model provided by the present invention.

[0072] This invention also provides a question-answering system based on a large language model, comprising: a determination module 1, used to determine the user's current question and natural language understanding prompts; the natural language understanding prompts include dialogue history tracking information; an understanding module 2, used to input the current question and natural language understanding prompts into a pre-trained large language model, and obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; a retrieval module 3, used to retrieve the answer from a knowledge graph based on semantic parsing when the question understanding result is a factual question; semantic parsing is parsing the question to be answered into a program that can be operated on the knowledge graph according to a neural symbol knowledge retrieval strategy; a verification module 4, used to input the retrieved answer and answer verification prompts into the pre-trained large language model, and obtain the answer verification result output by the large language model; a first result module 5, used to take the retrieved answer as the final answer to the current question when the answer verification result is valid; and a second result module 6, used to take the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question when the answer verification result is invalid.

[0073] As a preferred embodiment, it further includes: an interactive front-end module for displaying the generation process of the final answer to the current question in an interpretable chat interface; setting program editing options in the interpretable chat interface; the program editing options for jumping to the program editing interface; and the program editing interface for modifying programs that parse errors.

[0074] This invention develops a novel knowledge question-answering system based on multi-knowledge source fusion by demonstrating a knowledge-intensive question-answering application using multi-turn dialogue. The system aims to deeply understand user questions and effectively retrieve and integrate information from different knowledge sources, thereby providing accurate and efficient factual answers.

[0075] To verify the effectiveness of the invention, automated evaluation and user experiments were conducted.

[0076] For automated evaluation, tests were conducted on the ConvQuestions multi-turn dialogue dataset. This dataset contains five rounds of dialogue, with the first round consisting of a complete question, and the subsequent four rounds consisting of incomplete questions with references or omissions. Experiments were performed on 700 rounds of dialogues on this dataset, involving 3500 knowledge-based question-answering questions. Using the existing REWRITE method as a benchmark, experimental results show that the proposed method outperforms the benchmark by approximately 8%. Furthermore, ablation experiments were conducted to verify the effectiveness of the proposed dialogue strategy.

[0077] In user testing, this invention was compared with ChatGLM3, and 25 participants were recruited. Participants rated both systems based on multiple indicators, including answer accuracy, user preference, explainability, contextual understanding, interactivity of the program editing interface, and overall satisfaction. The system of this invention outperformed ChatGLM3 in all of the above indicators. This result not only proves the effectiveness of the system of this invention but also shows that it better meets the needs of actual users.

[0078] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logical instructions in the memory 830 to execute a question-answering method based on a large language model. This method includes: determining the user's current question and natural language understanding prompts; the natural language understanding prompts include dialogue history tracking information; inputting the current question and natural language understanding prompts into a pre-trained large language model to obtain the question understanding result and the question to be answered output by the large language model according to a natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; if the question understanding result is a factual question, retrieving the answer from a knowledge graph based on semantic parsing; semantic parsing is parsing the question to be answered into a program that can be operated on the knowledge graph according to a neural symbol knowledge retrieval strategy; inputting the retrieved answer and answer verification prompts into the pre-trained large language model to obtain the answer verification result output by the large language model; if the answer verification result is valid, using the retrieved answer as the final answer to the current question; if the answer verification result is invalid, using the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question.

[0079] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the question-answering method based on a large language model provided by the above methods. The method includes: determining the user's current question and natural language understanding prompts; the natural language understanding prompts include dialogue history tracking information; inputting the current question and natural language understanding prompts into a pre-trained large language model to obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; if the question understanding result is a factual question, retrieving the answer from a knowledge graph based on semantic parsing; the semantic parsing is a program that parses the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy; inputting the retrieved answer and answer verification prompts into the pre-trained large language model to obtain the answer verification result output by the large language model; if the answer verification result is valid, taking the retrieved answer as the final answer to the current question; if the answer verification result is invalid, taking the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question.

[0081] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the question-answering method based on a large language model provided by the above methods. This method includes: determining the user's current question and natural language understanding prompts; the natural language understanding prompts include dialogue history tracking information; inputting the current question and natural language understanding prompts into a pre-trained large language model to obtain a question understanding result and a question to be answered output by the large language model according to a natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; if the question understanding result is a factual question, retrieving an answer from a knowledge graph based on semantic parsing; semantic parsing is a program that parses the question to be answered into a program operable on the knowledge graph according to a neural symbol knowledge retrieval strategy; inputting the retrieved answer and answer verification prompts into the pre-trained large language model to obtain an answer verification result output by the large language model; if the answer verification result is valid, using the retrieved answer as the final answer to the current question; if the answer verification result is invalid, using the answer generated by the large language model based on dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-answering method based on a large language model, characterized in that, include: Determine the user's current question and natural language understanding prompts; The natural language understanding prompts include dialogue history tracking information; Input the current question and the natural language understanding prompts into a pre-trained large language model to obtain the question understanding result and the unanswered question output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes, completeness, and clarification; the question attributes include factual questions and arbitrary questions; the completeness includes complete questions and incomplete questions; The clarification includes questions that require clarification and questions that do not require clarification. If the problem understanding result is the factual problem, the answer is retrieved from the knowledge graph based on semantic parsing; the semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy; the program that can be operated on the knowledge graph is a KoPL program; Input the search answer and answer verification prompts into the pre-trained large language model, and obtain the answer verification result output by the large language model; If the answer verification result is deemed valid, the retrieved answer will be taken as the final answer to the current question. If the answer verification result is deemed unreasonable, the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge shall be taken as the final answer to the current question. The neural symbol knowledge retrieval strategy includes: Input the question to be answered into a pre-trained semantic parsing model, and obtain the output of the semantic parsing model, which can perform reasoning and searching on the knowledge graph; The semantic parsing model is trained based on the KQA Pro dataset; Also includes: The process of generating the final answer to the current question is displayed in an interpretable chat interface; the chat interface is customized with a decision flow box to show the decision status of the question-and-answer system and the knowledge sources used to generate the answer; The interpretability of the chat interface includes a program editing option; this option allows navigation to the program editing interface; the program editing interface is used to modify programs that malfunction during parsing.

2. The question-answering method based on a large language model according to claim 1, characterized in that, After inputting the current question and the natural language understanding prompts into the pre-trained large language model, and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: If the problem understanding result is the arbitrary problem, the answer to the unanswered question generated by the large language model shall be taken as the final answer to the current problem.

3. The question-answering method based on a large language model according to claim 1, characterized in that, If the understanding of the problem is complete, the current problem will be taken as the question to be answered. If the problem understanding result indicates that the problem is incomplete, the current problem is rewritten based on the natural language understanding prompts and the large language model, and the rewritten problem is used as the unanswered problem.

4. The question-answering method based on a large language model according to claim 1, characterized in that, If the problem understanding result indicates that the problem does not require clarification, the current problem is rewritten based on the large language model according to the natural language understanding prompts, and the rewritten problem is used as the question to be answered. If the understanding result of the problem indicates that the problem needs clarification, a clarification response is given to the current problem based on the large language model, and the clarified problem is taken as the question to be answered.

5. The question-answering method based on a large language model according to claim 1, characterized in that, After inputting the current question and the natural language understanding prompts into the pre-trained large language model, and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: If the question is understood to be a factual question and the answer cannot be retrieved from the knowledge graph based on the semantic parsing, the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge shall be taken as the final answer to the current question; the dialogue history tracking information includes decision history tracking information, backend chat history information and frontend chat history information.

6. The question-answering method based on a large language model according to claim 1, characterized in that, The method of retrieving answers from a knowledge graph based on semantic parsing includes: The VisKoP engine is used to perform reasoning and searching on the knowledge graph using the KoPL program to obtain the retrieval answer from the KoPL program.

7. A question-answering system based on a large language model, characterized in that, include: The determination module is used to determine the user's current question and natural language understanding prompts; The natural language understanding prompts include dialogue history tracking information; The understanding module is used to input the current question and the natural language understanding prompts into a pre-trained large language model, and obtain the question understanding result and the unanswered question output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes, completeness, and clarification; the question attributes include factual questions and arbitrary questions; the completeness includes complete questions and incomplete questions; The clarification includes questions that require clarification and questions that do not require clarification. The retrieval module is used to retrieve the answer from the knowledge graph based on semantic parsing when the question understanding result is the factual question; the semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy; the program that can be operated on the knowledge graph is a KoPL program; The verification module is used to input the search answer and answer verification prompts into the pre-trained large language model and obtain the answer verification result output by the large language model. The first result module is used to take the retrieved answer as the final answer to the current question if the answer verification result is deemed reasonable. The second result module is used to take the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question when the answer verification result is deemed unreasonable. The neural symbol knowledge retrieval strategy includes: Input the question to be answered into a pre-trained semantic parsing model, and obtain the output of the semantic parsing model, which can perform reasoning and searching on the knowledge graph; The semantic parsing model is trained based on the KQA Pro dataset; Also includes: An interactive front-end module is used to display the generation process of the final answer to the current question in an interpretable chat interface; the chat interface has a customized decision flow box that displays the decision status of the question-and-answer system and the knowledge source used to generate the answer; the interpretable chat interface has program editing options; the program editing options are used to jump to the program editing interface; the program editing interface is used to modify programs that parse errors.

Images