Question and answer method and system based on large language model

Abstract

The invention provides a question and answer method and system based on a large language model. The method comprises the steps of determining a current question and a natural language understanding cue word of a user; inputting the current question and the natural language understanding cue word to a pre-trained large language model, and obtaining a question understanding result and a to-be-answered question output by the large language model according to the natural language understanding strategy; if the question understanding result is a factual question, retrieving an answer from the knowledge graph based on semantic analysis; inputting the retrieval answer and the answer verification cue word to a pre-trained large language model to obtain an answer verification result; under the condition that the verification is reasonable, taking the retrieval answer as the final answer of the current question; and under the condition that the verification is unreasonable, generating an answer by the large language model. The question of the user can be deeply understood based on the context in multiple rounds of question and answer tasks, information of unused knowledge sources is effectively retrieved and integrated, and therefore the answer to the current question is accurately and efficiently provided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a question-answering method and system based on a large language model. Background Art

[0002] Intelligent question-answering has now become an important way for people to solve problems and quickly obtain relevant information. A question-answering system is a program that can answer questions asked by users in natural language as quickly and accurately as possible. Early knowledge question-answering tasks were generally set as single-round dialogues, with methods based on retrieval or semantic parsing. The retrieval-based knowledge question-answering method first converts the user's natural language question into an embedded form, and performs similarity matching with the embedding in the knowledge graph to extract relevant information from the knowledge graph and finally form an answer. The semantic parsing-based method parses the user's natural language question into a set of structured programs. Once these expressions or programs are executed, they can be directly queried on the knowledge base and finally give answers. However, existing question-answering methods often lack the ability to understand the context and cannot give accurate answers based on contextual information.

[0003] With the development of artificial intelligence (AI) technology, large language models (LLM) have gradually been widely used. Large language models are relatively complex AI models that can understand and generate human language. Large language models are characterized by huge scale, ranging from billions of parameters to tens of billions of parameters. Large language models are often used in scenarios such as text summarization, question answering, and translation. For example, in question-answering scenarios, large language models can perceive and understand the context through modeling and semantic understanding of the context, thereby providing more accurate and coherent answers. However, existing large language models face the problem of hallucinations and are unable to give accurate answers in knowledge-intensive multi-round question-answering tasks. Summary of the invention

[0004] The present invention provides a question-answering method and system based on a large language model, which is used to solve the problem of hallucination faced by large language models in the prior art, and the defect that accurate answers cannot be given in knowledge-intensive multi-round question-answering tasks. In multi-round question-answering tasks, the invention can deeply understand the user's questions based on the context, effectively retrieve and integrate information from different knowledge sources, and thus accurately and efficiently provide answers to current questions.

[0005] The present invention provides a question-answering method based on a large language model, comprising: determining a current question and a natural language understanding prompt word of a user; the natural language understanding prompt word comprises dialogue history tracking information; inputting the current question and the natural language understanding prompt word into a pre-trained large language model, obtaining a question understanding result and a question to be answered outputted by the large language model according to a natural language understanding strategy; the question understanding result comprises a question attribute; the question attribute comprises a factual question; in the case where the question understanding result is the factual question, retrieving an answer from a knowledge graph based on semantic parsing; the semantic parsing is parsing the question to be answered into a program operable on a knowledge graph according to a neural symbol knowledge retrieval strategy; inputting the retrieved answer and the answer verification prompt word into the pre-trained large language model, obtaining an answer verification result outputted by the large language model; in the case where the answer verification result is that the verification is reasonable, taking the retrieved answer as the final answer to the current question; in the case where the answer verification result is that the verification is unreasonable, taking the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question.

[0006] A question-answering method based on a large language model provided by the present invention also includes: displaying the generation process of the final answer to the current question in an interpretable chat interface; setting a program editing option in the interpretable chat interface; the program editing option is used to jump to the program editing interface; the program editing interface is used to modify the program with parsing errors.

[0007] According to a question-answering method based on a large language model provided by the present invention, the question attributes also include arbitrary questions; after the current question and the natural language understanding prompt words are input into the pre-trained large language model, and the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy are obtained, it also includes: when the question understanding result is the arbitrary question, the answer to the question to be answered generated by the large language model is used as the final answer to the current question.

[0008] According to a question-answering method based on a large language model provided by the present invention, the question understanding result also includes completeness; the completeness includes whether the question is complete or incomplete; when the question understanding result is that the question is complete, the current question is used as the question to be answered; when the question understanding result is that the question is incomplete, the current question is rewritten based on the large language model according to the natural language understanding prompt word, and the rewritten question is used as the question to be answered.

[0009] According to a question-answering method based on a large language model provided by the present invention, the question understanding result also includes clarification; the clarification includes whether the question needs clarification or not; when the question understanding result is that the question does not need clarification, the current question is rewritten based on the large language model according to the natural language understanding prompt word, and the rewritten question is used as the question to be answered; when the question understanding result is that the question needs clarification, a clarification response is given to the current question based on the large language model, and the clarified question is used as the question to be answered.

[0010] According to a question-answering method based on a large language model provided by the present invention, after inputting the current question and the natural language understanding prompt word into a pre-trained large language model and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, it also includes: when the question understanding result is the factual question and the answer cannot be retrieved from the knowledge graph based on the semantic parsing, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question; the dialogue history tracking information includes decision history tracking information, back-end chat history information and front-end chat history information.

[0011] According to a question-answering method based on a large language model provided by the present invention, the neural symbolic knowledge retrieval strategy includes: inputting the question to be answered into a pre-trained semantic parsing model, obtaining a program output by the semantic parsing model that can be used for reasoning and searching on the knowledge graph; wherein the semantic parsing model is trained based on the KQA Pro dataset.

[0012] According to a question-answering method based on a large language model provided by the present invention, the program that can be operated on the knowledge graph is a KoPL program; the retrieval of answers from the knowledge graph based on semantic parsing includes: using the KoPL program to perform reasoning and search on the knowledge graph through the VisKoP engine to obtain the retrieval answer of the KoPL program.

[0013] The present invention also provides a question-answering system based on a large language model, comprising: a determination module, used to determine a user's current question and a natural language understanding prompt word; the natural language understanding prompt word includes dialogue history tracking information; an understanding module, used to input the current question and the natural language understanding prompt word into a pre-trained large language model, and obtain a question understanding result and a question to be answered output by the large language model according to a natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; a retrieval module, used to retrieve an answer from a knowledge graph based on semantic parsing when the question understanding result is the factual question; the language The semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbolic knowledge retrieval strategy; the verification module is used to input the retrieved answer and the answer verification prompt word into the pre-trained large language model to obtain the answer verification result output by the large language model; the first result module is used to use the retrieved answer as the final answer to the current question when the answer verification result is reasonable; the second result module is used to use the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question when the answer verification result is unreasonable.

[0014] A question-answering system based on a large language model provided according to the present invention also includes: an interactive front-end module, which is used to display the generation process of the final answer to the current question in an interpretable chat interface; setting a program editing option in the interpretable chat interface; the program editing option is used to jump to the program editing interface; the program editing interface is used to modify the program with parsing errors.

[0015] The present invention provides a question-answering method and system based on a large language model, the method comprising: determining a user's current question and a natural language understanding prompt word; inputting the current question and the natural language understanding prompt word into a pre-trained large language model, obtaining a question understanding result and a question to be answered output by the large language model according to a natural language understanding strategy; in the case where the question understanding result is a factual question, retrieving an answer from a knowledge graph based on semantic parsing; inputting a retrieved answer and an answer verification prompt word into a pre-trained large language model, obtaining an answer verification result; in the case where the verification is reasonable, using the retrieved answer as the final answer to the current question; in the case where the verification is unreasonable, the large language model generates an answer. The present invention can deeply understand the user's question in a multi-round question-answering task, effectively retrieve and integrate information from different knowledge sources, and thus accurately and efficiently provide an answer to the current question. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a question-answering method based on a large language model provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the principle of a question-answering method based on a large language model provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the principle of the chat interface provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the principle of the program editing interface provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the principle of the conversation history tracker provided by the present invention.

[0022] Figure 6 It is a schematic diagram of the principles of the dialogue strategy and knowledge source provided by the present invention.

[0023] Figure 7 It is a structural diagram of a question-answering system based on a large language model provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Knowledge-intensive question answering refers to the process in which users input natural language questions into a question answering system, and the question answering system uses knowledge sources to retrieve relevant information and return accurate factual answers. In the era of Internet information retrieval, users generally rely on search engines such as Baidu, Google, and Bing to solve knowledge-intensive questions by entering keywords. In recent years, the development of artificial intelligence has made great progress, especially the rise of language models with powerful language capabilities. A new way of interaction has emerged: finding answers through multi-round dialogues. This change has changed people's habits of acquiring knowledge. Common large language models such as the GPT series, LLaMA series, GLM, Baichuan, Bloom, etc., which can have dialogues with people, have strong semantic understanding and multi-round dialogue capabilities. However. Using only large language models also faces the problem of hallucinations, and often answers questions that are not asked. It is impossible to give accurate answers in knowledge-intensive question answering tasks, especially when facing complex problems with multiple hops. Most of the current research and work is based on the assumption of single-round dialogue. However, when users need to decompose complex problems into multiple simple sub-problems, the setting of single-round dialogue cannot meet the actual needs of users. In addition, in multi-round dialogue scenarios, questions asked by users often contain references or omissions, which is an additional challenge for knowledge question answering systems.

[0027] Please refer to Figure 1 , Figure 1 A flowchart of a question-answering method based on a large language model provided by the present invention.

[0028] In order to solve the technical problems existing in the prior art, the present invention provides a question answering method based on a large language model, comprising: 101: Determine the user's current question and natural language understanding prompt words; the natural language understanding prompt words include conversation history tracking information; 102: Input the current question and the natural language understanding prompt words into the pre-trained large language model, and obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; and the question attributes include factual questions; 103: When the result of question understanding is a factual question, the answer is retrieved from the knowledge graph based on semantic parsing; semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbolic knowledge retrieval strategy; 104: inputting the search answer and the answer verification prompt word into the pre-trained large language model, and obtaining the answer verification result output by the large language model; 105: When the answer verification result is reasonable, the retrieved answer is used as the final answer to the current question; 106: When the answer verification result is unreasonable, the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question.

[0029] For the multi-round knowledge question answering backend, it is crucial to understand the user context in a multi-round dialogue. In this embodiment, the user's current question and natural language understanding prompt words are first determined, where the natural language understanding prompt words include the dialogue history tracking information tracked by the dialogue history tracker to assist the question rewriting operation in the dialogue decision of the large language model. The large language model Llama 3-70B is used as the underlying model of the controller in the dialogue strategy, and the natural language understanding prompt words are used as the prompt words of the context information to prompt the large language model to take corresponding decisions on the user's current question. The natural language understanding strategy defines the decision-making process for understanding user questions. The large language model determines whether the user's current question is a factual question or an arbitrary question according to the natural language understanding strategy, and outputs the question understanding result and the question to be answered. In the case where the question understanding result is a factual question, in order to give an accurate factual answer, a fusion neural network and a symbolic structured program (neural symbolic knowledge retrieval strategy) are used to retrieve relevant knowledge. The neural symbolic knowledge retrieval strategy checks whether the knowledge graph has an answer. Knowledge graph retrieval answers will produce two results: with an answer and without an answer. Semantic parsing technology converts natural language into a set of structured programs that can operate on the knowledge graph through a trained neural network. Since this technology fails to fully generalize natural language input, it cannot ensure the accuracy of search answers and there is a potential risk of wrong answers. In the case of an answer, the retrieved answer may be inaccurate because semantic parsing does not always guarantee accurate parsing. Therefore, the pre-trained large language model verifies whether the retrieved answer is reasonable based on the answer verification prompt word, and outputs the answer verification result. If the answer verification result is that the verification is reasonable, the retrieved answer is used as the final answer to the current question. If the answer verification result is that the verification is unreasonable, the answer generated by the large language model based on the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question. In the face of the complexity and continuity challenges of user dialogues in multi-round knowledge-intensive question-answering, the present invention proposes a dialogue history tracker and a dialogue strategy (natural language understanding strategy and neural symbol knowledge retrieval strategy), which can effectively handle topic transitions in dialogues and follow-up questions raised by users, and accurately obtain answers from knowledge sources, ensuring that the system can understand user intentions and accurately respond.

[0030] It should be noted that the knowledge graph is a graph-structured data model that organizes and manages real-world knowledge by modeling entities and their relationships.

[0031] The dialogue history tracking information is input into the large language model, and the large language model can give the same format or perform the same task as the dialogue history tracking information for the next input according to the format or task of the dialogue history tracking information.

[0032] Natural language understanding prompts refer to sentences that give instructions to the large language model. For example, if you want the model to determine whether a user's question is a factual question, the prompt can be "Given a question, your task is to determine whether the question is a factual question. Please choose from two options: yes or no." The large language model can complete the task according to the instruction after receiving the prompt.

[0033] The parameter knowledge of the large language model itself refers to the language and world knowledge stored in the weights and biases of the large language model after it is trained with a large amount of text data.

[0034] In a dialogue task, a response refers to the answer or reply given by the system to the user's input question.

[0035] In order to provide users with factual answers, the multi-round dialogue question-answering system of the present invention combines three knowledge sources: knowledge graph, large language model parameter knowledge and dialogue history tracking information. Specifically, Wikidata is used as the source of the knowledge graph, and the parameterized knowledge of Llama 3-70B is used to provide accurate factual answers.

[0036] In summary, the question-answering method based on a large language model of the present invention can deeply understand the current question asked by the user, and can effectively retrieve and integrate information from different knowledge sources, so as to provide factual answers accurately and efficiently.

[0037] Please refer to Figure 2 , Figure 2 A schematic diagram of the principles of a question-answering method based on a large language model provided by the present invention.

[0038] Please refer to Figure 3 , Figure 3 A schematic diagram of the principles of the chat interface provided by the present invention.

[0039] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the principles of the program editing interface provided by the present invention.

[0040] As a preferred embodiment, it also includes: displaying the generation process of the final answer to the current question in an explainable chat interface; setting a program editing option in the explainable chat interface; the program editing option is used to jump to the program editing interface; and the program editing interface is used to modify the program with parsing errors.

[0041] As a black box, the large language model has the disadvantage of poor interpretability, which makes it impossible for users to fully trust the answers given by the large language model. In this embodiment, the dialogue-based interpretable interaction front end includes a chat interface and a program editing interface.

[0042] Among them, the chat interface design of the question-answering system focuses on interpretability and interactivity. The front-end interface is equipped with an interpretable component that shows the decision process taken by the system when answering questions, which is easy for users to understand. In addition, a component is designed to ensure that users can interact effectively to correct the program with parsing errors. To this end, several specific functions and design elements are integrated into the chat interface. First, a foldable decision process box is customized to display the system decision status and the knowledge source used to generate answers, which enhances the interpretability of the system. Secondly, an "Edit Program" function button (program editing option) is added, allowing users to switch to the program editing interface and edit the program by dragging the operation box. These functions significantly improve the interactivity of the system, enabling users to accurately correct errors and effectively retrieve information from the knowledge base.

[0043] The program editing interface allows users to modify the program with parsing errors through intuitive drag and drop operations. Users can use this function by simply clicking the "Edit Program" button in the lower right corner of the chat interface.

[0044] The present invention innovatively designs a user intervention mechanism to address possible errors that may occur during program parsing. When a user discovers errors in the semantic parsing results during a conversation, the mechanism allows the user to intervene directly and correct the errors in real time, thereby ensuring the accuracy of the question-answering system and the smoothness of user interaction.

[0045] Please refer to Figure 5 , Figure 5 A schematic diagram of the principles of the conversation history tracker provided by the present invention.

[0046] Please refer to Figure 6 , Figure 6 A schematic diagram of the principles of the dialogue strategy and knowledge source provided by the present invention.

[0047] As a preferred embodiment, the question attributes also include random questions; after inputting the current question and natural language understanding prompt words into the pre-trained large language model, and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, it also includes: when the question understanding result is a random question, the answer to the question to be answered generated by the large language model is used as the final answer to the current question.

[0048] In this embodiment, when the question understanding result is a random question, that is, the user's current question is a casual question or a question that does not require searching a knowledge source, the large language model can answer (chat response) based on the model's own parameter knowledge.

[0049] As a preferred embodiment, the question understanding result also includes completeness; completeness includes complete question and incomplete question; when the question understanding result is that the question is complete, the current question is used as the question to be answered; when the question understanding result is that the question is incomplete, the current question is rewritten based on the large language model according to the natural language understanding prompt words, and the rewritten question is used as the question to be answered.

[0050] In view of the limitations of single-round question-answering systems in dealing with complex questions, for multi-round dialogue question-answering, since follow-up questions raised by users (such as questions involving omissions and references) pose a challenge to existing semantic parsers, in this embodiment, after confirming that the current question input requires a factual answer, the large language model will evaluate whether the current question is a complete or incomplete question. In the case where the question understanding result is a complete question, the current question is taken as the question to be answered, and then the above-mentioned knowledge graph retrieval step and the large model answer verification step are performed. In the case where the question understanding result is an incomplete question, the large language model needs to rewrite the current question, and the rewritten question is taken as the question to be answered, and then the above-mentioned knowledge graph retrieval step and the large model answer verification step are performed.

[0051] As a preferred embodiment, the question understanding result also includes clarification; clarification includes whether the question needs clarification or not; when the question understanding result is that the question does not need clarification, the current question is rewritten based on the large language model according to the natural language understanding prompt words, and the rewritten question is used as the question to be answered; when the question understanding result is that the question needs clarification, a clarification response is given to the current question based on the large language model, and the clarified question is used as the question to be answered.

[0052] In this embodiment, for incomplete questions, the large language model will check the context information from the conversation history tracking information to determine whether clarification is needed. If the current question does not need clarification, the large language model will continue to rewrite the question, and use the rewritten question as the question to be answered, and then execute the above-mentioned knowledge graph retrieval steps and the large model answer verification steps. If the current question needs clarification (the user's input question is an incomplete question and is not related to the previous conversation history), the large language model will interact with the user to clarify the current question, thereby using the question after the clarified response as the question to be answered, and then execute the above-mentioned knowledge graph retrieval steps and the large model answer verification steps. The present invention supports users to ask follow-up questions that contain omissions and reference relationships, which effectively improves the interactivity of the system and the depth of question answering.

[0053] As a preferred embodiment, after inputting the current question and natural language understanding prompt words into the pre-trained large language model, and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, it also includes: when the question understanding result is a factual question and the answer cannot be retrieved from the knowledge graph based on semantic parsing, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question; the dialogue history tracking information includes decision history tracking information, back-end chat history information and front-end chat history information.

[0054] In this embodiment, when the result of question understanding is a factual question and the answer cannot be retrieved from the knowledge graph based on semantic parsing, the large language model will check whether the answer can be inferred from the dialogue history tracking information (factual response based on the dialogue history). If the first two knowledge sources, namely the knowledge graph and the dialogue history tracking information, cannot retrieve useful information, the large language model will use the large language model's own parameter knowledge to answer the user's question (factual response based on the large language model's parameter knowledge). If there is still no available answer in the knowledge source in the end, the large language model will clearly indicate that it cannot answer the user's question (no answer response).

[0055] The conversation history tracking information includes decision history tracking information, backend chat history information and frontend chat history information.

[0056] Decision history information is used to track past decision-making processes, which can also be used as context examples for large language models to improve their decision-making capabilities.

[0057] The backend chat history tracking information is used to track short answers obtained from the knowledge graph, avoiding tracking lengthy answers generated by large language models. This history can be used as context information for the dialogue decision module.

[0058] The front-end chat history tracking information is used to track the complete answers given by the question-and-answer system and to display the answers given by the question-and-answer system to users.

[0059] As a preferred embodiment, the neural symbolic knowledge retrieval strategy includes: inputting the question to be answered into a pre-trained semantic parsing model, obtaining a program output by the semantic parsing model that can be used for reasoning and searching on a knowledge graph; wherein the semantic parsing model is trained based on the KQA Pro dataset.

[0060] As a preferred embodiment, the program that can be operated on the knowledge graph is a KoPL program; retrieving answers from the knowledge graph based on semantic analysis includes: using the KoPL program to perform reasoning and search on the knowledge graph through the VisKoP engine to obtain the retrieval answers of the KoPL program.

[0061] In this embodiment, the BART model is trained as a semantic parsing model based on the KQA Pro dataset. The trained semantic parsing model can translate the questions to be answered into a program called KoPL. Then, the VisKoP engine uses the KoPL program to perform reasoning and search on the knowledge graph to obtain the retrieval answer of the KoPL program.

[0062] It should be noted that KoPL is a programming language that operates on various knowledge elements in the knowledge graph (such as concepts, entities, relationships, attributes, modifiers, etc.). It is a programming language specifically used for complex reasoning questions and answers, composed of multiple basic functions. Running this function combination can obtain answers from the knowledge graph.

[0063] The KQA Pro dataset contains 90,000 pieces of data on 'question-KoPL' pairs, which are not particularly limited in the present invention.

[0064] The question-answering system based on a large language model provided by the present invention is described below. The question-answering system based on a large language model described below and the question-answering method based on a large language model described above can refer to each other.

[0065] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of a question-answering system based on a large language model provided by the present invention.

[0066] The present invention also provides a question-answering system based on a large language model, comprising: a determination module 1, used to determine a user's current question and a natural language understanding prompt word; the natural language understanding prompt word includes dialogue history tracking information; an understanding module 2, used to input the current question and the natural language understanding prompt word into a pre-trained large language model, and obtain a question understanding result and a question to be answered output by the large language model according to a natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; a retrieval module 3, used to retrieve an answer from a knowledge graph based on semantic parsing when the question understanding result is a factual question; the semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to a neural symbol knowledge retrieval strategy; a verification module 4, used to input a retrieved answer and an answer verification prompt word into a pre-trained large language model, and obtain an answer verification result output by the large language model; a first result module 5, used to use the retrieved answer as the final answer to the current question when the answer verification result is a reasonable verification; a second result module 6, used to use the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question when the answer verification result is an unreasonable verification.

[0067] As a preferred embodiment, it also includes: an interactive front-end module, which is used to display the generation process of the final answer to the current question in an explainable chat interface; setting program editing options in the explainable chat interface; the program editing options are used to jump to the program editing interface; and the program editing interface is used to modify the program with parsing errors.

[0068] This paper demonstrates the application of knowledge-intensive question-answering in multi-round dialogues and develops a new knowledge question-answering system based on the fusion of multiple knowledge sources. The system aims to have the ability to deeply understand user questions and effectively retrieve and integrate information from different knowledge sources, so as to provide factual answers accurately and efficiently.

[0069] In order to verify the effectiveness of the present invention, automatic evaluation and user experiments were implemented.

[0070] In terms of automatic evaluation, the test was conducted on the ConvQuestions multi-round dialogue dataset. This dataset contains five rounds of dialogue, the first round is a complete question, and the subsequent four rounds are incomplete questions containing references or omissions. Experiments were conducted on 700 rounds of dialogue in this dataset, involving 3,500 knowledge questions and answers. Taking the existing REWRITE method as a benchmark, the experimental results show that the method of the present invention outperforms the benchmark by about 8%. In addition, an ablation experiment was conducted to verify the effectiveness of the dialogue strategy of the present invention.

[0071] In terms of user experiments, the present invention was compared with ChatGLM3, and 25 subjects were recruited to participate in the user experiment. The subjects were asked to rate the two systems based on multiple indicators, including answer accuracy, user preference, explainability, context understanding ability, interactivity of the program editing interface, and overall satisfaction. The system of the present invention is superior to ChatGLM3 in the above indicators. This result not only proves the effectiveness of the system of the present invention, but also shows that it better meets the needs of actual users.

[0072] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the question-answering method based on the large language model, which includes: determining the user's current question and natural language understanding prompt words; the natural language understanding prompt words include dialogue history tracking information; inputting the current question and natural language understanding prompt words into the pre-trained large language model to obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; when the question understanding result is a factual question, retrieving the answer from the knowledge graph based on semantic parsing; semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy; inputting the retrieved answer and the answer verification prompt words into the pre-trained large language model to obtain the answer verification result output by the large language model; when the answer verification result is that the verification is reasonable, the retrieved answer is used as the final answer to the current question; when the answer verification result is that the verification is unreasonable, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question.

[0073] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the question-answering method based on the large language model provided by the above methods, the method including: determining the user's current question and natural language understanding prompt words; the natural language understanding prompt words include dialogue history tracking information; inputting the current question and natural language understanding prompt words into a pre-trained large language model, and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; when the question understanding result is a factual question, retrieving the answer from the knowledge graph based on semantic parsing; the semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy; inputting the retrieved answer and the answer verification prompt words into the pre-trained large language model, and obtaining the answer verification result output by the large language model; when the answer verification result is that the verification is reasonable, the retrieved answer is used as the final answer to the current question; when the answer verification result is that the verification is unreasonable, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question.

[0075] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the question-answering method based on the large language model provided by the above methods, the method comprising: determining the user's current question and natural language understanding prompt words; the natural language understanding prompt words include dialogue history tracking information; inputting the current question and natural language understanding prompt words into a pre-trained large language model, and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; when the question understanding result is a factual question, retrieving the answer from the knowledge graph based on semantic parsing; the semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbol knowledge retrieval strategy; inputting the retrieved answer and the answer verification prompt words into the pre-trained large language model, and obtaining the answer verification result output by the large language model; when the answer verification result is that the verification is reasonable, the retrieved answer is used as the final answer to the current question; when the answer verification result is that the verification is unreasonable, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-answering method based on a large language model, characterized in that: include: Determine the user's current question and natural language understanding prompt words; The natural language understanding prompt words include conversation history tracking information; Input the current question and the natural language understanding prompt word into the pre-trained large language model, and obtain the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; In the case where the question understanding result is the factual question, the answer is retrieved from the knowledge graph based on semantic parsing; the semantic parsing is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbolic knowledge retrieval strategy; Inputting the retrieval answer and the answer verification prompt word into the pre-trained large language model to obtain the answer verification result output by the large language model; If the answer verification result is reasonable, the retrieved answer is used as the final answer to the current question; When the answer verification result is unreasonable, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question.

2. The question-answering method based on a large language model according to claim 1, characterized in that: Also includes: Displaying the generation process of the final answer to the current question in an explainable chat interface; Setting program editing options in the explainable chat interface; The program editing option is used to jump to the program editing interface; the program editing interface is used to modify the program with parsing errors.

3. The question-answering method based on a large language model according to claim 1, characterized in that: The problem attributes also include random problems; After inputting the current question and the natural language understanding prompt words into the pre-trained large language model and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: In the case where the question understanding result is the random question, the answer to the question to be answered generated by the large language model is used as the final answer to the current question.

4. The question-answering method based on a large language model according to claim 1, characterized in that: The problem understanding result also includes completeness; the completeness includes complete problems and incomplete problems; When the question understanding result is that the question is complete, taking the current question as the question to be answered; When the question understanding result is that the question is incomplete, the current question is rewritten based on the large language model according to the natural language understanding prompt word, and the rewritten question is used as the question to be answered.

5. The question-answering method based on a large language model according to claim 1, characterized in that: The problem understanding result also includes clarification; the clarification includes whether the problem needs clarification or not; When the question understanding result is that the question does not need to be clarified, rewriting the current question based on the large language model according to the natural language understanding prompt word, and using the rewritten question as the question to be answered; When the question understanding result is that the question needs to be clarified, a clarification response is made to the current question based on the large language model, and the clarified question is used as the question to be answered.

6. The question-answering method based on a large language model according to claim 1, characterized in that: After inputting the current question and the natural language understanding prompt words into the pre-trained large language model and obtaining the question understanding result and the question to be answered output by the large language model according to the natural language understanding strategy, the method further includes: When the question understanding result is the factual question and the answer cannot be retrieved from the knowledge graph based on the semantic analysis, the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge is used as the final answer to the current question; the dialogue history tracking information includes decision history tracking information, back-end chat history information and front-end chat history information.

7. The question-answering method based on a large language model according to any one of claims 1 to 6, characterized in that: The neural symbolic knowledge retrieval strategy includes: Input the question to be answered into a pre-trained semantic parsing model, and obtain a program output by the semantic parsing model that can be used for reasoning and searching on the knowledge graph; The semantic parsing model is trained based on the KQA Pro dataset.

8. The question-answering method based on a large language model according to claim 7, characterized in that: The program that can operate on the knowledge graph is a KoPL program; The method of retrieving answers from the knowledge graph based on semantic parsing includes: The VisKoP engine uses the KoPL program to perform reasoning and search on the knowledge graph to obtain the retrieval answer of the KoPL program.

9. A question-answering system based on a large language model, characterized in that: include: A determination module, used to determine the user's current question and natural language understanding prompt words; The natural language understanding prompt words include conversation history tracking information; An understanding module, used for inputting the current question and the natural language understanding prompt words into a pre-trained large language model, and obtaining a question understanding result and a question to be answered output by the large language model according to a natural language understanding strategy; the question understanding result includes question attributes; the question attributes include factual questions; A retrieval module, used for retrieving an answer from a knowledge graph based on semantic analysis when the question understanding result is the factual question; the semantic analysis is to parse the question to be answered into a program that can be operated on the knowledge graph according to the neural symbolic knowledge retrieval strategy; A verification module, used for inputting the search answer and the answer verification prompt word into the pre-trained large language model to obtain the answer verification result output by the large language model; A first result module, configured to use the retrieved answer as the final answer to the current question if the answer verification result is reasonable; The second result module is used to use the answer generated by the large language model according to the dialogue history tracking information and / or the model's own parameter knowledge as the final answer to the current question when the answer verification result is unreasonable.

10. The question-answering system based on a large language model according to claim 9, characterized in that: Also includes: An interactive front-end module, used to display the generation process of the final answer to the current question in an interpretable chat interface; Setting program editing options in the explainable chat interface; The program editing option is used to jump to the program editing interface; The program editing interface is used to modify programs with parsing errors.

Images