Open-domain natural language reasoning question answering system and method driven by large language model

Abstract

The application provides a large language model driven open domain natural language reasoning question and answer system and method, a question rewriting module rewrites a user question into a rewritten question; a center calculation and management module manages calculation and knowledge resources of a large language model, and outputs the calculation and knowledge resources of the large language model required by a question core engine module to one or more sub-question and answer modules in a question and answer core engine module according to the type of the rewritten question; the question and answer core engine module reasons to obtain one or more candidate answers of the rewritten question and explainability information of the candidate answers according to the rewritten question and the calculation and knowledge resources of the large language model; and an aggregation reasoning module aggregates and reasons to obtain a final answer of the rewritten question and explainability information of the final answer according to the one or more candidate answers of the rewritten question and the explainability information of the candidate answers, and supports comprehensive question types, is easy to expand, is explainable, and has strong universality by using a large language model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to an open domain natural language reasoning question and answer system and method driven by a large language model. BACKGROUND

[0002] With the development of the Internet and the explosive growth of digital information, how to improve the efficiency and accuracy of intelligent question answering has become a research hotspot. Intelligent question answering (QA) aims to automatically provide answers to natural language questions raised by users, and is one of the most representative tasks in the field of natural language processing (NLP). Through unified task learning or instruction learning technology, various NLP tasks can be unified into question answering tasks, which has become an important trend in the development of NLP.

[0003] Existing open domain natural language reasoning question and answer systems generally include three stages: question analysis, knowledge retrieval and answer generation, but have problems such as low coverage of reasoning type support, poor scalability, weak interpretability, etc. The current open domain dialogue large language model ChatGPT cannot utilize large-scale structured knowledge graphs, external text resources (including local and Internet text resources) and other information for knowledge reasoning and question answering, and still has problems such as weak information timeliness and untrustworthiness, especially in processing field professional tasks. SUMMARY

[0004] The present application provides an open domain natural language reasoning question and answer system and method driven by a large language model, which solves the defects of the existing question and answer system, such as low coverage of reasoning type support, poor scalability, weak interpretability, etc., and adopts a large language model to support comprehensive problem types, easy expansion, interpretability and strong universality.

[0005] The application provides a large language model driven open domain natural language reasoning question and answer system, comprising a question rewriting module, a center calculation and management module, a question and answer core engine module and an aggregation reasoning module; the question and answer core engine module comprises a plurality of sub-question and answer modules, and the question reasoning logics of the plurality of sub-question and answer modules are different; an output end of the question rewriting module is connected with an input end of the center calculation and management module, and is used for rewriting a user question to obtain a rewritten question; an output end of the center calculation and management module is connected with an input end of the question and answer core engine module, and is used for managing calculation and knowledge resources of the large language model, and outputting the calculation and knowledge resources of the large language model required by the question and answer core engine module to one or more sub-question and answer modules in the question and answer core engine module according to the type of the rewritten question; an output end of the question and answer core engine module is connected with an input end of the aggregation reasoning module, and is used for reasoning one or more candidate answers of the rewritten question and explainability information of the candidate answers according to the rewritten question and the calculation and knowledge resources of the large language model; and the aggregation reasoning module is used for aggregating and reasoning a final answer of the rewritten question and explainability information of the final answer according to the one or more candidate answers of the rewritten question and the explainability information of the candidate answers.

[0006] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the reply generation module connected with the aggregation reasoning module is used for converting the final answer of the rewritten question and the explainability information of the final answer into a machine reply to obtain a reply answer of the user question and explainability information of the reply answer; and the dialogue interaction module connected with the reply generation module is used for carrying out dialogue interaction with the user question according to the reply answer of the user question and the explainability information of the reply answer.

[0007] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the question rewriting module is specifically used for carrying out anaphora resolution or ellipsis resolution on the user question according to historical question and answer data to obtain a rewritten question, and triggering a rewriting prompt when the rewritten question is different from the user question, so that the user can select.

[0008] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the center computing and management module comprises: a knowledge graph library management module for providing knowledge query and reasoning execution of a knowledge graph library; a text resource library management module for providing text query support of a local text resource library; a large language model computing and management module for providing model computing and task adaptation support of a large language model library and a small amount of fine-tuning parameter library adapted to the large language model task; a management hub module for scheduling the requirements of a plurality of sub-question and answer modules for the knowledge graph library, the local text resource library, the large language model library and the small amount of fine-tuning parameter library adapted to the large language model task, and selecting one or more sub-question and answer modules from a plurality of sub-question and answer modules according to the type of the rewritten question to reason the rewritten question to obtain a candidate answer of the rewritten question and explainability information of the candidate answer.

[0009] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the aggregation reasoning module is specifically configured to aggregate and reason the final answer of the rewritten question and the explainability information of the final answer from the candidate answer of the rewritten question and the explainability information of the candidate answer based on a score fusion mechanism, an optimization mechanism of user feedback and an iterative comprehensive reasoning mechanism.

[0010] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the plurality of sub-question and answer modules comprise: a knowledge base question and answer module for reasoning the candidate answer of the rewritten question and the explainability information of the candidate answer based on a large-scale knowledge graph; a reading comprehension question and answer module for reasoning the candidate answer of the rewritten question and the explainability information of the candidate answer based on locally collected text and / or online Internet retrieved text; and a large language model question and answer module for reasoning the candidate answer of the rewritten question and the explainability information of the candidate answer based on linguistic knowledge, common sense knowledge and factual knowledge inside the large language model.

[0011] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the knowledge base question and answer module comprises: a semantic parsing module based on entity linking and entity replacement, configured to identify entities in the rewritten question, replace the entities, and map semantic parsing of the rewritten question to a logical expression executable by a logical expression execution module supporting neural-symbol hybrid reasoning; and the logical expression execution module supporting neural-symbol hybrid reasoning is configured to query information from a structured knowledge base or an unstructured text library based on the logical expression to reason the candidate answer of the rewritten question and the explainability information of the candidate answer.

[0012] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the reading comprehension question and answer module comprises: an evidence retrieval module for retrieving local text data and Internet data to obtain evidence candidates; and a knowledge reasoning model for reasoning to obtain candidate answers to the rewritten question and explainability information of the candidate answers according to the evidence candidates.

[0013] According to the large language model driven open domain natural language reasoning question and answer system provided by the application, the large language model question and answer module comprises: an answer generation module based on knowledge detection, which uses a non-autoregressive generative pre-training language model as a large language model, and performs answer generation task adaptation through a pluggable small amount of parameter efficient learning technology; an explanation generation module based on demonstration, which performs example construction and large language model explanation generation based on the answer generation task adaptation and using a generative large language model as a large language model, to reason to obtain candidate answers to the rewritten question and explainability information of the candidate answers; and the large language model question and answer module is further used to determine whether the rewritten question is a preset standpoint question according to a preset standpoint refutation mechanism and generate a large language model refutation explanation.

[0014] The application further provides a large language model driven open domain natural language reasoning question and answer method, which is applicable to the large language model driven open domain natural language reasoning question and answer system described above, and comprises: the question rewriting module rewrites a user question to obtain a rewritten question; the center calculation and management module outputs the rewritten question and calculation and knowledge resources of the large language model required by the question core engine module to one or more sub-question and answer modules in the question and answer core engine module according to the type of the rewritten question; the question and answer core engine module reasons to obtain candidate answers to one or more rewritten questions and explainability information of the candidate answers according to the rewritten question and the calculation and knowledge resources of the large language model; and the aggregation reasoning module aggregates and reasons to obtain a final answer to the rewritten question and explainability information of the final answer according to the candidate answers to one or more rewritten questions and the explainability information of the candidate answers.

[0015] The application provides a large language model driven open domain natural language reasoning question and answer system and method, a question rewriting module rewrites a user question to obtain a rewritten question; a center calculation and management module manages calculation and knowledge resources of a large language model, and according to the type of the rewritten question, outputs the calculation and knowledge resources of the large language model required by a question core engine module to one or more sub-question and answer modules in the question and answer core engine module; the question and answer core engine module reasons to obtain one or more candidate answers of the rewritten question and explainability information of the candidate answers according to the rewritten question and the calculation and knowledge resources of the large language model; and an aggregation reasoning module aggregates and reasons to obtain a final answer of the rewritten question and explainability information of the final answer according to the one or more candidate answers of the rewritten question and the explainability information of the candidate answers. The whole system is supported by a large language model, supports comprehensive question types, is easy to expand, is explainable and has strong universality. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0017] Figure 1 is a structural schematic diagram of a large language model driven open domain natural language reasoning question and answer system provided by the present application;

[0018] Figure 2 is a technical framework schematic diagram of a large language model driven open domain natural language reasoning question and answer system provided by the present application;

[0019] Figure 3 is an example diagram of question rewriting effect in the system provided by the present application;

[0020] Figure 4 is a principle schematic diagram of a question rewriting module provided by the present application;

[0021] Figure 5 is a principle schematic diagram of a center calculation and management module provided by the present application;

[0022] Figure 6 is a principle schematic diagram of an aggregation reasoning module provided by the present application;

[0023] Figure 7 is an iterative comprehensive reasoning flow schematic diagram of an aggregation reasoning module provided by the present application;

[0024] Figure 8It is the principle schematic diagram of the knowledge base question and answer module based on semantic analysis and logical expression execution provided by the application;

[0025] Figure 9 It is the principle schematic diagram of the reading comprehension question and answer module based on open domain knowledge reasoning provided by the application;

[0026] Figure 10 It is the principle schematic diagram of the evidence retrieval module provided by the application;

[0027] Figure 11 It is the principle schematic diagram of the knowledge reasoning model provided by the application;

[0028] Figure 12 It is the principle schematic diagram of the large language model question and answer module based on knowledge detection and demonstration provided by the application;

[0029] Figure 13 It is one of the interface schematic diagrams of the open domain natural language reasoning question and answer system driven by the large language model provided by the application;

[0030] Figure 14 It is the second interface schematic diagram of the open domain natural language reasoning question and answer system driven by the large language model provided by the application;

[0031] Figure 15 It is the third interface schematic diagram of the open domain natural language reasoning question and answer system driven by the large language model provided by the application;

[0032] Figure 16 It is the flow schematic diagram of the open domain natural language reasoning question and answer method driven by the large language model provided by the application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application will be described clearly and completely below in combination with the drawings in the present application. The described embodiments are part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the present application.

[0034] Intelligent question answering aims to automatically provide answers to natural language questions raised by users, and is one of the most representative tasks in the field of natural language processing (NLP). In recent years, thanks to the massive growth of Internet data, the rapid improvement of hardware computing power, and the great progress of NLP and deep learning technology, intelligent question answering methods have made great progress, and various successful applications have emerged, such as dialogue robots, intelligent voice assistants, search engines, etc. Although great progress has been made in question answering technology, due to the complexity of human language and the openness of the question answering scene, there are high requirements for the intelligence level of the system, which requires the system to accurately understand user questions and have various complex reasoning capabilities (such as multi-hop reasoning, numerical reasoning, common sense reasoning, etc.), and the ability to answer questions from different knowledge source data (such as text data, knowledge graphs, etc.). Building an open-domain question answering system that supports complex scenarios has always been the direction of efforts in the field of intelligent question answering. Due to the huge demand for intelligent question answering, the academic and industrial communities have been exploring it without any effort.

[0035] However, these existing question answering technologies also exhibit many serious problems or limitations:

[0036] (1) Low coverage of reasoning type support. User questions often involve various reasoning type problems, such as multi-hop reasoning, numerical reasoning, common sense reasoning, and logical reasoning. Current systems can only cover a part of them.

[0037] (2) Poor scalability. The language intelligence models currently used by question answering systems are mainly small models, which are limited to specific domains, and the system technology is difficult to quickly expand to other application domains. Even if large language model technology is used, there is a lack of efficient computing management design, leading to problems such as difficulty in domain adaptation and high computing cost.

[0038] (3) Weak explainability. The system has weak explainability for answers, and usually only provides answers without presenting the source, execution process, and explanation of the answers. Current large language model question answering technologies such as ChatGPT have a serious "hallucination" problem in reply generation, i.e., generating unsafe, untrustworthy, and imaginary content.

[0039] (4) Lack of refutation ability. Current reading comprehension question answering and large language model question answering lack the ability to refute pre-set premises (False Premise) questions, such as the question "How many eyes do people have?" TextQA and GPT-3-based question answering systems cannot correctly answer this question and will provide an absurd answer of "2 eyes".

[0040] (5) Open domain question answering performance is insufficient. The technology currently used by intelligent voice assistants is still a traditional small model, and the general intelligence level is poor. In addition, the source of knowledge for answering questions is single, mainly using knowledge graph or text retrieval, and lacking the use of Internet online text, large language model knowledge, etc.

[0041] (6) Context understanding is insufficient. In the question answering system, users are usually accustomed to continuous questioning, and the question often involves reference, omission, etc. The existing question answering system lacks the ability to understand the historical dialogue context in such scenarios, resulting in insufficient question understanding accuracy.

[0042] To break through the above problems existing in the existing question answering system, the present application is supported by large language model technology, and a complex question answering system supporting open domain, explainability and easy expansion is invented.

[0043] The application is described below Figures 1-16 A large language model driven open domain natural language reasoning question answering system and method are described.

[0044] Please refer to Figure 1 , Figure 1 A structural diagram of a large language model driven open domain natural language reasoning question answering system provided by the present application.

[0045] Please refer to Figure 2 , Figure 2 A technical framework diagram of a large language model driven open domain natural language reasoning question answering system provided by the present application.

[0046] The present application provides a large language model driven open domain natural language reasoning question answering system, which comprises a question rewriting module 1, a center calculation and management module 2, a question answering core engine module 3 and an aggregation reasoning module 4; the question answering core engine module 3 comprises a plurality of sub-question answering modules, and the question reasoning logic of the plurality of sub-question answering modules is different;

[0047] The output end of the question rewriting module 1 is connected with the input end of the center calculation and management module 2, which is used for rewriting the user question to obtain a rewritten question;

[0048] The output end of the center calculation and management module 2 is connected with the input end of the question answering core engine module 3, which is used for managing the calculation of the large language model and the knowledge resources of the system, and outputting the calculation of the large language model and the knowledge resources required by the question core engine module 3 to one or more sub-question answering modules in the question answering core engine module 3 according to the type of the rewritten question;

[0049] The output end of the question and answer core engine module 3 is connected with the input end of the aggregation reasoning module 4, and is used for obtaining one or more candidate answers of the rewritten question and the explainability information of the candidate answers according to the calculation and knowledge resource reasoning of the rewritten question and the large language model.

[0050] The aggregation reasoning module 4 is used for obtaining the final answer of the rewritten question and the explainability information of the final answer according to the candidate answers of the rewritten question and the explainability information of the candidate answers.

[0051] Specifically, the application provides a large language model driven open domain natural language reasoning question and answer system, which adopts modular design and mainly includes a question rewriting module 1, a central calculation and management module 2, a question and answer core engine module 3 and an aggregation reasoning module 4.

[0052] The question rewriting module 1 (Rewriter) rewrites the user question according to the question and answer or dialogue history data, mainly solves the problems such as reference and omission in the question, and realizes the function of fusing historical information and dialogue interaction. This method is helpful to improve the semantic understanding ability of the subsequent question and answer engine.

[0053] The central calculation and management module 2 (Central Computation and Management, CCM) uniformly manages the large language model used by the system, and provides efficient calculation support for the pluggable small parameter efficient learning technology (DeltaTuning) of the related task. Reduce the memory occupation of the large language model, the difficulty of domain task adaptation, and improve the scalability of the system. At the same time, the module can provide knowledge integration, knowledge query, database management and other functions for large-scale knowledge graph (general domain knowledge graph such as Wikidata, domain knowledge graph).

[0054] The question and answer core engine module 3 can include a knowledge base question and answer module KBQA, an open domain reading comprehension question and answer module TextQA and a large language model question and answer module BMQA.

[0055] Knowledge Base Question Answering Module KBQA: provides the ability to obtain knowledge from large-scale knowledge graphs for fact knowledge question answering, such as "Which is taller, Yao Ming or James?" The module mainly includes two parts: mapping of question text to logical expressions and execution of logical expressions. The module provides symbolic reasoning, neural-symbolic hybrid reasoning, and other functions for the system. The KBQA method based on logical expressions searches for answers from large-scale structured knowledge graphs, has the advantages of accurate answers and explainability, and is a strict symbolic reasoning method. In addition, the KBQA module also designs a knowledge reasoning channel that integrates question-related entity subgraph retrieval (KBQuery), i.e., KBQuery generation + subgraph retrieval + neural-symbolic reasoning. First, the rewritten question is input into the Program generation model to generate a subgraph query statement KBQuery. Then, the knowledge graph database management module uses subgraph retrieval to obtain a knowledge subgraph from the knowledge graph. Finally, the retrieved knowledge subgraph is converted into a sequence of text through JSON or direct concatenation, and input into a large language model for knowledge reasoning, which can realize the fuzzy knowledge reasoning ability of the knowledge base question answering.

[0056] Open Domain Reading Comprehension Question Answering Module TextQA: uses a retrieval + reading comprehension technology framework to support the retrieval of question-related text fragments from locally collected text and online Internet retrieved text for reading comprehension and answering. Through open domain text, the system supports various open domain questions. In text retrieval, a dense retrieval + neural retrieval technology of re-ranking based on a large language model is used. In order to improve the efficiency of dense vector retrieval, a hierarchical navigable small world vector search algorithm (Hierarchical Navigable Small Worlds, HNSW) is used. At the same time, a reading comprehension model that integrates numerical reasoning + logical reasoning is constructed. This module greatly improves the accuracy of text retrieval and the ability to answer open domain questions through large language model technology, and supports numerical reasoning, logical reasoning, and other functions.

[0057] Big Language Model Question Answering Module BMQA: aims to use big language model as knowledge source to answer questions. Big language model is trained on massive general text, which has encoded rich linguistic knowledge, common sense knowledge and even rich factual knowledge. It adopts prompt question answering + demonstration answer explanation technology framework. This module is similar to the human brain answering questions through hippocampal memory information. In practice, it shows better common sense reasoning ability compared with other question answering modules. To address the lack of rebuttal ability for preset positions in existing systems, a preset position dataset is constructed, and combined with delta tuning, few-shot prompting, pareto multi-task learning, data replay learning, etc. to realize the big language model's ability to answer questions with preset positions, and has better rebuttal and explanation effect.

[0058] Aggregator: aims to aggregate and reason the best reply based on the replies of the three question answering modules, considering that different question answering modules are good at different types of questions. This model plays the role of mutual complementation and verification of different question answering modules. In the design, an online optimization mechanism based on user feedback is constructed, and the system reply effect is continuously optimized through user feedback during system operation and use.

[0059] Deep learning big language models represented by pre-trained language models (PLM) have greatly refreshed the performance of various NLP tasks and become a breakthrough technology in the field of NLP and artificial intelligence. PLM adopts the "pre-training-fine-tuning" paradigm, that is, a pre-trained model is obtained through self-supervised learning on a large amount of internet text corpus, and then fine-tuned on a small amount of labeled data to adapt to downstream tasks. Compared with traditional artificial intelligence models, big language models have three advantages: first, the data cost of big language models is low, mainly using unlabeled public data for large-scale training. At the same time, only a small amount of specific domain data is needed to fine-tune the big language model to complete the downstream task learning, which speeds up the deployment and application of the model in specific fields; second, big language models are versatile and can be applied to various artificial intelligence tasks without the need to develop corresponding models for specific tasks, improving research and development efficiency; third, big language models have good comprehensive performance, and the general big language model can outperform traditional artificial intelligence models trained for specific tasks in each specific task. Using big language model technology makes it possible to build a high-level and general intelligent question answering system. In this patent, unless otherwise specified, big language model refers to large-scale pre-trained language model.

[0060] To solve the problems of poor scalability of problem (2) and insufficient performance of open-domain question answering of problem (5), the solution is large language model support, center computing and management. We use the current pre-trained large language model as the basic model of the question answering system. Through the general intelligent potential and strong language understanding ability of the large language model, the system can support different fields and different problem types in the open domain scenario. Considering the problems of high computing cost, difficulty in fine-tuning of domain task data and slow model inference speed caused by the large-scale parameters of the large language model, we design the center computing and management module 2 to share and uniformly compute and schedule multiple large language models involved in the system. For each sub-task application, we use the pluggable few-parameter efficient fine-tuning (Delta-Tuning) method for adaptation, that is, we fix the parameters of the large language model and only fine-tune a small number of model parameters (such as prompt fine-tuning, adapter fine-tuning, bias fine-tuning, etc.), reducing the system's demand for computing platform video memory and GPU video card. Through the embedding of TextQA, BMQA, KBQA, etc., we can answer questions from open-domain text, large language models, and knowledge graph data sources.

[0061] To build a large language model supported question answering system, for example, in KBQA, we convert user questions into executable program (Program) language to retrieve and reason from knowledge graphs, achieving accurate answers to a variety of complex reasoning questions. Here we can use powerful generative PLM models (such as T5 models) to generate program (Program) language. In open TextQA, we accurately retrieve relevant documents from collected or Internet massive text data, and then generate answers from relevant documents according to the reading comprehension model, building a large language model driven neural retrieval model (dense retrieval + re-ranking) method. Here, dense retrieval refers to using a large language model to encode documents into semantic representation vectors for retrieval. Experiments show that it can significantly outperform traditional statistical models (such as BM25). Then we can use the large language model to generate answers based on the relevant retrieval documents.

[0062] For problem (1), the coverage of reasoning types is low, and the solution is to integrate multi-type reasoning and multi-information source mechanism. We draw on the semantic understanding and symbolic reasoning mechanisms of the human brain to build a system architecture with multiple reasoning mechanisms such as symbolic reasoning, neural reasoning, and symbolic-neural hybrid reasoning. Specifically, considering the three representative functional structures of human brain language information processing, namely the language loop structure that is good at symbolic reasoning, the hippocampus structure that reads and writes through memory mechanism, and the linkage structure of hippocampus and cerebral cortex with reading perception, we embed the knowledge base question answering module KBQA that supports symbolic reasoning, the large language model question answering module BMQA that supports neural reasoning, and the reading comprehension question answering module TextQA that has numerical reasoning. KBQA uses the Program method to convert questions into executable programs (such as KoPL (Knowledge oriented Programming Language, a programming language designed for complex problem reasoning, SPARQL (SPARQL Protocol and RDF Query Language, a query language for knowledge graph resource description framework)), etc.), and converts the reasoning process into function operation, with functions such as multi-hop reasoning, numerical reasoning, logical reasoning, and factual reasoning. In TextQA, a new reading comprehension architecture is designed, integrating functions such as multi-hop reasoning, numerical reasoning, and logical reasoning. In BMQA, the large language model is used as the knowledge source, and through mechanisms such as preset stance refutation ability activation and knowledge detection, functions such as preset stance question refutation, common sense reasoning, and factual reasoning are realized.

[0063] For problem (3), the explainability is weak, and the solution is modular design, showing reasoning paths, answer analysis, etc. The system uses modular design, dividing the system into question rewriting, information retrieval, reading comprehension reasoner, large language model question answering, large language model explanation generation, and aggregated reasoning modules. The results of each module can be output, allowing the system to grasp the intermediate step information in the question answering process, facilitating system control and understanding of the operation mechanism. In addition, in the KBQA process, the execution process and intermediate results of the answer are given through program visualization; in TextQA, the evidence sources of the answer are listed; in BMQA, the answer is explained and demonstrated through the large language model demonstration method. The system provides explainability from question analysis, answer execution process, evidence and explanation of the answer, etc.

[0064] For problem (4), the rebuttal ability is weak, and the solution is to preset the activation mechanism of the stance question and answer data. By constructing the question and answer data set of the preset stance, the mechanism of Pareto multi-task learning + few-parameter efficient fine-tuning (Delta Tuning) + data replay (Data Replay) + few-sample prompt method is used to activate the rebuttal ability of the large language model to the preset stance question, and the answering ability of BMQA to common sense questions is improved.

[0065] For problem (6), the solution is to rewrite the question in combination with the context. In combination with the dialogue history data, a coreference resolution question rewriting model is constructed, the question is rewritten before answering the question, and the question understanding ability of the subsequent question and answer module is improved.

[0066] The large language model driven open domain natural language reasoning question and answer system provided by the application is an open question and answer system framework that can support complex reasoning. The system draws on the semantic understanding and symbolic reasoning mechanism of the human brain, can support multi-hop reasoning, logical reasoning, knowledge reasoning, numerical reasoning and other complex reasoning, and has the characteristics of explainability and scalability. The system mainly takes large language model, knowledge base question and answer, information retrieval, neural / symbolic reasoning as core technologies, takes large-scale structured knowledge graph, unstructured text data, large language model as knowledge sources for question answering, verifies and supplements each other between different types of knowledge sources, supports the question and answer ability of common sense questions, factual questions, logical reasoning questions, numerical reasoning questions and other complex questions. The system realizes the explainability of question and answer through question analysis, answer evidence and answer explanation. The system designs a center computing and management module to build an easy-to-expand and high-performance question and answer system. The system technology can be migrated to various language processing intelligent systems to provide information services and decision support.

[0067] In summary, the large language model driven open domain natural language reasoning question and answer system provided by the application adopts a large language model, combines a pluggable few-parameter efficient learning technology to realize efficient calculation and expansion management of the large language model in the system, and can obtain knowledge from structured knowledge graphs, unstructured text and large language models to answer questions, support comprehensive question types, easy expansion, explainability and strong universality.

[0068] On the basis of the above embodiment:

[0069] As a preferred embodiment, the system further comprises a reply generation module connected to the output end of the aggregation reasoning module 4, configured to convert the final answer to the rewritten question and the explainability information of the final answer into machine replies, to obtain a reply answer to the user question and explainability information of the reply answer; and a dialogue interaction module connected to the reply generation module, configured to perform dialogue interaction with the user question according to the reply answer to the user question and the explainability information of the reply answer.

[0070] In the embodiment, the large language model driven open domain natural language reasoning question and answer system further comprises a reply generation module and a dialogue interaction module. The reply generation module is configured to provide more friendly machine replies through style conversion, controllable generation, visual display and the like based on the aggregation reasoning result. The dialogue interaction is configured to integrate the Human-in-the-Loop technology in the system, to construct a question and answer system with dialogue interaction, and to support more accurate or continuous question and answer ability through dialogue interaction.

[0071] The above modules can be roughly divided into three categories: question understanding, information retrieval, reasoning and answering. The three types of modules have multiple implementation methods, and are combined to form three types of question and answer system technical routes: KBQA based on structured knowledge base and symbolic reasoning, TextQA based on open domain text knowledge and neural reasoning, and BMQA based on large language model internal knowledge and knowledge probe. The three technical routes complete the answer to the question in a cooperative manner.

[0072] Please refer to Figure 3 , Figure 3 The question rewriting effect provided by the present application is shown in the system example diagram.

[0073] Please refer to Figure 4 , Figure 4 The principle diagram of the question rewriting module provided by the present application is shown in the system example diagram.

[0074] As a preferred embodiment, the question rewriting module 1 is specifically configured to perform anaphora resolution or ellipsis resolution on the user question according to the question and answer history data to obtain a rewritten question, and not to trigger a rewriting prompt when the rewritten question is the same as the user question, and to trigger the rewriting prompt when the rewritten question is different from the user question, for the user to select.

[0075] Specifically, the question rewriting module 1 is mainly responsible for rewriting the input question. After the user inputs a question, the question and answer system first translates the Chinese question into English, and then uses the question rewriting module 1 to process the input question, rewrites the question into a form convenient for the question and answer module to process, and inputs the question as the center of the computing management module and each question and answer module. The question and answer task refers to that the machine finds the answer in a certain information source according to the question put forward by the user. Since the user cannot necessarily obtain all the information he wants through the answer to one question, he often asks further questions based on the existing question and answer to understand more related information. Since the question is put forward based on the existing information, the user may simplify the question, flexibly use the reference and omission, and make the question more concise and compact. The effect example is shown in Table 1:

[0076] Table 1: Example of the role of question rewriting in multi-round question and answer

[0077]

[0078]

[0079] In this example, the user first asks “Who wrote The Old Man and the Sea?” and the system gives the answer “Ernest Hemingway”. When the user wants to ask further questions on this topic, he asks the question “What awards did he win?” The pronoun “he” needs to be understood in combination with the previous question and answer history. The question rewriting module 1 understands that “he” refers to the answer “Ernest Hemingway” in the previous question through reference resolution, and therefore rewrites the question as “What awards did Ernest Hemingway win?” so that the system can accurately understand the user’s intention. The user then asks “Who is the main character of this book?” and the system rewrites the question as “Who is the main character of The Old Man and the Sea?” and answers “The old fisherman named Santiago”. Then the user asks “What fish did he catch?” In addition to the reference of “he”, the information of the book name is also omitted, so the question rewriting module 1 completes the question as “What fish did Santiago catch in The Old Man and the Sea?” so that the system can answer.

[0080] In the interface of the question and answer system, when the user inputs a question, the system will rewrite the question in real time and ask the user whether the rewriting is correct through the prompt below the input box. If the rewriting is correct, the user can click on the rewritten question, which will be automatically filled into the input box.

[0081] After integrating the question rewriting module 1, the user can continuously ask the system about the same topic and use references and omissions in the questions, making the questioning process more concise and efficient, and making the interaction between the user and the system more coherent and interactive.

[0082] The input of the question rewriting module 1 includes two parts, one is the history of the previous user and the system, and the other is the current input question of the user; the output is the rewritten question, which should not contain any ambiguous information such as reference and omission, so that the question and answer module can accurately understand the user's intention and give the answer. The core part of the module is a large-scale language model based on self-attention transformation network (such as T5 series model, GPT (Generative Pre-trained Transformer) series model, etc.), which can encode the input language text into vector representation, and then use the decoder to output the language text that meets the task requirements. For the question rewriting task, the question rewriting module 1 first splices the question and answer history and the current question together as the input of the language model, and the trained model outputs the rewritten question.

[0083] Prompt judgment mechanism: the module will judge whether the rewritten question is consistent with the original question. If the question text does not change, it means that the question does not need to be rewritten, and the function of prompting rewriting in the interface will not be triggered; if the question text changes, it means that the question contains ambiguous information that needs to be understood in combination with the question and answer history, which will trigger the rewriting prompt function in the system interface for the user to choose to confirm.

[0084] In summary, the question rewriting module 1 constructs a question rewriting process suitable for an intelligent question and answer system, which designs a prompt judgment mechanism to determine whether to give the user a rewriting prompt by judging whether the rewritten question is consistent with the original question, making the interaction between the user and the system more coherent and interactive.

[0085] Please refer to Figure 5 , Figure 5 The principle diagram of the central computing and management module provided by the application is shown.

[0086] As a preferred embodiment, the central computing and management module 2 includes: a knowledge graph library management module for providing knowledge query and reasoning execution of the knowledge graph library; a text resource library management module for providing text query support of the local text resource library; a large language model computing and management module for providing model computing and task adaptation support of the large language model library and a small number of fine-tuning parameter library for task adaptation of the large language model; a management hub module for scheduling the needs of the multiple sub-question and answer modules for the knowledge graph library, the local text resource library, the large language model library and the small number of fine-tuning parameter library for task adaptation of the large language model, and selecting one or more sub-question and answer modules from the multiple sub-question and answer modules to reason the rewritten question according to the type of the rewritten question, to obtain one or more candidate answers of the rewritten question and the explainability of the candidate answers.

[0087] Specifically, the center computing and management module 2 aims to uniformly manage the large language model library, the small parameter fine-tuning (Delta) library, the knowledge graph library, the text resource library, etc. used by the question and answer system, and provide different sources of knowledge for related tasks, such as large language model knowledge, factual knowledge, and text knowledge. At the same time, it provides efficient adaptation support for large language models, such as fine-tuning a small number of parameters (Delta), model compression, and inference acceleration. This module reduces the memory usage of the system, reduces the difficulty of domain task adaptation, and improves the scalability of the system by uniformly managing and scheduling large language model computing and knowledge resources.

[0088] The center computing and management module 2 mainly includes a knowledge graph library management module, a text resource library management module, a large language model computing and management module, and a management hub module.

[0089] The knowledge graph library management module mainly provides factual knowledge support for the question and answer system through large-scale knowledge graph queries, including two parts: knowledge graph library and knowledge base efficient management. The knowledge graph library refers to large-scale knowledge graphs such as Wikidata and DBpedia. Knowledge base efficient management provides knowledge query and reasoning execution functions, including 1) Program logic layer, such as KoPL and SPARQL knowledge graph query language; 2) Entity linking, which links entity mentions in given problem text to entities in the knowledge graph to achieve entity semantic disambiguation; 3) Subgraph retrieval, which retrieves subgraph information related to entities in the problem text, which can be used for entity visualization and problem background knowledge augmentation.

[0090] The text resource library management module mainly provides efficient text query support for local text resources. This text resource can be collected encyclopedic text data or accumulated domain text data, including sparse retrieval (such as BM25), text vectorization, and dense vector retrieval (such as HNSW).

[0091] The large language model computing and management module aims to provide unified large language models and efficient computing support for various modules of the system, including a large language model library, a Delta library, a large language model efficient computing and management sub-module. Among them, the large language model library refers to large language models such as T5, CPM, GPT-3, etc., the Delta library refers to a library formed by adapting a small number of parameters (referred to as Delta) of a large language model based on Delta Tuning technology, such as Delta Delta1 for adapting program generation, Delta Delta2 for adapting question rewriting, Delta Delta3 for adapting dense retrieval models, etc., where Delta1, Delta2, Delta3,..., etc. form the Delta library. Delta Tuning is a parameter-efficient learning technology for large language models. When fine-tuning a large language model for a downstream task j , fix the large language model parameters theta, add a small number of parameters Delta j to the model, and only fine-tune the incremental parameters to obtain the task j model Delta fine-tuned model The model prediction effect of large language model full parameter fine-tuning is a "four ounces displace a thousand catties" technology for large language models. Fine-tuning only a small number of incremental parameters Delta helps to avoid saving a large language model for different tasks, only needs to save a base model, and can call the corresponding Delta from the Delta library for different tasks, to reduce the system's memory and achieve the purpose of easy expansion of the system. In addition, the large language model efficient computing and management sub-module, including Delta fine-tuning, large language model efficient compression, large language model efficient inference, etc., realizes efficient adaptation of large language models to various modules and reduces computing costs.

[0092] The management hub module mainly schedules the requirements of various modules for knowledge graphs, local text resources, large language models, Delta libraries, etc., to ensure efficient operation of the system. At the same time, it can select appropriate modules from KBQA, TextQA, and BMQA to answer questions according to the type of the question.

[0093] In summary, the central computing and management module 2 can select appropriate modules to answer questions according to the type of the question; the method of managing large language model libraries and Delta libraries reduces the computing cost of the system and improves the scalability of the system.

[0094] Please refer to Figure 6 , Figure 6 The principle diagram of the aggregation reasoning module provided by the present application.

[0095] Please refer to Figure 7 , Figure 7 The iteration comprehensive reasoning flowchart of the aggregation reasoning module provided by the present application.

[0096] As a preferred embodiment, the aggregation reasoning module 4 is specifically configured to aggregate reasoning to obtain the final answer and the explainability information of the final answer of the rewriting question from the candidate answers and the explainability information of the candidate answers of one or more rewriting questions based on a score fusion mechanism, an optimization mechanism of user feedback and an iterative comprehensive reasoning mechanism.

[0097] Specifically, a more intelligent question and answer system often needs to have multiple different question and answer capabilities. In the technical framework of the present patent, a modular design is adopted, and different expert question and answer modules (i.e., the above-mentioned BMQA, TextQA, and KBQA) are used to realize different question and answer capabilities. Different question and answer modules are good at different fields, have different skills, and can handle different types of problems. In terms of fields, some modules are good at encyclopedic knowledge, some modules are good at biological knowledge or literary knowledge. In terms of reasoning ability, some modules have common sense understanding and multi-hop reasoning ability, some modules have numerical reasoning and logical reasoning ability, and the final answer is selected from the candidate expert question and answer module answers according to the ability of different modules and the type of the question and returned to the user.

[0098] The aggregation reasoning module 4 inputs the answers and explanations output by each sub-question and answer module and the corresponding question, and outputs to the reply generation module. For the user input question “Who is longer, the Yangtze River or the Nile River?”, different expert modules give different candidate answers, module 1 gives the candidate answer “Nile River”, module 2 gives the candidate answer “Yangtze River”, and module 3 gives the candidate answer “panda”. We know that the answer of module 1 is correct, and the evidence it gives is also correct; while the evidence of module 2 is consistent with the fact, but its answer is wrong, and according to “the Yangtze River is the first long river in China”, it cannot be inferred that “the Yangtze River is longer than the Nile River”; while the answer given by module 3 is irrelevant. The aggregation reasoning module 4 takes the question and the candidate answers of different question and answer modules as input, and finally selects one of the candidate answers as the final answer, which is “Nile River” here.

[0099] Score fusion based answer selection: The aggregation reasoning module 4 makes decisions based on two parts of scores. The aggregation reasoning module 4 uses a score fusion mechanism to obtain the final score of the candidate answers and selects the final answer based on the large language model candidate answer score and the confidence score given by the question answering module itself. The first part of the score of the aggregation reasoning module 4 is based on the prediction of the large language model. The features of the question and the candidate answers are extracted using the large language model. In order to enable the large language model to focus on all the candidate answers given by the question answering module at the same time, the question and the candidate answers are concatenated into a character sequence and input into the large language model. In the process of extracting the features of the question and the candidate answers by the large language model, the large language model can focus on all the input sequences, combine the knowledge stored in the parameters of the large language model and the answers of different question answering modules for reasoning. After extracting the features of the question and the candidate answers using the large language model, we use a linear layer as an answer score regressor to obtain the score of each answer.

[0100] The second part of the score of the aggregation reasoning module 4 is composed of the confidence score given by the question answering module. This part of the score is given by the question answering module itself and represents the confidence estimate of the question answering module for the candidate answers it gives. By introducing this part of the score, we can effectively avoid the misjudgment caused by the over-reliance of the aggregation reasoning module 4 on the judgment of a single large language model. Since the first part of the large language model predicts the candidate answer score, it relies on a single model for prediction. In the training process of a single model, overfitting may occur, which reduces the generalization performance of the model. Moreover, it is difficult for a single model to have the reasoning ability of multiple question answering modules at the same time, and it is difficult to ensure that the model does not make errors in judging questions outside the training set in actual use. However, the confidence score given by each question answering module contains the judgment of the module itself on the candidate answers. On the one hand, from the perspective of the single model being prone to overfitting, the introduction of the confidence score given by the question answering module is equivalent to using different models for comprehensive judgment, which can help to alleviate the overfitting situation. On the other hand, from the perspective of the single model being difficult to have the reasoning ability of multiple question answering modules at the same time, the confidence score given by the question answering module is based on the reasoning ability of different modules themselves for judgment. The use of this score increases the reasoning ability of the aggregation reasoning module 4 to some extent.

[0101] Based on the two parts of scores, the candidate answer score given by the large language model and the confidence score given by the question answering module itself, we use a score fusion mechanism to obtain the final score, and select the candidate answer with the highest score as the final answer for output. Since the scores given by different question answering modules may not be consistent in scale and are not the same in distribution, we first linearly normalize the confidence score so that its score distribution is between 0 and 1. Let us have n question answering modules, and for the i-th (i = 1, 2,..., n) candidate answer, the candidate answer score given by the large language model is a i , and the normalized question answering module confidence score is b i , then we can obtain the final score s i of the i-th candidate answer by multiplying the two scores, s i = a i × b i , and finally we select the candidate answer with the highest final score as the final answer i answer = argmax i {s1, s2,..., s n}.

[0102] Optimization mechanism based on user feedback: In the user interaction interface of the question answering system, a user feedback channel is designed, so that users can give feedback on the effect of the current aggregation reasoning module 4 in the process of using the system, including whether the correct answer is included in the current question answering modules and whether the aggregation reasoning module 4 selects correctly. The system background collects these data, and when the data reaches a certain amount, the aggregation reasoning module 4 is further trained and updated, realizing online optimization based on user feedback.

[0103] Iterative comprehensive reasoning mechanism: An iterative comprehensive reasoning mechanism is constructed through the aggregation reasoning module 4 and each question answering module. The aggregation reasoning module 4 first receives the answers, explanations and confidence scores given by each question answering module, and selects the answer using the answer selection mechanism based on reasoning score fusion. At this time, the aggregation reasoning module 4 feeds back the selected answer to the remaining question answering modules and provides the corresponding explanation and confidence score to help other modules for further reasoning, and the question answering modules can update their own answers according to the new information obtained. Then, the aggregation reasoning module 4 selects the answer again using the answer selection mechanism based on score fusion according to the updated answer. If the answer does not change, the answer is output as the final selected answer, otherwise the iterative reasoning is continued according to the above method. Through the iterative comprehensive reasoning mechanism, each question answering module realizes the interaction of information and reasoning with the help of the aggregation reasoning module 4, further improving the accuracy of the question answering system.

[0104] In summary, the aggregation reasoning module 4 constructs a score fusion mechanism based on large language model prediction scores and question and answer module confidence scores, improves the robustness and accuracy of the aggregation reasoning module 4. An optimization mechanism based on user feedback is constructed, and the user feedback is used to optimize the system model effect during the running of the system. An iterative comprehensive reasoning mechanism is constructed, realizing the interaction of multi-module information and reasoning, giving full play to the reasoning ability of each module, and having multi-step reasoning ability.

[0105] As a preferred embodiment, the plurality of sub-question and answer modules comprises: a knowledge base question and answer module for reasoning to obtain candidate answers and explainability information of the candidate answers of the rewritten question based on a large-scale knowledge graph; a reading comprehension question and answer module for reasoning to obtain candidate answers and explainability information of the candidate answers of the rewritten question based on locally collected texts and online Internet retrieved texts; and a large language model question and answer module for reasoning to obtain candidate answers and explainability information of the candidate answers of the rewritten question based on linguistic knowledge, common sense knowledge and factual knowledge inside the large language model.

[0106] Reference is made to Figure 8 , Figure 8 The principle diagram of the knowledge base question and answer module based on semantic parsing and logical expression execution provided by the application is shown.

[0107] As a preferred embodiment, the knowledge base question and answer module comprises: a semantic parsing module based on entity linking and entity replacement, for identifying entities in the rewritten question and replacing the entities, and mapping the semantic parsing of the rewritten question into a logical expression executable by a logical expression execution module supporting neural-symbol hybrid reasoning; and the logical expression execution module supporting neural-symbol hybrid reasoning is used to query information from a structured knowledge base or an unstructured text library based on the logical expression to reason to obtain candidate answers and explainability information of the candidate answers of the rewritten question.

[0108] Specifically, the knowledge base question and answer module (KBQA) based on the structured knowledge base is mainly used to answer factual questions, including but not limited to: querying related knowledge of an object, comparing attributes of multiple objects, and screening objects meeting requirements (for example, as shown in Table 2 below). The system reasons to obtain answers based on logical expressions, can display intermediate process reasoning operations and reasoning results, and has good explainability. The system can perform accurate symbolic reasoning or neural-symbol hybrid reasoning based on logical expressions, and the accuracy of answer prediction is high.

[0109] Table 2 KBQA example

[0110]

[0111] The knowledge base question answering module takes the rewritten question obtained by the rewriting module as input, and constructs a semantic parser based on entity linking and large language models to generate a logical expression. The subsequent neural-symbolic reasoning module executes the expression to find relevant information from the knowledge base and reason to obtain an answer. Finally, the answer is input to the aggregation reasoning module 4. The KBQA module includes two sub-modules:

[0112] The semantic parsing module based on entity linking and entity replacement aims to map the question into an executable logical expression to guide the reasoning direction of the subsequent reasoning module. A "question-logical expression" pair is used for supervised training to obtain a semantic parser based on a "encoding-decoding" non-autoregressive generative large language model (such as T5). In the training stage, there may be a lack of training data. Here, we propose to augment the data through rule generation, model generation, and word sequence scrambling to form a large-scale training data to improve the performance of the semantic parser.

[0113] In the application stage, there may be entities in the input question that have not appeared in the training set. In order to improve the generalization ability of the semantic parser for new entities, we propose a semantic parsing method based on entity linking and entity replacement. First, the entities in the question are identified through entity linking, and then these entities are replaced with placeholders (such as the original question "When was the author of old man and sea born?" replaced with "When was the author of <placeholder-1>born?") ; the replaced question is mapped by the semantic parser into the logical expression "Find( <placeholder-1>"Find(The Old Man and the Sea)(Relate(author)(QueryAttr(date of birth)"; finally, the placeholders in the logical expression are restored to the target entities, obtaining the executable logical expression "Find(The Old Man and the Sea)(Relate(author)(QueryAttr(date of birth)".

[0114] The logic expression execution module supporting neural-symbolic hybrid reasoning (Neural-symbolic Executor) queries information from a structured knowledge base (such as Wikidata) or an unstructured text base (such as Wikipedia) based on the instructions of the logic expression, and executes each operation of the logic expression step by step. For example, the "Find(The Old Man and the Sea)" operation requires the executor to find the entity The Old Man and the Sea from the knowledge base, and the "Relate(author)" query queries the author of the target entity. Symbolic reasoning can directly find related information from the structured knowledge base, with high accuracy, but the recall rate of related information is low due to the low coverage of the structured knowledge base. In order to improve the recall rate of information and the generalization ability of the executor, neural reasoning attempts to infer related information from the unstructured text base, which covers a large amount of knowledge, so the recall rate of related information is high.

[0115] In addition, the KBQA module also designs a knowledge reasoning channel that fuses question-related entity subgraph retrieval (KBQuery), that is, KBQuery generation + subgraph retrieval + neural-symbolic reasoning. First, the rewritten question is input into the Program generation model to generate a subgraph query statement KBQuery, then the subgraph retrieval function is used from the knowledge graph base management module to obtain a knowledge subgraph from the knowledge graph, and finally the retrieved knowledge subgraph is converted into a sequence text through JSON or direct splicing, etc. Input into a large language model for knowledge reasoning, which can realize the fuzzy knowledge reasoning ability of the knowledge base question answering.

[0116] In summary, the knowledge base question answering module has an efficient data augmentation method: when the training samples are insufficient, a large amount of diversified data can be quickly constructed for training, improving the question answering ability and generalization ability of the model; entity linking and entity replacement: locating the entities in the question and replacing them with placeholders, and restoring the placeholders to the target entities after converting the question into a logical expression. This innovation can enhance the generalization ability of the model to new entities; neural-symbolic reasoning: this module combines the advantages of neural reasoning and symbolic reasoning. Symbolic reasoning has accuracy and explainability, and neural reasoning provides good generalization.

[0117] Please refer to Figure 9 , Figure 9 The principle schematic diagram of the reading comprehension module based on open domain knowledge reasoning provided by the present application.

[0118] Please refer to Figure 10 , Figure 10 The principle schematic diagram of the evidence retrieval module provided by the present application.

[0119] Please refer to Figure 11 , Figure 11 The principle schematic diagram of the knowledge reasoning model provided by the present application.

[0120] As a preferred embodiment, the reading comprehension question answering module comprises: an evidence retrieval module, used for retrieving local text data and Internet data to obtain evidence candidates; a knowledge reasoning model, used for reasoning to obtain a candidate answer of the rewritten question and explainability information of the candidate answer according to the evidence candidates.

[0121] Specifically, the reading comprehension question answering module mainly focuses on answering open domain factual questions, and has certain logical reasoning and numerical reasoning capabilities. The system obtains relevant unstructured text knowledge from the Internet and the domain-specific retrieval corpus according to the user's question, and obtains the answer to the question through the integration and reasoning of the knowledge. The characteristics of the system mainly reflect in two aspects: 1) the system can utilize massive unstructured text knowledge, which has the characteristics of low acquisition difficulty, wide coverage, fast updating speed and strong real-time performance, to support various open domain questions; 2) in addition, the logical reasoning and numerical reasoning modules equipped by the system can more fully integrate text information, further improve the utilization rate of knowledge, and support complex questions.

[0122] The reading comprehension question answering module takes the rewritten question obtained by the rewriting module as input, retrieves structured text knowledge related to the question, integrates information through the reasoning model (Reasoner), and generates the answer to the question. The TextQA module includes two sub-modules:

[0123] The evidence retrieval (Evidence Retrieval) module supports retrieving relevant text fragments from the domain-specific retrieval corpus collected locally and online Internet, which is used for subsequent information integration and reasoning. It mainly includes two parts: local text data retrieval and Internet data retrieval.

[0124] Local text data retrieval, using a hybrid method of sparse retrieval algorithm and dense retrieval. After using sparse retrieval (BM25) and dense vector retrieval (Dense Retriever) to get the candidate document set respectively, we merge the two candidate sets and add the sparse retrieval BM25 relevance score and the dense retrieval vector relevance score to get the final relevance score. Sparse retrieval algorithm has relatively stable performance in different fields, while dense retrieval algorithm can better capture semantics, so the combination of the two can better serve the knowledge retrieval needs of open domain question answering systems.

[0125] Here, dense vector retrieval uses large language model-based dense vector retrieval and re-ranking techniques: first, fine-tune a large language model (such as the dense passage retrieval algorithm Dense Passage Retriever (DPR), the unsupervised dense information retrieval algorithm Contriever, etc.) using supervised or unsupervised dense retrieval algorithms from training data. Here, the large language model fine-tuning uses a parameter-efficient fine-tuning method based on Delta Tuning. Dense vector retrieval performs coarse-grained evidence filtering based on the semantic vector similarity between the user's question and the candidate text, and the re-ranking model performs fine-grained evidence filtering based on the deep matching between the user's question and the retrieval text. This two-level evidence retrieval method from coarse to fine can efficiently and accurately obtain relevant text knowledge. The re-ranking model is a large language model fine-tuned on supervised data, also using Delta Tuning.

[0126] The core of dense vector retrieval is semantic vector matching. When the size of text data is large, it involves large-scale dense vector retrieval, but the storage and dot product calculation of semantic vectors pose great difficulties for model deployment. To solve this difficulty, we further use the Hierarchical Navigable SmallWorlds (HNSW) algorithm. This algorithm is a graph-based vector retrieval algorithm that organizes the graph into a hierarchical structure, similar nodes form high-level nodes, and nearest neighbor queries are performed in high-level nodes to quickly find similar nodes in the lower-level nodes.

[0127] Retrieval data merging. Merge the Internet retrieval text obtained through the search engine API and the local text data retrieval results to get the final query results. This result, as evidence candidates (Evidence Candidates), will be directly input to the downstream question answering model for reasoning and answer generation.

[0128] The knowledge reasoning model adopts an encoder-decoder framework, and the encoder is composed of a text representation module (Encoder), a logic reasoning module (Logic Reasoner), and a numeric reasoning module (Numeric Reasoner). The text representation module encodes the user question and the retrieved evidence to form an encoded representation matrix; the logic reasoning module and the numeric reasoning module perform reasoning based on the encoded representation matrix. The logic reasoning module adopts the structure of Transformer, and through multiple iterations in the continuous semantic space, it realizes implicit multi-hop logical reasoning, and the reasoning result is represented by a fixed-size matrix. The numeric reasoning module also adopts the structure of Transformer, and outputs a directed acyclic computation graph, where the nodes represent mathematical operations and the edges represent the dependency relationship between operations; the numerical results calculated by the calculator are further used to improve the node representation in the graph. Finally, the encoded representation matrix of the text representation module, the reasoning matrix of the logic reasoning module, and the graph representation matrix of the numeric reasoning module are used as the input of the decoder to generate the final answer.

[0129] In summary, the reading comprehension question and answer module constructs a logic reasoning module and a numeric reasoning module, realizes multi-step continuous semantic reasoning and discrete numeric reasoning, and enables the knowledge reasoning model to more fully utilize the retrieved unstructured text evidence to support complex user questions.

[0130] Please refer to Figure 12 , Figure 12 The principle schematic diagram of the large language model question and answer module based on knowledge detection and demonstration provided by the application.

[0131] As a preferred embodiment, the large language model question and answer module comprises: an answer generation module based on knowledge detection, which uses a non-autoregressive generative pre-training language model as a large language model, and performs answer generation task adaptation through a pluggable small number of parameter efficient learning technology; an explanation generation module based on demonstration, which is used for example construction and large language model explanation generation based on answer generation task adaptation, using an autoregressive generative large language model as a large language model to reason to obtain a candidate answer to a rewritten question and explainability information of the candidate answer; the large language model question and answer module is also used for judging whether the rewritten question is a preset stand question according to a preset stand question refutation mechanism and generating a large language model refutation explanation.

[0132] In particular, the large language model question and answer module mainly supports fuzzy reasoning and perceptual reasoning questions. It can solve the problems that the knowledge graph-based question and answer system and the reading comprehension-based question and answer system cannot solve, such as preset position problems, creative problems (such as writing poems and writing scripts), and viewpoint attitude type problems. The system activates the corpus by careful construction, so that the internal question and answer ability and knowledge ability of the large language model are effectively activated. In addition, the system is equipped with an internal explanation module, which gives a long explanation of the answer given by the large language model itself.

[0133] The large language model question and answer module takes the rewritten question obtained by the rewriting module (Rewriter) as input, takes the large language model as the knowledge source, generates answers and explanations, and gives the answer confidence, and then inputs it into the aggregation reasoning module 4. The BMQA module includes two sub-modules:

[0134] 1) Knowledge probing-based answer generation, taking a non-autoregressive generative PLM as the large language model (such as T5, UnifiedQA, etc.), and performing answer generation task adaptation (such as judging whether it is a preset position problem, answer generation, etc.) through Delta Tuning. Here, the central computing and management module 2 provides efficient computing support for the large language model and DeltaTuning. This sub-module is composed of a knowledge probe and a PLM. The knowledge probe is constructed by defining a prompt template, which changes the answer generation into a fill-in-the-blank question. For example, given the question "How many eyes does the sun have?" We can construct the template: "Question: {question} The answer to this question is _____", where question fills in the user question, and "___" is the model fill-in position.

[0135] 2) Demonstration-based explanation generation, taking an autoregressive generative PLM as the large language model (such as GPT series models, etc.), which includes example construction and large language model explanation generation. Example construction mainly uses Few-shot / Zero-shot example template to construct example input, such as "Example 1.... Example k. Question: {Question} Answer: {Answer} Explanation: ____".

[0136] Here k represents that there are k examples, each example is composed of a question, an answer, and an explanation, and Question and Answer represent the filled question and answer. For example: when k = 2, the example construction result is:

[0137] Question 1: Would Shakespeare use a computer? Answer 1: No, Explanation 1: Shakespeare lived during the Renaissance period, when computers had not yet been invented.

[0138] Question 2: Who is taller, Yao Ming or LeBron James? Answer 2: Yao Ming, Explanation 2: Yao Ming is 229 cm tall, while LeBron James is 203 cm tall.

[0139] Question 3: How many eyes does the sun have? Answer 3: None, Explanation 3: ______

[0140] It is worth noting that further combining Delta Tuning with the demonstration-based explanation generation task adaptation will help to further improve the performance of large language models in generating explanations, such as the ability to refute pre-set positions. Similarly, the central computing and management module 2 provides support for large language models and Delta Tuning.

[0141] Other functions: BMQA pre-set position refutation. Pre-set position questions mainly refer to questions that have incorrect assumptions about common sense, facts, and other knowledge, such as the question "How many eyes does the sun have?" which has a pre-set position "The sun has eyes." The question "What computer did Shakespeare use to write his plays?" has a pre-set position "Computers have been invented during Shakespeare's time." For such questions, GPT-3 with 175B parameters may also provide incorrect answers such as "two eyes" and "Apple computer," lacking the ability to refute explanations. To activate the BMQA's ability to refute pre-set positions, two steps are taken: 1) Pre-set position question judgment: Add a pre-set position judgment to the answer generation module; 2) Refutation explanation generation: Activate the large language model's ability to generate refutation explanations through pre-set position question and answer data sets. The two steps use Pareto multi-objective optimization + Delta Tuning + data playback + few-shot learning to achieve BMQA's ability to answer questions including pre-set position questions. We can construct a pre-set position question and answer data set for model training, as shown in Table 3:

[0142] Table 3: Pre-set position data set

[0143]

[0144] In summary, the large language model question and answer module constructs a BMQA technical framework based on knowledge exploration and demonstration, which involves a large language model and Delta fine-tuning and plug-in unified support through a central computing and management module 2, the module has strong scalability, and the task data is flexible. Preset stance question support. This module constructs a Pareto multi-objective optimization prediction + data playback method to realize the rebuttal explanation of the preset stance question and has common sense reasoning ability. Currently, large language models cannot well support such questions, such as OPT, GPT-3, T5-XXL, etc.

[0145] Please refer to Figure 13 , Figure 13 The large language model driven open domain natural language reasoning question and answer system interface schematic diagram provided by the application is one.

[0146] Please refer to Figure 14 , Figure 14 The large language model driven open domain natural language reasoning question and answer system interface schematic diagram provided by the application is two.

[0147] Please refer to Figure 15 , Figure 15 The large language model driven open domain natural language reasoning question and answer system interface schematic diagram provided by the application is three.

[0148] The application has the following beneficial effects:

[0149] (1) Support comprehensive problem types. Can simultaneously support factual knowledge questions, common sense knowledge questions, and text knowledge questions, and has certain multi-hop reasoning, numerical reasoning, and common sense reasoning capabilities. The open domain complex scene natural language reasoning intelligent question and answer system framework learns from the human brain language understanding mechanism, adopts a hybrid reasoning mechanism of symbolic reasoning and neural reasoning, and has the ability to answer questions from large-scale knowledge graphs, large-scale unstructured text, and large language models as knowledge sources. The system uses the existing technical framework in knowledge base question and answer and reading comprehension, and innovates the modules based thereon, specifically including: 1) TextQA's reasoning device with logical reasoning and numerical reasoning; 2) BMQA's preset stance rebuttal mechanism; 3) Feedback aggregation reasoning; 4) Program generalization mechanism (entity replacement, data augmentation, neural-symbolic reasoning reinforcement learning) in KBQA.

[0150] (2) Strong versatility. The large language model is used as the system base model throughout the entire system, which has good prediction performance and strong adaptability to complex scenes. In KBQA program generation, TextQA dense retrieval, reordering, reading comprehension reasoning device, large language model question and answer, and large language model explanation generation, the large language model is used.

[0151] (3) Strong scalability. The system designs a central computing and management platform to address the problems of large hardware memory and GPU requirements, and difficulty in domain adaptation caused by large language models and large-scale knowledge graphs. It proposes a large language model scheduling and adaptation technology based on pluggable few-parameter efficient fine-tuning, supporting dynamic adaptation and flexible invocation of models.

[0152] (4) Strong interpretability. The system uses modular design, program execution visualization, evidence display, answer explanation, and other methods to realize explanation functions such as problem analysis and answer analysis. It gives a credibility score to the answer output of each module to support users in selecting reliable answers. KoPL is used to display the inference process, evidence recall shows the source of the problem answer, and demonstrates the explanation of the large language model answer.

[0153] (5) Reject and refute ability. In large language model question answering, a method is provided to activate the refutation ability of the large language model by presetting the position data, i.e., the Pareto multi-objective optimization + Delta Tuning + data playback + few-shot learning preset position question refutation ability activation strategy.

[0154] Please refer to Figure 16 , Figure 16 The flowchart of the large language model driven open domain natural language reasoning question answering method provided by the application is shown.

[0155] The application also provides a large language model driven open domain natural language reasoning question answering method, which is applicable to the above-mentioned large language model driven open domain natural language reasoning question answering system, comprising:

[0156] 1601: The problem rewriting module 1 rewrites the user question to obtain a rewritten question;

[0157] 1602: The central computing and management module 2 outputs the rewritten question and the computing and knowledge resources of the large language model required by the problem core engine module to one or more sub-question answering modules in the question answering core engine module 3 according to the type of the rewritten question;

[0158] 1603: The question answering core engine module 3 reasons to obtain one or more candidate answers to the rewritten question and the explainability information of the candidate answers according to the rewritten question and the computing and knowledge resources of the large language model;

[0159] 1604: The aggregation reasoning module 4 aggregates the reasoning to obtain the final answer to the rewritten question and the explainability information of the final answer according to the candidate answers to one or more rewritten questions and the explainability information of the candidate answers.

[0160] For the introduction of the large language model driven open domain natural language reasoning question and answer method provided by the application, please refer to the above system embodiment, which will not be described hereinafter.

[0161] Finally, it should be pointed out that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A large language model driven open domain natural language reasoning question answering system, characterized in that, The system comprises a question rewriting module, a central computing and management module, a question and answer core engine module, and an aggregation reasoning module; the question and answer core engine module comprises a plurality of sub-question and answer modules, and the question reasoning logics of the plurality of sub-question and answer modules are different; an output end of the question rewriting module is connected with an input end of the central computing and management module, and the question rewriting module is configured to rewrite a user question to obtain a rewritten question; an output end of the central computing and management module is connected with an input end of the question and answer core engine module, and the central computing and management module is configured to manage computing and knowledge resources of the large language model, and output the rewritten question and the computing and knowledge resources of the large language model required by the question and answer core engine module to one or more sub-question and answer modules in the question and answer core engine module according to a type of the rewritten question; an output end of the question and answer core engine module is connected with an input end of the aggregation reasoning module, and the question and answer core engine module is configured to reason to obtain a candidate answer of one or more rewritten questions and explainability information of the candidate answer according to the rewritten question and the computing and knowledge resources of the large language model; the aggregation reasoning module is configured to aggregate and reason to obtain a final answer of the rewritten question and explainability information of the final answer according to the candidate answer of one or more rewritten questions and the explainability information of the candidate answer; the plurality of sub-question and answer modules comprise a knowledge base question and answer module configured to reason to obtain the candidate answer of the rewritten question and the explainability information of the candidate answer according to a large-scale knowledge graph; the knowledge base question and answer module comprises: a semantic analysis module based on entity linking and entity replacement, configured to identify an entity in the rewritten question, replace the entity, and map semantic analysis of the rewritten question to a logic expression executable by a logic expression execution module supporting neural-symbol hybrid reasoning; the logic expression execution module supporting neural-symbol hybrid reasoning is configured to query information from a structured knowledge base or an unstructured text library based on the logic expression to reason to obtain the candidate answer of the rewritten question and the explainability information of the candidate answer.

2. The large language model driven open-domain natural language reasoning question answering system according to claim 1, characterized in that, Further comprising: a reply generation module connected with an output end of the aggregation reasoning module, configured to convert the final answer of the rewritten question and the explainability information of the final answer into a machine reply to obtain a reply answer of the user question and explainability information of the reply answer; a dialogue interaction module connected with the reply generation module, configured to perform dialogue interaction with the user question according to the reply answer of the user question and the explainability information of the reply answer.

3. The large language model driven open-domain natural language reasoning question answering system according to claim 1, characterized in that, The question rewriting module is specifically configured to perform anaphora resolution or ellipsis resolution on the user question according to historical question and answer data to obtain a rewritten question, and not trigger a rewriting prompt when the rewritten question is the same as the user question, and trigger the rewriting prompt when the rewritten question is different from the user question, for user selection.

4. The large language model driven open-domain natural language reasoning question answering system according to claim 1, characterized in that, The central computing and management module comprises: a knowledge graph base management module configured to provide knowledge query and reasoning execution of a knowledge graph base; A text resource library management module is configured to provide text query support for a local text resource library; A large language model calculation and management module is configured to provide model calculation and task adaptation support for a large language model library and a fine-tuning parameter library for large language model task adaptation; A management hub module is configured to schedule demands of a plurality of sub-question answering modules for the knowledge graph library, the local text resource library, the large language model library, and the fine-tuning parameter library for large language model task adaptation, and select one or more sub-question answering modules from the plurality of sub-question answering modules to reason on the rewritten question according to the type of the rewritten question, to obtain a candidate answer to the rewritten question and explainability information of the candidate answer.

5. The large language model driven open-domain natural language reasoning question answering system according to claim 1, wherein, The aggregation reasoning module is specifically configured to aggregate and reason on the candidate answer to the rewritten question and the explainability information of the candidate answer based on a score fusion mechanism, an optimization mechanism of user feedback, and an iterative comprehensive reasoning mechanism, to obtain a final answer to the rewritten question and explainability information of the final answer.

6. The large language model driven open-domain natural language reasoning question answering system according to any one of claims 1 to 5, characterized in that, The plurality of sub-question answering modules include: A reading comprehension question answering module is configured to reason on a candidate answer to the rewritten question and explainability information of the candidate answer based on locally collected text and / or online Internet retrieved text; A large language model question answering module is configured to reason on a candidate answer to the rewritten question and explainability information of the candidate answer based on linguistic knowledge, common sense knowledge, and factual knowledge inside a large language model.

7. The large language model driven open-domain natural language reasoning question answering system according to claim 6, characterized in that, The reading comprehension question answering module includes: An evidence retrieval module is configured to retrieve local text data and Internet data to obtain evidence candidates; A knowledge reasoning model is configured to reason on a candidate answer to the rewritten question and explainability information of the candidate answer based on the evidence candidates.

8. The large language model driven open-domain natural language reasoning question answering system according to claim 6, characterized in that, The large language model question answering module includes: An answer generation module based on knowledge exploration is configured to use a non-autoregressive generative pre-training language model as a large language model, and perform answer generation task adaptation through a pluggable small number of parameter efficient learning technology; An explanation generation module based on demonstration is configured to perform example construction and large language model explanation generation based on the answer generation task adaptation, using a generative large language model as a large language model, to reason on a candidate answer to the rewritten question and explainability information of the candidate answer; The large language model question answering module is further configured to determine whether the rewritten question is a preset stance question according to a preset stance refutation mechanism, and generate a large language model refutation explanation.

9. An open domain natural language reasoning question answering method driven by a large language model, characterized in that, The method is applicable to the large language model driven open domain natural language reasoning question answering system as claimed in any one of claims 1 to 8, and includes: The question rewriting module rewrites a user question to obtain a rewritten question; The central calculation and management module outputs the rewritten question and calculation and knowledge resources of the large language model required by the question answering core engine module to one or more sub-question answering modules in the question answering core engine module according to the type of the rewritten question. The central calculation and management module outputs the rewritten question and calculation and knowledge resources of the large language model required by the question answering core engine module to one or more sub-question answering modules in the question answering core engine module according to the type of the rewritten question. The question and answer core engine module obtains one or more candidate answers to the rewritten question and explainability information of the candidate answers according to calculation and knowledge resource reasoning of the rewritten question and the large language model; The aggregation reasoning module obtains a final answer to the rewritten question and explainability information of the final answer according to the candidate answers to the rewritten question and the explainability information of the candidate answers.

Images