Credible question and answer method based on knowledge verification

Abstract

The invention discloses a credible question and answer method based on knowledge verification, which comprises the following steps: identifying and eliminating anaphora and omission contents in a question of a user, relying the question of the user, and extracting to obtain a keyword expressing the intention of the user. And taking the re-described keyword set as a query, retrieving related articles, calculating the correlation score of the retrieved content and questions, and screening out the multi-modal knowledge with the highest correlation score. And generating a multi-modal statement about a retrieval result, and performing multi-round and multi-path verification and reasoning on the multi-modal statement. And carrying out authenticity verification on the iterated multi-mode statement, and integrating filtered and verified knowledge and evidence to generate a final trusted user reply. Compared with the prior art, the method has the advantages that anaphora and omission can be eliminated by injecting associated historical context information in the question characterization process, and credible multi-modal knowledge can be obtained by utilizing retrieval enhancement and a self-checking mechanism, so that questions proposed by users can be answered more accurately.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a credible question-answering method based on knowledge verification. Background Art

[0002] With the rapid development of the Internet and artificial intelligence technology, the "Internet + traditional industry" model has become increasingly important in various fields. Taking the tourism industry as an example, the country has vigorously developed "Internet + tourism" and used cutting-edge technologies such as big data and deep learning to innovate the tourism management model. This has become an important trend in the development of tourism in my country. In the past, when tourists visited scenic spots during peak tourist seasons, they often needed to obtain information through manual consultation. This kind of information acquisition is inefficient and untimely. In order to solve this problem, the intelligent question-and-answer system came into being, which improves the efficiency of consultation by allowing machines to automatically answer tourists' questions. Compared with traditional search engines, the question-and-answer system can directly provide answers without users having to search through lengthy search lists. This can significantly improve the efficiency of information retrieval and provide users with a better travel experience. This technology is of great significance for the further implementation of smart tourism and digital tourism.

[0003] Users' questions often involve information of different dimensions, multiple modes and levels of scenic spots, which requires the question-answering model to have an in-depth understanding of the information. Traditional methods usually only focus on the content of the text to generate answers, ignoring the domain knowledge of multiple modes involved behind the text, resulting in one-sided judgment and reasoning. To obtain domain knowledge, a common strategy is to build a domain-related knowledge graph. However, the construction process relies on expert knowledge, which is costly, and the knowledge coverage is low, and the knowledge may become outdated. Another idea is to use large language models (LLMs) as domain knowledge bases. After pre-training on large-scale Internet data, LLMs record rich domain knowledge. By using prompt learning technology, LLMs can be guided to output knowledge related to the question. However, LLMs have hallucination problems. In addition, there are a large number of low-quality articles containing false content on the Internet, which can easily aggravate this lack of hallucination. Existing question-answering models lack a knowledge verification mechanism and are prone to generate unreliable results, causing losses at multiple levels, such as replying to tourists based on malicious negative reviews, which damages the image of the scenic spot. With the rapid development of automatic generation technologies such as ChatGPT and Stable Diffusion, it has become easier to create fake articles. Traditional manual review mechanisms are no longer able to cope with the massive amount of articles automatically created by machines. Therefore, trusted question-answering technology can enhance the reliability of output results, which is of great significance and value in purifying the network environment, restoring the credibility of the media, and improving the efficiency of information acquisition.

[0004] Traditional methods retrieve relevant articles from the Internet or dedicated databases to answer questions. Due to outdatedness, malicious fabrication, etc., the retrieved content may be false, resulting in incorrect answers that mislead users. In addition, these methods only consider text content, ignoring the rich knowledge contained in other modal content such as images, which can easily lead to one-sided answers. In real application scenarios, users generally learn about scenic spots through multiple rounds of dialogues. User questions often refer to content in historical dialogues, but this reference may not be obvious or may even be omitted. Existing methods mainly focus on single-round question and answering, ignoring this implicit contextual association when representing questions, and it is difficult to effectively combine historical content to form an accurate response.

[0005] The prior art discloses a method and system for reducing the illusion of a large language model by verifying an external knowledge base, which relates to the technical field of large language models. The method comprises the following steps: obtaining text information output by a large language model and converting it into structured fact triple data; generating questions based on the structured fact triple data; collecting answers for verifying the fact triple data based on a retrieval system and verifying them; and revising the fact triple data based on the verification results. The drawback of this solution is that relevant articles are retrieved from an external dedicated database to answer questions. Due to reasons such as being outdated or maliciously fabricated, the retrieved content may be false, resulting in the generation of wrong answers that mislead users.

[0006] To this end, in combination with the above requirements and the defects of the existing technology, this application proposes a trusted question-answering method based on knowledge verification. Summary of the invention

[0007] The present invention provides a credible question-answering method based on knowledge verification, which can integrate historical context information to resolve reference and omission, and use retrieval enhancement and self-verification mechanisms to obtain multimodal knowledge with high confidence, so as to answer users' questions more comprehensively.

[0008] The primary purpose of the present invention is to solve the above technical problems. The technical solution of the present invention is as follows:

[0009] A first aspect of the present invention provides a trusted question-answering method based on knowledge verification, the method comprising the following steps:

[0010] S1. Combine the context of the user's historical conversations to identify and eliminate references and omitted content in the user's questions, use the pre-trained model to restate the user's questions, and extract keywords that express the user's intentions.

[0011] S2. Use the keyword set in the restated question as a query to retrieve relevant articles from the external knowledge base and large language model, calculate the relevance score between the retrieved content and the question, and screen out the multimodal knowledge with the highest relevance score.

[0012] S3. Generate a multimodal statement about the retrieval result, and perform multi-round, multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached.

[0013] S4. Verify the authenticity of the iterated multimodal statements, construct verification questions based on the fact-checking explanations generated by the large language model, generate answers using retrieval enhancement technology, and verify the preliminary verification results through correction check prompts.

[0014] S5. Integrate the filtered and verified knowledge and evidence to generate the final credible user response.

[0015] Furthermore, in step S1, the identification and elimination of references and omitted contents in the user's question specifically includes: obtaining concepts or pronouns in the historical conversation context related to the question, and performing synonymous replacement on omitted contents or unclear references in the user's question according to the association between the pronouns and the concepts.

[0016] Furthermore, the pre-trained model M is used Q Before restating, the pre-trained model needs to be fine-tuned, and the fine-tuning process is guided by optimizing the following objective function:

[0017]

[0018] Among them, C i-1 Indicates the context information of the previous text, q i is the original question, q′ i is a new question after restating, by maximizing M Q In C i-1 and q i Under the condition of generating q′ i The probability of completing the optimization objective function; using a large language model To extract the keywords, the model uses the following optimization objective function:

[0019]

[0020] Among them, K i Represents the extracted keyword set, by maximizing C i-1 ,q i and q′ i Generate K for the condition i The probability of training

[0021] Furthermore, step S2 includes the following specific steps: using the keyword set K i As a query, obtain the relevant knowledge set {D1,D2,…,D M}, each retrieved article D is decomposed into a set of paragraphs and illustrations {p1,p2,…,p |D|}, where for any paragraph and illustration p j Calculate q′ i and K i The relevance score of ij , the specific formula is as follows:

[0022]

[0023] Where E(·) represents the embedding function based on images and texts. The multimodal embedding model is used to transform images and texts into a unified high-dimensional vector representation. f(·) is the scoring function, which measures the similarity by calculating the cosine value between two embedded vectors.

[0024] Further, the correlation score s is calculated ij After that, the selected relevant texts and pictures are re-ranked, and the multimodal embedding model is used to q′ i and p j After characterization and re-ranking, the top n candidate knowledge are selected, denoted as P→p, and used as the input of the reply generator.

[0025] Furthermore, step S3 includes the following specific steps: generating a statement about the search result according to each search result and the corresponding publication date, and verifying the authenticity of the statement about the search result by using the knowledge search chain and the knowledge search tree. First, based on the rewritten user question q′ i and multimodal context information, generate an initial question Q, use the retrieval engine to obtain evidence related to the question from the external knowledge base, and generate an answer A to determine whether the current answer is sufficient to verify the authenticity of the original statement; if not, generate a new follow-up question Q′ based on the existing question-answer pair (Q, A), continue to search and answer, otherwise, end the search; iterate repeatedly to form a continuous question-answer chain {(Q1, A1), (Q2, A2), …, (Q n ,A n )} until the preset maximum number of iterations is reached.

[0026] Furthermore, the specific process of the knowledge retrieval chain is as follows: Input the rewritten question q′ i , multimodal context information, according to q′ iand context, generate the initial question Q, question-answer pair sequence QAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, use the retrieval engine to obtain evidence related to the current question Q, generate answer A, add (Q, A) to QAPairs, and determine whether to generate subsequent questions based on Q and A. If necessary, generate a new question Q′ and update Q←Q′; otherwise, set followUpNeeded = False, update the counter counter+ = 1, and output the question-answer pair sequence QAPairs.

[0027] Furthermore, the specific process of the knowledge retrieval tree is as follows: input the rewritten question q′ i , multimodal context information, according to q′ i and context, generate the initial question set {Q1, Q2, Q3}, initialize the best question-answer pair sequence BestQAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, for each question Q j , use the search engine to obtain relevant evidence and generate answer A j , forming a question-answer pair (Q j ,A j ), for all (Q j ,A j ) to evaluate and select the best question-answer pair (Q b ,A b ), add it to BestQAPairs, and determine whether to generate follow-up questions based on BestQAPairs. If necessary, based on (Q b ,A b ) Generate a new question set {Q′1, Q′2, Q′3}; otherwise, set followUpNeeded=False, update the counter counter+=1, and output the question-answer pair sequence BestQAPairs.

[0028] Furthermore, the step S4 specifically includes the following steps: generating an accuracy label supporting or refuting the statement based on the retrieved question-answer pair information; generating a failure label if the information is insufficient for judgment; constructing a verification question based on the fact-checking explanation generated by the large language model, generating an answer using retrieval enhancement technology, and verifying the preliminary verification result through correction check prompts; the step S5 specifically includes: integrating the comprehensive information provided by step S2, the reliable evidence confirmed by steps S3 and S4, and the contextual information in the historical conversation, and outputting a final, trustworthy user response.

[0029] The second aspect of the present invention provides a trusted question-answering system based on knowledge verification, which is used in the trusted question-answering method based on knowledge verification, and includes: a question analysis unit, a multimodal knowledge retrieval unit, and a trusted response unit.

[0030] The question analysis unit identifies and eliminates references and omitted contents in the user's question in combination with the context of the user's historical conversation, uses a pre-trained model to restate the user's question, and extracts keywords that express the user's intention; the multimodal knowledge retrieval unit uses the keyword set in the restated question as a query, retrieves relevant articles from an external knowledge base and a large language model, calculates the relevance score between the retrieved content and the question, and screens out the multimodal knowledge with the highest relevance score; the credible response unit includes a multimodal statement generation module, a retrieval enhancement reasoning module, and an authenticity verification module. The multimodal statement generation module generates a multimodal statement about the retrieval result. The retrieval enhancement reasoning module performs multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached. The authenticity verification module verifies the authenticity of the iterated multimodal statement, constructs a verification question based on the fact-checking explanation generated by the large language model, generates an answer using the retrieval enhancement technology, and verifies the preliminary verification result through a correction check prompt. The credible response unit integrates the filtered and verified knowledge and evidence to generate a final credible user response.

[0031] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0032] The present invention provides a trusted question-answering method based on knowledge verification, which can eliminate reference and omission by injecting relevant historical context information into the question representation process, and obtain trusted multimodal knowledge by utilizing retrieval enhancement and self-verification mechanisms, combined with extracting key multimodal knowledge from retrieval results, to alleviate the hallucination problem of large models and guide them to generate more trusted replies, thereby more accurately answering questions raised by users. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flow chart of a trusted question-answering method based on knowledge verification.

[0034] Figure 2 The present invention is a flowchart of question restatement, fine-grained retrieval and trusted answering in one embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of a trusted question-answering system based on knowledge verification according to the present invention. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0038] Example 1

[0039] like Figure 1 As shown, the present invention provides a trusted question-answering method based on knowledge verification, and the method comprises the following steps:

[0040] S1. Combine the context of the user's historical conversations to identify and eliminate references and omitted content in the user's questions, use the pre-trained model to restate the user's questions, and extract keywords that express the user's intentions.

[0041] S2. Use the keyword set in the restated question as a query to retrieve relevant articles from the external knowledge base and large language model, calculate the relevance score between the retrieved content and the question, and screen out the multimodal knowledge with the highest relevance score.

[0042] S3. Generate a multimodal statement about the retrieval result, and perform multi-round, multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached.

[0043] S4. Verify the authenticity of the iterated multimodal statements, construct verification questions based on the fact-checking explanations generated by the large language model, generate answers using retrieval enhancement technology, and verify the preliminary verification results through correction check prompts.

[0044] S5. Integrate the filtered and verified knowledge and evidence to generate the final credible user response.

[0045] According to the above technical features, the present invention can integrate relevant historical context information and introduce a self-verification mechanism to provide users with highly reliable and trustworthy answers. Specifically, the current question is first rewritten according to the conversation history, the referents of the pronouns in the question are clarified and the omitted content is supplemented, so as to understand the user's query content more finely. Subsequently, a fine-grained retriever is designed to extract articles related to the rewritten question from the external knowledge base and the large model LLMs. Next, a self-verification mechanism is developed to filter out potential false retrieval content. Unlike traditional unimodal methods, this mechanism combines multimodal content such as text and images for cross-modal reasoning, which can more comprehensively evaluate the confidence of the retrieval results. By injecting these credible retrieval results into the large model, the problem of period hallucination can be alleviated, allowing it to extract key multimodal clues from them, thereby generating reliable answers to the questions.

[0046] The present invention aims to provide users with fast and reliable consulting services, which can be applied in the field of tourism to efficiently and concisely answer users' questions about tourist attractions. In practical applications, it can help tourism platforms to more comprehensively integrate multimodal knowledge (such as images and texts) to meet users' information acquisition needs, and at the same time help companies analyze tourism behavior, thereby improving user experience and operational results.

[0047] It should be noted that, compared with the prior art, the present invention has the following advantages:

[0048] By injecting relevant historical conversation context information and using a fine-tuned large language model to restate the user's current question, eliminating references and omissions, we can more accurately understand the user's true intentions. This process not only ensures the accuracy of the question semantics, but also optimizes the subsequent knowledge retrieval steps through keyword extraction, thus laying a solid foundation for generating high-quality answers.

[0049] A fine-grained retriever is designed to extract text and image content that is highly relevant to the restated question from external knowledge bases and large language models. The retrieval content is uniformly represented through a multimodal embedding model, and the most relevant paragraphs and images are screened out using a re-ranking mechanism. This not only enriches the knowledge base of the question-answering system, but also ensures the comprehensiveness and accuracy of the information obtained, providing multi-dimensional support for generating credible answers.

[0050] In order to alleviate the "hallucination" problem of large language models, a self-verification mechanism is introduced to filter out low-quality content containing false information through a multimodal fact-checking module. Reasoning methods such as knowledge retrieval chains and retrieval trees are used to generate and verify multiple rounds of question-answer pairs, combined with standard authenticity prompts and verification chain accuracy prediction prompts to ensure that the generated answers have higher accuracy and credibility.

[0051] Example 2

[0052] Based on the above-mentioned embodiment 1, this embodiment describes in detail the specific process of the present invention.

[0053] like Figure 1 As shown, further, in step S1, the identification and elimination of references and omitted contents in user questions are specifically as follows: concepts or pronouns in the historical conversation context related to the question are obtained, and the omitted contents or unclear references in the user questions are replaced with synonyms according to the association between the pronouns and the concepts.

[0054] Furthermore, the pre-trained model M is used Q Before restating, the pre-trained model needs to be fine-tuned, and the fine-tuning process is guided by optimizing the following objective function:

[0055]

[0056] Among them, C i-1 Indicates the context information of the previous text, q i is the original question, q′ i is a new question after restating, by maximizing M Q In C i-1 and q i Under the condition of generating q′ i The probability of completing the optimization objective function; using a large language model To extract the keywords, the model uses the following optimization objective function:

[0057]

[0058] Among them, K i Represents the extracted keyword set, by maximizing C i-1 ,q i and q′ i Generate K for the condition i The probability of training

[0059] Furthermore, step S2 includes the following specific steps: using the keyword set K i As a query, obtain the relevant knowledge set {D1,D2,…,D M}, each retrieved article D is decomposed into a set of paragraphs and illustrations {p1,p2,…,p |D|}, where for any paragraph and illustration p j Calculate q′ i and K i The relevance score of ij , the specific formula is as follows:

[0060]

[0061] Where E(·) represents the embedding function based on images and texts. The multimodal embedding model is used to transform images and texts into a unified high-dimensional vector representation. f(·) is the scoring function, which measures the similarity by calculating the cosine value between two embedded vectors.

[0062] Further, the correlation score s is calculated ij After that, the relevant texts and pictures that have been filtered out are re-

[0063] New ranking, the multimodal embedding model used for q i and p j After characterization and re-ranking, the top n candidate knowledge are selected, denoted as P→p, and used as the input of the reply generator.

[0064] Furthermore, step S3 includes the following specific steps: generating a statement about the search result according to each search result and the corresponding publication date, and verifying the authenticity of the statement about the search result by using the knowledge search chain and the knowledge search tree. First, based on the rewritten user question q′ i and multimodal context information, generate an initial question Q, use the retrieval engine to obtain evidence related to the question from the external knowledge base, and generate an answer A to determine whether the current answer is sufficient to verify the authenticity of the original statement; if not, generate a new follow-up question Q′ based on the existing question-answer pair (Q, A), continue to search and answer, otherwise, end the search; iterate repeatedly to form a continuous question-answer chain {(Q1, A1), (Q2, A2), …, (Q n ,A n )} until the preset maximum number of iterations is reached.

[0065] Furthermore, the specific process of the knowledge retrieval chain is as follows: Input the rewritten question q′ i , multimodal context information, according to q′ i and context, generate the initial question Q, question-answer pair sequence QAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, use the retrieval engine to obtain evidence related to the current question Q, generate answer A, add (Q, A) to QAPairs, and determine whether to generate subsequent questions based on Q and A. If necessary, generate a new question Q′ and update Q←Q′; otherwise, set followUpNeeded = False, update the counter counter+ = 1, and output the question-answer pair sequence QAPairs.

[0066] Furthermore, the specific process of the knowledge retrieval tree is as follows: input the rewritten question q′ i , multimodal context information, according to q′ i and context, generate the initial question set {Q1, Q2, Q3}, initialize the best question-answer pair sequence BestQAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, for each question Q j , use the search engine to obtain relevant evidence and generate answer A j , forming a question-answer pair (Q j ,A j ), for all (Q j ,A j ) to evaluate and select the best question-answer pair (Q b ,A b ), add it to BestQAPairs, and determine whether to generate follow-up questions based on BestQAPairs. If necessary, based on (Q b ,A b ) Generate a new question set {Q′1, Q′2, Q′3}; otherwise, set followUpNeeded=False, update the counter counter+=1, and output the question-answer pair sequence BestQAPairs.

[0067] Furthermore, the step S4 specifically includes the following steps: generating an accuracy label supporting or refuting the statement based on the retrieved question-answer pair information; generating a failure label if the information is insufficient for judgment; constructing a verification question based on the fact-checking explanation generated by the large language model, generating an answer using retrieval enhancement technology, and verifying the preliminary verification result through correction check prompts; the step S5 specifically includes: integrating the comprehensive information provided by step S2, the reliable evidence confirmed by steps S3 and S4, and the contextual information in the historical conversation, and outputting a final, trustworthy user response.

[0068] Example 3

[0069] like Figure 2-Figure 3 As shown, the present invention also provides a trusted question-answering system based on knowledge verification, which is used for the trusted question-answering method based on knowledge verification, and includes: a question analysis unit 101, a multimodal knowledge retrieval unit 102, and a trusted response unit 103.

[0070] The question analysis unit 101 identifies and eliminates references and omitted contents in the user's question in combination with the context of the user's historical conversation, uses a pre-trained model to restate the user's question, and extracts keywords that express the user's intention; the multimodal knowledge retrieval unit 102 uses the keyword set in the restated question as a query, retrieves relevant articles from an external knowledge base and a large language model, calculates the relevance score between the retrieved content and the question, and screens out the multimodal knowledge with the highest relevance score; the credible response unit 103 includes a multimodal statement generation module, a retrieval enhancement reasoning module, and an authenticity verification module. The multimodal statement generation module generates a multimodal statement about the retrieval result. The retrieval enhancement reasoning module performs multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached. The authenticity verification module verifies the authenticity of the iterated multimodal statement, constructs a verification question based on the fact-checking explanation generated by the large language model, generates an answer using the retrieval enhancement technology, and verifies the preliminary verification result through a correction check prompt. The credible response unit integrates the filtered and verified knowledge and evidence to generate a final credible user response.

[0071] In a specific embodiment, Figure 2 As shown, the contents of each unit and module are introduced in detail.

[0072] Question analysis unit 101: This unit aims to analyze the details of the user's question, eliminate references in the question, complete omitted content based on the conversation history context related to the question, and thus more accurately understand the user's intention. In practical applications, the user's question is usually related to the concepts or pronouns mentioned above, such as Figure 2 As shown in the figure, "this scenic spot" actually refers to "Hangzhou West Lake". This unclear reference will confuse the model with the questioner's intention. Therefore, the question needs to be restated to make the questioner's intention clearer. Figure 2 As shown in the figure, when the user asks "What are the must-see attractions in this scenic area?", the intention is "What are the must-see attractions in West Lake, Hangzhou?" In order to let the machine accurately understand this intention literally, we replace the unclear part of the original question "this scenic area" with "West Lake, Hangzhou" by referring to the context, so as to make the question clearer and more specific.

[0073] Multimodal knowledge retrieval unit 102: This unit aims to use the external knowledge base to retrieve relevant and credible knowledge to eliminate the illusion problem of LLM, thereby improving the reliability of the answer. Specifically, this patent first uses the keyword set K i As a query, relevant articles are searched through the Google search engine. The goal of this process is to preliminarily screen a set of potentially relevant knowledge sets from a wide range of Internet resources, denoted as {D1, D2, …, D M}.like Figure 2 As shown, it can convert pictures and texts into a unified high-dimensional vector representation, has good multimodal feature representation capabilities, and is suitable for text, images, videos, audio, and even thermal data and gyroscope data.

[0074] Trusted response generation unit 103: This unit aims to filter out low-quality retrieval results containing false content through a self-checking mechanism, thereby alleviating the “hallucination” problem of large model LLMs, such as Figure 2 As shown, there are three modules in total, among which the multimodal statement generation module generates a statement about the search results based on each search result and the corresponding release date. In a specific embodiment, GPT-4V is used to represent multimodal content, which has a strong text and image understanding ability. The generated new statement not only contains the original retrieved text content, but also incorporates relevant detailed information extracted from the image. This can obtain a wider range of background contexts to provide the model with sufficient basis for judgment.

[0075] The retrieval-enhanced reasoning module aims to combine the retrieval-enhanced reasoning strategy to deeply verify the authenticity of the multimodal statements generated by the previous modules, thereby improving the credibility and accuracy of the generated responses. To this end, two novel reasoning methods, namely "knowledge retrieval chain" and "knowledge retrieval tree", are introduced to achieve multi-round, multi-path verification and reasoning of user questions. In this process, a reasoning strategy based on continuous question-answering chain is adopted. The knowledge retrieval tree algorithm process firstly calculates the rewritten user question q′ i and multimodal context information to generate an initial question Q, which is aimed at the key elements in the question. Then, the retrieval engine is used to obtain evidence related to the question from the external knowledge base and generate an answer A. Next, the model determines whether the current answer is sufficient to verify the authenticity of the original statement; if not, a new follow-up question Q′ is generated based on the existing question-answer pair (Q, A) to continue searching and answering. This process is iterated repeatedly to form a continuous question-answer chain {(Q1, A1), (Q2, A2),…, (Q n ,A n )} until the model believes that sufficient evidence has been obtained or the preset maximum number of iterations has been reached. The core of the knowledge retrieval chain method is to guide the model to gradually generate follow-up questions with strong relevance, and continuously enrich and verify the understanding of the original question through retrieval-enhanced answers. This iterative process enables the model to deeply explore useful information in the external knowledge base, reduce the illusion phenomenon caused by insufficient information, and improve the accuracy and reliability of the response.

[0076] However, the knowledge retrieval chain method may easily miss some important verification information because it only focuses on a single reasoning path, resulting in insufficient comprehensive verification of complex statements. Therefore, in order to improve this problem, we also introduced the knowledge retrieval tree method based on the knowledge retrieval chain. In the knowledge retrieval tree method, in order to further improve the verification ability of the model, we introduced a multi-branch question-answering strategy in each reasoning step to form a tree-like reasoning structure. The knowledge retrieval tree algorithm step will simultaneously generate multiple different sets of follow-up questions {Q1, Q2, Q3} at each step to verify the questions from different angles. For each generated question Q j , use the search engine to obtain relevant evidence and generate answer A j Subsequently, the model performs j ,A j ) to evaluate the original statement and select the question-answer pair (Q b ,A b ) as candidate evidence. Then, based on (Q b ,A b ) Generate new follow-up questions, repeat the above process, and form a tree-like reasoning structure. This process continues until the model believes that enough evidence has been collected, or reaches the preset maximum iteration depth, where we generally set the maximum depth to 6. The advantage of the knowledge retrieval tree method is that it fully verifies the authenticity of the original statement from different perspectives through multi-branch exploration. This tree structure avoids the deviation that may be caused by a single reasoning path in the knowledge retrieval chain, improves the comprehensiveness and robustness of the verification process, and further reduces the possibility of model hallucinations.

[0077] The authenticity verification module generates an accuracy label that supports or refutes the statement based on the retrieved question-answer pair information; if the information is insufficient to make a judgment, a failure label is generated. There are three main ways to generate this label: (1) Standard authenticity prompt (Standard VP): directly input the statement and related evidence pairs, output the authenticity rating and explanation, and do not perform additional reasoning. (2) Zero-shot chain of thought accuracy prediction prompt (CoT VP): use the prompt "Let's think step by step" to guide the model to evaluate authenticity through chain of thought reasoning. (3) Verification chain of verification accuracy prediction prompt (Cove): construct verification questions based on the fact-checking explanation generated by LLM, use the retrieval enhancement technology RAG to generate answers, and verify the preliminary verification results through correction check prompts to correct possible "hallucination" problems. Among them, the Cove method is introduced in detail in the paper "Chain-of-verification reduces hallucination in large language models." Through experiments, we found that the prediction performance obtained by the Cove method is the best.

[0078] The trusted response generation module is based on the filtered and verified knowledge and evidence provided by the previous modules. This module is responsible for generating the final, trustworthy user response. Specifically, this module integrates the comprehensive information provided by the multimodal statement generation module, the reliable evidence confirmed by the retrieval-enhanced reasoning module and the authenticity verification module, and the contextual information in the historical conversation.

[0079] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media capable of storing program codes.

[0080] Alternatively, if the above-mentioned embodiments of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media capable of storing program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0081] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. The icons describing the structural positional relationships in the accompanying drawings are only used for exemplary purposes and are not to be construed as limiting the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and is not possible to list all the embodiments here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the claims of the present invention.

Claims

1. A trusted question-answering method based on knowledge verification, characterized in that: The following steps are involved: S1. Combine the context of the user's historical conversations to identify and eliminate references and omissions in the user's questions, use the pre-trained model to restate the user's questions, and extract keywords that express the user's intentions; S2. Use the keyword set in the restated question as a query to retrieve relevant articles from the external knowledge base and large language model, calculate the relevance score between the retrieved content and the question, and select the multimodal knowledge with the highest relevance score; S3, generating a multimodal statement about the search results, and performing multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached; S4. Verify the authenticity of the iterated multimodal statements, construct verification questions based on the fact-checking explanations generated by the large language model, generate answers using retrieval enhancement technology, and verify the preliminary verification results through correction check prompts; S5. Integrate the filtered and verified knowledge and evidence to generate the final credible user response.

2. A trusted question-answering method based on knowledge verification according to claim 1, characterized in that: In step S1, the identification and elimination of references and omitted contents in the user's question specifically includes: obtaining concepts or pronouns in the historical conversation context related to the question, and performing synonymous replacement on omitted contents or unclear references in the user's question according to the association between the pronouns and the concepts.

3. A trusted question-answering method based on knowledge verification according to claim 2, characterized in that: Using pre-trained model M Q Before restating, the pre-trained model needs to be fine-tuned, and the fine-tuning process is guided by optimizing the following objective function: Among them, C i-1 Indicates the context information of the previous text, q i is the original question, q′ i is a new question after restating, by maximizing M Q In C i-1 and q i Under the condition of generating q′ i The probability of completing the optimization objective function; using a large language model To extract the keywords, the model uses the following optimization objective function: Among them, K i Represents the extracted keyword set, by maximizing C i-1 ,q i and q′ i Generate K for the condition i The probability of training 4. A trusted question-answering method based on knowledge verification according to claim 3, characterized in that: The step S2 includes the following specific steps: using the keyword set K i As a query, obtain the relevant knowledge set {D1,D2,…,D M }, each retrieved article D is decomposed into a set of paragraphs and illustrations {p1,p2,…,p |D| }, where for any paragraph and illustration p j Calculate q′ i and K i The relevance score of ij , the specific formula is as follows: Where E(·) represents the embedding function based on images and texts. The multimodal embedding model is used to transform images and texts into a unified high-dimensional vector representation. f(·) is the scoring function, which measures the similarity by calculating the cosine value between two embedded vectors.

5. A trusted question-answering method based on knowledge verification according to claim 4, characterized in that: Calculate the correlation score s ij After that, the selected relevant texts and pictures are re-ranked, and the multimodal embedding model is used to q′ i and p j After characterization and re-ranking, the top n candidate knowledge are selected, denoted as P→p, and used as the input of the reply generator.

6. A trusted question-answering method based on knowledge verification according to claim 1, characterized in that: The step S3 comprises the following specific steps: generating a statement about the search result according to each search result and the corresponding publication date, and verifying the authenticity of the statement about the search result by using the knowledge search chain and the knowledge search tree. First, based on the rewritten user question q′ i and multimodal context information, generate an initial question Q, use the retrieval engine to obtain evidence related to the question from the external knowledge base, and generate an answer A to determine whether the current answer is sufficient to verify the authenticity of the original statement; if not, generate a new follow-up question Q′ based on the existing question-answer pair (Q, A), continue to search and answer, otherwise, end the search; iterate repeatedly to form a continuous question-answer chain {(Q1, A1), (Q2, A2), …, (Q n ,A n )} until the preset maximum number of iterations is reached.

7. A trusted question-answering method based on knowledge verification according to claim 6, characterized in that: The specific process of the knowledge retrieval chain is as follows: Input the rewritten question q′ i , multimodal context information, according to q′ i and context, generate the initial question Q, question-answer pair sequence QAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, use the retrieval engine to obtain evidence related to the current question Q, generate answer A, add (Q, A) to QAPairs, and determine whether to generate subsequent questions based on Q and A. If necessary, generate a new question Q′ and update Q←Q′; otherwise, set followUpNeeded = False, update the counter counter+ = 1, and output the question-answer pair sequence QAPairs.

8. A trusted question-answering method based on knowledge verification according to claim 7, characterized in that: The specific process of the knowledge retrieval tree is as follows: Input the rewritten question q′ i , multimodal context information, according to q′ i and context, generate the initial question set {Q1, Q2, Q3}, initialize the best question-answer pair sequence BestQAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, for each question Q j , use the search engine to obtain relevant evidence and generate answer A j , forming a question-answer pair (Q j ,A j ), for all (Q j ,A j ) to evaluate and select the best question-answer pair (Q b ,A b ), add it to BestQAPairs, and determine whether to generate follow-up questions based on BestQAPairs. If necessary, based on (Q b ,A b ) Generate a new question set {Q′1, Q′2, Q′3}; otherwise, set followUpNeeded=False, update the counter counter+=1, and output the question-answer pair sequence BestQAPair.

9. The method for credible question-answering based on knowledge verification according to claim 1, characterized in that: The step S4 specifically includes the following steps: generating an accuracy label supporting or refuting a statement based on the retrieved question-answer pair information; generating a failure label if the information is insufficient for judgment; constructing a verification question based on the fact-checking explanation generated by the large language model, generating an answer using retrieval enhancement technology, and verifying the preliminary verification result through correction check prompts; the step S5 specifically includes: integrating the comprehensive information provided by step S2, the reliable evidence confirmed by steps S3 and S4, and the contextual information in the historical conversation, and outputting a final, trustworthy user response.

10. A trusted question-answering system based on knowledge verification, the system being used in a trusted question-answering method based on knowledge verification as claimed in any one of claims 1 to 9, characterized in that: It includes: question analysis unit, multimodal knowledge retrieval unit, and credible response unit; The question analysis unit identifies and eliminates references and omitted contents in the user's question in combination with the context of the user's historical conversation, uses a pre-trained model to restate the user's question, and extracts keywords that express the user's intention; the multimodal knowledge retrieval unit uses the keyword set in the restated question as a query, retrieves relevant articles from an external knowledge base and a large language model, calculates the relevance score between the retrieved content and the question, and screens out the multimodal knowledge with the highest relevance score; the credible response unit includes a multimodal statement generation module, a retrieval enhancement reasoning module, and an authenticity verification module. The multimodal statement generation module generates a multimodal statement about the retrieval result. The retrieval enhancement reasoning module performs multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached. The authenticity verification module verifies the authenticity of the iterated multimodal statement, constructs a verification question based on the fact-checking explanation generated by the large language model, generates an answer using the retrieval enhancement technology, and verifies the preliminary verification result through a correction check prompt. The credible response unit integrates the filtered and verified knowledge and evidence to generate a final credible user response.

Images