A knowledge verification-based trusted question and answer method

Abstract

The application discloses a kind of based on knowledge verification's trusted question and answer method, including the following steps: identifying and eliminating the reference and omission content in user's asking question, restating user's asking question, and extracting the keyword expressing user's intention.Use the keyword set after restating as query, retrieve relevant articles, calculate the relevance score of the content retrieved and the asking question, and filter out the highest relevance score multi-modal knowledge.Generating multi-modal statements about search results, multi-round, multi-path verification and reasoning are carried out on the multi-modal statements.The authenticity of the iterated multi-modal statements is verified, and the filtered and verified knowledge and evidence are integrated to generate the final trusted user reply.Compared with the prior art, the application can eliminate references and omissions by injecting associated historical context information during the question representation process, and obtain trusted multi-modal knowledge using retrieval enhancement and self-checking mechanisms, thereby more accurately answering the questions asked by the user.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a credible question-answering method based on knowledge verification. Background Art

[0002] With the rapid development of the internet and artificial intelligence (AI) technologies, the "Internet + Traditional Industries" model is becoming increasingly important across various sectors. For example, in the tourism industry, China is vigorously developing "Internet + Tourism," leveraging cutting-edge technologies like big data and deep learning to innovate tourism management models. This has become a key trend in my country's tourism development. In the past, during peak tourist seasons, tourists visiting scenic spots often relied on manual consultation to obtain information. This approach was inefficient and untimely. To address this issue, intelligent question-and-answer systems have emerged, enabling machines to automatically answer tourists' questions and improve consultation efficiency. Compared to traditional search engines, question-and-answer systems can provide direct answers, eliminating the need for users to search through lengthy search lists. This significantly improves information retrieval efficiency and provides users with a better travel experience. This technology is of great significance for the further implementation of smart tourism and digital tourism.

[0003] User questions often involve information from diverse dimensions, modalities, and levels of a scenic spot, requiring question-answering models to deeply understand this information. Traditional approaches typically focus solely on the text's content to generate answers, ignoring the multimodal domain knowledge underlying the text. This leads to one-sided judgments and reasoning. To acquire domain knowledge, a common strategy is to construct domain-specific knowledge graphs. However, this process relies on expert knowledge, which is costly, has low knowledge coverage, and can become outdated. Another approach is to use large language models (LLMs) as domain knowledge bases. LLMs are pre-trained on large-scale internet data to capture rich domain knowledge. By applying hint learning techniques, LLMs can be guided to output knowledge relevant to the question. However, LLMs suffer from the problem of hallucinations. Furthermore, the prevalence of low-quality articles containing false content on the internet can exacerbate this problem. Existing question-answering models lack knowledge verification mechanisms, making them prone to generating unreliable results and causing multiple losses. For example, responding to tourists based on malicious negative reviews can damage the image of a scenic spot. With the rapid development of automated generation technologies like ChatGPT and Stable Diffusion, the creation of fake articles has become much easier. Traditional manual review mechanisms are no longer able to cope with the massive volume of machine-generated articles. Therefore, trusted question-answering technology can enhance the reliability of output results, which is of great significance and value in purifying the online environment, restoring media credibility, and improving information acquisition efficiency.

[0004] Traditional methods retrieve relevant articles from the internet or dedicated databases to answer questions. The retrieved content may be false due to outdatedness, malicious fabrication, and other reasons, resulting in incorrect answers that mislead users. Furthermore, these methods only consider textual content, ignoring the rich knowledge contained in other modalities such as images, which can easily lead to one-sided answers. In real-world application scenarios, users generally learn about scenic spots through multiple rounds of dialogue. User questions often refer to content in historical dialogues, but this reference may not be obvious or may even be omitted. Existing methods mainly focus on single-round question answering, ignoring this implicit contextual association when representing questions, making it difficult to effectively combine historical content to form accurate responses.

[0005] The prior art discloses a method and system for mitigating the hallucination of large language models by using an external knowledge base for verification. This relates to the technical field of large language models. The method includes the following steps: obtaining textual information output by a large language model and converting it into structured fact triples; generating questions based on the structured fact triples; collecting and verifying answers used to verify the fact triples using a retrieval system; and revising the fact triples based on the verification results. A drawback of this solution is that relevant articles are retrieved from an external, dedicated database to answer questions. Due to outdated content, malicious fabrication, and other factors, the retrieved content may contain false information, resulting in incorrect answers that mislead users.

[0006] To this end, in combination with the above requirements and the shortcomings of the existing technology, this application proposes a trusted question-answering method based on knowledge verification. Summary of the Invention

[0007] The present invention provides a credible question-answering method based on knowledge verification, which can integrate historical context information to resolve reference and omission, and use retrieval enhancement and self-verification mechanisms to obtain multimodal knowledge with high confidence, thereby answering users' questions more comprehensively.

[0008] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows:

[0009] A first aspect of the present invention provides a trusted question-answering method based on knowledge verification, the method comprising the following steps:

[0010] S1. Combining the context of the user's historical conversation, identify and eliminate references and omissions in the user's questions, use the pre-trained model to restate the user's questions, and extract keywords that express the user's intentions.

[0011] S2. Use the keyword set in the restated question as a query to retrieve relevant articles from the external knowledge base and large language model, calculate the relevance score between the retrieved content and the question, and screen out the multimodal knowledge with the highest relevance score.

[0012] S3. Generate a multimodal statement about the retrieval results, and perform multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached.

[0013] S4. Verify the authenticity of the iterated multimodal statements, construct verification questions based on the fact-checking explanations generated by the large language model, generate answers using retrieval enhancement technology, and verify the preliminary verification results through correction check prompts.

[0014] S5. Integrate the filtered and verified knowledge and evidence to generate the final credible user response.

[0015] Furthermore, in step S1, the identification and elimination of references and omitted content in the user's question is specifically: obtaining concepts or pronouns in the historical conversation context related to the question, and performing synonymous replacement on the omitted content or unclear reference in the user's question based on the association between the pronoun and the concept.

[0016] Furthermore, the pre-trained model M is used Q To reiterate, the pre-trained model needs to be fine-tuned before re-stating, and the fine-tuning process is guided by optimizing the following objective function:

[0017]

[0018] Among them, C i-1 Indicates the context information of the previous text, q i is the original question, q′ i is a new question after restating, by maximizing M Q In C i-1 and q i Generate q′ under the condition i The probability of completing the optimization objective function; using a large language model To extract the keywords, the model uses the following optimization objective function:

[0019]

[0020] Among them, K i Represents the extracted keyword set, by maximizing C i-1 ,q i and q′ i Generate K for the condition i The probability of training

[0021] Furthermore, the step S2 includes the following specific steps: using the keyword set K i As a query, obtain the relevant knowledge set {D1,D2,…,D M}, each retrieved article D is decomposed into a set of paragraphs and illustrations {p1,p2,…,p |D|}, where for any paragraph and illustration p j Calculate q′ i and K i The relevance score s ij , the specific formula is as follows:

[0022]

[0023] Where E(·) represents the embedding function based on images and texts. A multimodal embedding model is used to convert images and texts into a unified high-dimensional vector representation. f(·) is a scoring function that measures the similarity by calculating the cosine value between two embedding vectors.

[0024] Furthermore, the correlation score s is calculated ij After that, the filtered relevant texts and pictures are re-ranked, and the multimodal embedding model is used to q′ i and p j After characterization and re-ranking, the top n candidate knowledge are selected, denoted as P→p, and used as the input of the reply generator.

[0025] Furthermore, step S3 includes the following specific steps: generating a statement about the search result based on each search result and the corresponding release date, and verifying the authenticity of the statement about the search result using the knowledge search chain and knowledge search tree. First, based on the rewritten user question q′ i Based on the multimodal context information, an initial question Q is generated. The retrieval engine is used to obtain evidence related to the question from the external knowledge base and generate an answer A. It is judged whether the current answer is sufficient to verify the authenticity of the original statement. If not, a new follow-up question Q' is generated based on the existing question-answer pair (Q, A) and the retrieval and answer are continued. Otherwise, the retrieval is terminated. It is repeated to form a continuous question-answer chain {(Q1, A1), (Q2, A2), ..., (Q n ,A n )} until the preset maximum number of iterations is reached.

[0026] Furthermore, the specific process of the knowledge retrieval chain is as follows: Input the rewritten question q′ i , multimodal context information, according to q′ iand context, generate the initial question Q, question-answer pair sequence QAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, use the retrieval engine to obtain evidence related to the current question Q, generate answer A, add (Q, A) to QAPairs, and determine whether to generate subsequent questions based on Q and A. If necessary, generate a new question Q′ and update Q←Q′; otherwise, set followUpNeeded = False, update the counter counter + = 1, and output the question-answer pair sequence QAPairs.

[0027] Furthermore, the specific process of the knowledge retrieval tree is as follows: input the rewritten question q′ i , multimodal context information, according to q′ i And context, generate the initial question set {Q1, Q2, Q3}, initialize the best question-answer pair sequence BestQAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, for each question Q j , use the search engine to obtain relevant evidence and generate answer A j , forming a question-answer pair (Q j ,A j ), for all (Q j ,A j ) to evaluate and select the best question-answer pair (Q b ,A b ), add it to BestQAPairs, and judge whether to generate follow-up questions based on BestQAPairs. If necessary, based on (Q b ,A b ) Generate a new question set {Q′1, Q′2, Q′3}; otherwise, set followUpNeeded=False, update the counter counter+=1, and output the question-answer pair sequence BestQAPairs.

[0028] Furthermore, step S4 specifically includes the following steps: generating an accuracy label supporting or refuting the statement based on the retrieved question-answer pair information; generating a failure label if the information is insufficient for judgment; constructing a verification question based on the fact-checking explanation generated by the large language model, generating an answer using retrieval enhancement technology, and verifying the preliminary verification result through correction check prompts; step S5 specifically includes: integrating the comprehensive information provided by step S2, the reliable evidence confirmed by steps S3 and S4, and the contextual information in the historical conversation, and outputting the final, trustworthy user response.

[0029] The second aspect of the present invention provides a trusted question-answering system based on knowledge verification, which is used in the trusted question-answering method based on knowledge verification and includes: a question analysis unit, a multimodal knowledge retrieval unit, and a trusted response unit.

[0030] The question analysis unit identifies and eliminates references and omitted content in user questions based on the context of the user's historical conversations, uses a pre-trained model to restate the user's question, and extracts keywords that express the user's intent. The multimodal knowledge retrieval unit uses the keyword set in the restated question as a query to retrieve relevant articles from an external knowledge base and a large language model, calculates the relevance score between the retrieved content and the question, and selects the multimodal knowledge with the highest relevance score. The trusted response unit includes a multimodal statement generation module, a retrieval-enhanced reasoning module, and an authenticity verification module. The multimodal statement generation module generates a multimodal statement about the retrieval results. The retrieval-enhanced reasoning module performs multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached. The authenticity verification module verifies the authenticity of the iterated multimodal statement, constructs a verification question based on the fact-checking explanation generated by the large language model, generates an answer using retrieval-enhanced technology, and verifies the preliminary verification results through correction check prompts. The trusted response unit integrates the filtered and verified knowledge and evidence to generate a final trusted user response.

[0031] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0032] The present invention provides a credible question-answering method based on knowledge verification, which can eliminate references and omissions by injecting relevant historical context information into the question representation process, and obtain credible multimodal knowledge by utilizing retrieval enhancement and self-verification mechanisms. In combination with extracting key multimodal knowledge from retrieval results, it alleviates the hallucination problem of large models and guides them to generate more credible replies, thereby more accurately answering questions raised by users. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of a trusted question-answering method based on knowledge verification of the present invention.

[0034] Figure 2 The flowchart of question restatement, fine-grained retrieval and trusted answer in one embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of a trusted question-answering system based on knowledge verification according to the present invention. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0038] Example 1

[0039] like Figure 1 As shown, the present invention provides a trusted question-answering method based on knowledge verification, which includes the following steps:

[0040] S1. Combining the context of the user's historical conversation, identify and eliminate references and omissions in the user's questions, use the pre-trained model to restate the user's questions, and extract keywords that express the user's intentions.

[0041] S2. Use the keyword set in the restated question as a query to retrieve relevant articles from the external knowledge base and large language model, calculate the relevance score between the retrieved content and the question, and screen out the multimodal knowledge with the highest relevance score.

[0042] S3. Generate a multimodal statement about the retrieval results, and perform multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached.

[0043] S4. Verify the authenticity of the iterated multimodal statements, construct verification questions based on the fact-checking explanations generated by the large language model, generate answers using retrieval enhancement technology, and verify the preliminary verification results through correction check prompts.

[0044] S5. Integrate the filtered and verified knowledge and evidence to generate the final credible user response.

[0045] According to the above technical features, the present invention can integrate relevant historical context information and introduce a self-verification mechanism to provide users with highly reliable and trustworthy answers. Specifically, the current question is first rewritten based on the conversation history, the referents of the pronouns in the question are clarified and the omitted content is supplemented, so as to understand the user's query content more finely. Subsequently, a fine-grained retriever is designed to extract articles related to the rewritten question from the external knowledge base and the large model LLMs. Then, a self-verification mechanism is developed to filter out potential false retrieval content. Unlike traditional unimodal methods, this mechanism combines multimodal content such as text and images for cross-modal reasoning, which can more comprehensively evaluate the confidence of the retrieval results. By injecting these credible retrieval results into the large model, the problem of period hallucination can be alleviated, allowing it to extract key multimodal clues from them, thereby generating reliable answers to the questions.

[0046] This invention aims to provide users with fast and reliable consulting services. It can be applied in the tourism sector to efficiently and concisely answer users' questions about tourist attractions. In practical applications, it can help tourism platforms more comprehensively integrate multimodal knowledge (such as images and text) to meet users' information needs. It can also help companies analyze tourist behavior, thereby improving user experience and operational effectiveness.

[0047] It should be noted that, compared with the prior art, the present invention has the following advantages:

[0048] By injecting relevant historical conversation context and leveraging a fine-tuned large-scale language model to restate the user's current question, eliminating references and ellipsis, we can more accurately understand the user's true intent. This process not only ensures the semantic accuracy of the question but also optimizes subsequent knowledge retrieval through keyword extraction, laying a solid foundation for generating high-quality answers.

[0049] A fine-grained retriever was designed to extract text and image content highly relevant to the restated question from an external knowledge base and a large language model. A multimodal embedding model was used to uniformly represent the retrieved content, and a re-ranking mechanism was employed to select the most relevant paragraphs and images. This not only enriched the knowledge base of the question-answering system but also ensured the comprehensiveness and accuracy of the information obtained, providing multi-dimensional support for generating trustworthy answers.

[0050] To mitigate the hallucination problem of large language models, a self-verification mechanism was introduced. A multimodal fact-checking module was used to filter out low-quality content containing false information. Reasoning methods such as knowledge retrieval chains and retrieval trees were used to generate and verify multiple rounds of question-answer pairs. This, combined with standard authenticity cues and verification chain accuracy prediction cues, ensured that the generated answers were more accurate and trustworthy.

[0051] Example 2

[0052] Based on the above embodiment 1, this embodiment describes the specific process of the present invention in detail.

[0053] like Figure 1 As shown, further, in step S1, the identification and elimination of references and omitted content in the user's question is specifically: obtaining concepts or pronouns in the historical conversation context related to the question, and performing synonymous replacement on the omitted content or unclear reference in the user's question based on the association between the pronoun and the concept.

[0054] Furthermore, the pre-trained model M is used Q To reiterate, the pre-trained model needs to be fine-tuned before re-stating, and the fine-tuning process is guided by optimizing the following objective function:

[0055]

[0056] Among them, C i-1 Indicates the context information of the previous text, q i is the original question, q′ i is a new question after restating, by maximizing M Q In C i-1 and q i Generate q′ under the condition i The probability of completing the optimization objective function; using a large language model To extract the keywords, the model uses the following optimization objective function:

[0057]

[0058] Among them, K i Represents the extracted keyword set, by maximizing C i-1 ,q i and q′ i Generate K for the condition i The probability of training

[0059] Furthermore, the step S2 includes the following specific steps: using the keyword set K i As a query, obtain the relevant knowledge set {D1,D2,…,D M}, each retrieved article D is decomposed into a set of paragraphs and illustrations {p1,p2,…,p |D|}, where for any paragraph and illustration p j Calculate q′ i and K i The relevance score s ij , the specific formula is as follows:

[0060]

[0061] Where E(·) represents the embedding function based on images and texts. A multimodal embedding model is used to convert images and texts into a unified high-dimensional vector representation. f(·) is a scoring function that measures the similarity by calculating the cosine value between two embedding vectors.

[0062] Furthermore, the correlation score s is calculated ij After that, the relevant texts and pictures that have been filtered out are re-

[0063] New ranking, the multimodal embedding model used for q i and p j After characterization and re-ranking, the top n candidate knowledge are selected, denoted as P→p, and used as the input of the reply generator.

[0064] Furthermore, step S3 includes the following specific steps: generating a statement about the search result based on each search result and the corresponding release date, and verifying the authenticity of the statement about the search result using the knowledge search chain and knowledge search tree. First, based on the rewritten user question q′ i Based on the multimodal context information, an initial question Q is generated. The retrieval engine is used to obtain evidence related to the question from the external knowledge base and generate an answer A. It is judged whether the current answer is sufficient to verify the authenticity of the original statement. If not, a new follow-up question Q' is generated based on the existing question-answer pair (Q, A) and the retrieval and answer are continued. Otherwise, the retrieval is terminated. It is repeated to form a continuous question-answer chain {(Q1, A1), (Q2, A2), ..., (Q n ,A n )} until the preset maximum number of iterations is reached.

[0065] Furthermore, the specific process of the knowledge retrieval chain is as follows: Input the rewritten question q′ i , multimodal context information, according to q′ i and context, generate the initial question Q, question-answer pair sequence QAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, use the retrieval engine to obtain evidence related to the current question Q, generate answer A, add (Q, A) to QAPairs, and determine whether to generate subsequent questions based on Q and A. If necessary, generate a new question Q′ and update Q←Q′; otherwise, set followUpNeeded = False, update the counter counter + = 1, and output the question-answer pair sequence QAPairs.

[0066] Furthermore, the specific process of the knowledge retrieval tree is as follows: input the rewritten question q′ i , multimodal context information, according to q′ i And context, generate the initial question set {Q1, Q2, Q3}, initialize the best question-answer pair sequence BestQAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, for each question Q j , use the search engine to obtain relevant evidence and generate answer A j , forming a question-answer pair (Q j ,A j ), for all (Q j ,A j ) to evaluate and select the best question-answer pair (Q b ,A b ), add it to BestQAPairs, and judge whether to generate follow-up questions based on BestQAPairs. If necessary, based on (Q b ,A b ) Generate a new question set {Q′1, Q′2, Q′3}; otherwise, set followUpNeeded=False, update the counter counter+=1, and output the question-answer pair sequence BestQAPairs.

[0067] Furthermore, step S4 specifically includes the following steps: generating an accuracy label supporting or refuting the statement based on the retrieved question-answer pair information; generating a failure label if the information is insufficient for judgment; constructing a verification question based on the fact-checking explanation generated by the large language model, generating an answer using retrieval enhancement technology, and verifying the preliminary verification result through correction check prompts; step S5 specifically includes: integrating the comprehensive information provided by step S2, the reliable evidence confirmed by steps S3 and S4, and the contextual information in the historical conversation, and outputting the final, trustworthy user response.

[0068] Example 3

[0069] like Figure 2-Figure 3 As shown, the present invention also provides a trusted question-answering system based on knowledge verification, which is used for the trusted question-answering method based on knowledge verification, and includes: a question analysis unit 101, a multimodal knowledge retrieval unit 102, and a trusted response unit 103.

[0070] The question analysis unit 101 identifies and eliminates references and omitted content in the user's question based on the context of the user's historical conversation, uses a pre-trained model to restate the user's question, and extracts keywords that express the user's intent. The multimodal knowledge retrieval unit 102 uses the keyword set in the restated question as a query to retrieve relevant articles from an external knowledge base and a large language model, calculates the relevance score between the retrieved content and the question, and selects the multimodal knowledge with the highest relevance score. The trusted response unit 103 includes a multimodal statement generation module, a retrieval-enhanced reasoning module, and an authenticity verification module. The multimodal statement generation module generates a multimodal statement about the retrieval results. The retrieval-enhanced reasoning module performs multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached. The authenticity verification module verifies the authenticity of the iterated multimodal statement, constructs a verification question based on the fact-checking explanation generated by the large language model, generates an answer using retrieval-enhanced technology, and verifies the preliminary verification result through a correction check prompt. The trusted response unit integrates the filtered and verified knowledge and evidence to generate a final trusted user response.

[0071] In a specific embodiment, Figure 2 As shown, the contents of each unit and module are introduced in detail.

[0072] Question analysis unit 101: This unit aims to analyze the details of the user's question, eliminate references in the question, complete omitted content based on the conversation history context related to the question, and thus more accurately understand the user's intention. In actual applications, the user's question is usually related to the concepts or pronouns mentioned above, such as Figure 2 As shown in the figure, "this scenic spot" actually refers to "Hangzhou West Lake". This unclear reference will confuse the model about the questioner's intention. Therefore, the question needs to be restated to make the questioner's intention clearer. Figure 2 As shown in the figure, when a user asks "What are the must-see attractions in this scenic area?", the intended meaning is "What are the must-see attractions in West Lake, Hangzhou?" To enable the machine to accurately understand this intention literally, we refer to the context and replace the ambiguous part of the original question "this scenic area" with "West Lake, Hangzhou", making the question clearer and more specific.

[0073] Multimodal knowledge retrieval unit 102: This unit aims to use the external knowledge base to retrieve relevant and credible knowledge to eliminate the illusion problem of LLM, thereby improving the reliability of the answer. Specifically, this patent first uses the keyword set K i As a query, relevant articles are searched through the Google search engine. The goal of this process is to preliminarily screen a set of potentially relevant knowledge sets from a wide range of Internet resources, denoted as {D1, D2, …, D M}.like Figure 2 As shown in the figure, it can convert pictures and text into a unified high-dimensional vector representation, has good multimodal feature representation capabilities, and is applicable to text, images, videos, audio, and even thermal data and gyroscope data.

[0074] Trusted Response Generation Unit 103: This unit aims to filter out low-quality retrieval results containing false content through a self-checking mechanism, thereby alleviating the “hallucination” problem of large model LLMs, e.g. Figure 2 The system consists of three modules. The multimodal statement generation module generates statements about each search result based on its publication date. In a specific embodiment, GPT-4V is used to represent multimodal content, which has strong text and image understanding capabilities. The generated new statements not only contain the original retrieved text content but also incorporate relevant details extracted from the image. This allows for a wider range of context to be obtained, providing the model with sufficient basis for judgment.

[0075] The retrieval-enhanced reasoning module aims to combine the retrieval-enhanced reasoning strategy to deeply verify the authenticity of the multimodal statements generated by the above modules, thereby improving the credibility and accuracy of the generated responses. To this end, two novel reasoning methods, namely "knowledge retrieval chain" and "knowledge retrieval tree", are introduced to achieve multi-round and multi-path verification and reasoning of user questions. In this process, a reasoning strategy based on continuous question-answering chain is adopted. The knowledge retrieval tree algorithm process first calculates the user question q′ according to the rewritten user question q′. i and multimodal context information to generate an initial question Q, which aims to target the key elements in the question. Then, the retrieval engine is used to obtain evidence related to the question from the external knowledge base and generate an answer A. Next, the model determines whether the current answer is sufficient to verify the authenticity of the original statement; if not, it generates a new follow-up question Q' based on the existing question-answer pair (Q, A) and continues to search and answer. This process is iterated repeatedly to form a continuous question-answer chain {(Q1, A1), (Q2, A2), ..., (Q n ,A n )} until the model deems sufficient evidence has been obtained or reaches a preset maximum number of iterations. The core of the knowledge retrieval chain approach is to guide the model to gradually generate highly relevant follow-up questions. By retrieving enhanced answers, the model continuously enriches and verifies its understanding of the original question. This iterative process enables the model to deeply tap into useful information in external knowledge bases, reduce illusions caused by insufficient information, and improve the accuracy and reliability of responses.

[0076] However, the knowledge retrieval chain method may easily miss some important verification information because it only focuses on a single reasoning path, resulting in insufficient comprehensive verification of complex statements. Therefore, in order to improve this problem, we also introduced the knowledge retrieval tree method based on the knowledge retrieval chain. In the knowledge retrieval tree method, in order to further improve the verification ability of the model, we introduced a multi-branch question-answering strategy in each reasoning step to form a tree-like reasoning structure. The knowledge retrieval tree algorithm step will simultaneously generate multiple different sets of follow-up questions {Q1, Q2, Q3} at each step to verify the questions from different angles. For each generated question Q j , use the search engine to obtain relevant evidence and generate answer A j Subsequently, the model analyzes these question-answer pairs (Q j ,A j ) is evaluated and the question-answer pair (Q b ,A b ) as candidate evidence. Then, based on (Q b ,A b ) generates new follow-up questions, and the above process repeats, forming a tree-like reasoning structure. This process continues until the model deems sufficient evidence has been collected or reaches a preset maximum iteration depth, which we generally set to 6. The advantage of the knowledge retrieval tree method is that it comprehensively verifies the truth of the original statement from different perspectives through multi-branch exploration. This tree-like structure avoids the potential bias caused by a single reasoning path in the knowledge retrieval chain, improves the comprehensiveness and robustness of the verification process, and further reduces the possibility of model hallucinations.

[0077] The authenticity verification module generates an accuracy label that supports or refutes the statement based on the retrieved question-answer pair information; if the information is insufficient to make a judgment, a failure label is generated. There are three main methods to generate this label: (1) Standard authenticity prompt (Standard VP): directly input the statement and related evidence pairs, output the authenticity rating and explanation, and do not perform additional reasoning. (2) Zero-shot chain of thought accuracy prediction prompt (CoT VP): use the prompt "Let's think step by step" to guide the model to evaluate the authenticity through chain of thought reasoning. (3) Verification chain of verification prediction prompt (Cove): construct verification questions based on the fact-checking explanation generated by LLM, use the retrieval enhancement technology RAG to generate answers, and verify the preliminary verification results through correction check prompts to correct possible "hallucination" problems. Among them, the Cove method is introduced in detail in the paper "Chain-of-verification reduces hallucination in large language models." Through experiments, we found that the prediction performance obtained by the Cove method is the best.

[0078] The Trusted Response Generation Module, based on the filtered and verified knowledge and evidence provided by the aforementioned modules, is responsible for generating the final, trustworthy user response. Specifically, this module integrates the comprehensive information provided by the Multimodal Statement Generation Module, the reliable evidence confirmed by the Retrieval-Enhanced Reasoning Module and the Authenticity Verification Module, and contextual information from historical conversations.

[0079] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. Those skilled in the art will understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks or optical disks.

[0080] Alternatively, if the above-mentioned embodiments of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0081] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. The icons in the accompanying drawings that describe the structural positional relationships are only for illustrative purposes and are not to be construed as limiting the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A trusted question-answering method based on knowledge verification, characterized in that: The following steps are involved: S1. Combining the context of the user's historical conversations, identify and eliminate references and omissions in the user's questions, use a pre-trained model to restate the user's questions, and extract keywords that express the user's intent; S2. Use the keyword set in the restated question as a query to retrieve relevant articles from the external knowledge base and large language model, calculate the relevance score between the retrieved content and the question, and select the multimodal knowledge with the highest relevance score; S3, generate a multimodal statement about the search results, and perform multiple rounds and multi-path verification and reasoning on the multimodal statement until the preset maximum number of iterations is reached; Step S3 includes the following specific steps: based on each search result and the corresponding release date, generate a statement about the search result, and use the knowledge search chain and knowledge search tree to deeply verify the authenticity of the statement of the search result. First, based on the rewritten user question q i ′ and multimodal context information, generate an initial question Q, use the retrieval engine to obtain evidence related to the question from the external knowledge base, and generate an answer A, and judge whether the current answer is sufficient to verify the authenticity of the original statement; if not, generate a new follow-up question Q′ based on the existing question-answer pair (Q, A), continue to search and answer, otherwise, end the search; iterate repeatedly to form a continuous question-answer chain {(Q1, A1), (Q2, A2), …, (Q n ,A n )}, until the preset maximum number of iterations is reached; the specific process of the knowledge retrieval chain is: input the rewritten question q i ′, multimodal context information, according to q′ i and context, generate the initial question Q, the question-answer pair sequence QAPairs = [], counter counter = 0, and the termination flag followUpNeeded = True. When counter < the maximum iteration depth and followUpNeeded is true, use the retrieval engine to obtain evidence related to the current question Q, generate the answer A, and add (Q, A) to QAPairs. Based on Q and A, determine whether a subsequent question needs to be generated. If so, generate a new question Q′ and update Q←Q′; otherwise, set followUpNeeded = False, update the counter counter + = 1, and output the question-answer pair sequence QAPairs; S4. Verify the authenticity of the iterated multimodal statements, construct verification questions based on the fact-checking explanations generated by the large language model, generate answers using retrieval enhancement technology, and verify the preliminary verification results through correction check prompts; S5. Integrate the filtered and verified knowledge and evidence to generate the final credible user response.

2. A trusted question-answering method based on knowledge verification according to claim 1, characterized in that: In step S1, the identification and elimination of references and omitted content in the user's question specifically includes: obtaining concepts or pronouns in the historical conversation context related to the question, and performing synonymous replacement on the omitted content or unclear reference in the user's question based on the association between the pronoun and the concept.

3. A trusted question-answering method based on knowledge verification according to claim 2, characterized in that: Using pre-trained model M Q To reiterate, the pre-trained model needs to be fine-tuned before re-stating, and the fine-tuning process is guided by optimizing the following objective function: Among them, C i-1 Indicates the context information of the previous text, q i is the original question, q′ i is a new question after restating, by maximizing M Q In C i-1 and q i Generate q′ under the condition i The probability of completing the optimization objective function; using a large language model To extract the keywords, the model uses the following optimization objective function: Among them, K i Represents the extracted keyword set, by maximizing C i-1 ,q i and q′ i Generate K for the condition i The probability of training 4. A trusted question-answering method based on knowledge verification according to claim 3, characterized in that: The step S2 includes the following specific steps: using the keyword set K i As a query, obtain the relevant knowledge set {D1,D2,…,D M }, each retrieved article D is decomposed into a set of paragraphs and illustrations {p1,p2,…,p |D| }, where for any paragraph and illustration p j Calculate q′ i and K i The relevance score s ij , the specific formula is as follows: Where E(·) represents the embedding function based on images and texts. A multimodal embedding model is used to convert images and texts into a unified high-dimensional vector representation. f(·) is a scoring function that measures the similarity by calculating the cosine value between two embedding vectors.

5. A trusted question-answering method based on knowledge verification according to claim 4, characterized in that: Calculate the correlation score s ij After that, the filtered relevant texts and pictures are re-ranked, and the multimodal embedding model is used to q′ i and p j After characterization and re-ranking, the top n candidate knowledge are selected, denoted as P→p, and used as the input of the reply generator.

6. A trusted question-answering method based on knowledge verification according to claim 1, characterized in that: The specific process of the knowledge retrieval tree is as follows: Input the rewritten question q i ′, multimodal context information, according to q i ′ and context, generate the initial question set {Q1, Q2, Q3}, initialize the best question-answer pair sequence BeatQAPairs = [], counter counter = 0, termination flag followUpNeeded = True, when counter < maximum iteration depth and followUpNeeded is true, for each question Q j , use the search engine to obtain relevant evidence and generate answer A j , forming a question-answer pair (Q j ,A j ), for all (Q j ,A j ) to evaluate and select the best question-answer pair (Q b ,A b ), add it to BestQAPairs, and judge whether to generate follow-up questions based on BestQAPairs. If necessary, based on (Q b ,A b ) Generate a new question set {Q′1, Q′2, Q′3}; otherwise, set followUpNeeded=False, update the counter counter+=1, and output the question-answer pair sequence BestQAPair.

7. A trusted question-answering method based on knowledge verification according to claim 1, characterized in that: The step S4 specifically includes the following steps: generating an accuracy label supporting or refuting the statement based on the retrieved question-answer pair information; generating a failure label if the information is insufficient for judgment; constructing a verification question based on the fact-checking explanation generated by the large language model, generating an answer using retrieval enhancement technology, and verifying the preliminary verification result through correction check prompts; the step S5 specifically includes: integrating the comprehensive information provided by step S2, the reliable evidence confirmed by steps S3 and S4, and the contextual information in the historical conversation, and outputting a final, trustworthy user response.

8. A trusted question-answering system based on knowledge verification, the system being used in a trusted question-answering method based on knowledge verification according to any one of claims 1 to 7, characterized in that: Includes: question analysis unit, multimodal knowledge retrieval unit, and credible response unit; The question analysis unit identifies and eliminates references and omitted content in user questions based on the context of the user's historical conversations, uses a pre-trained model to restate the user's question, and extracts keywords that express the user's intent. The multimodal knowledge retrieval unit uses the keyword set in the restated question as a query to retrieve relevant articles from an external knowledge base and a large language model, calculates the relevance score between the retrieved content and the question, and selects the multimodal knowledge with the highest relevance score. The trusted response unit includes a multimodal statement generation module, a retrieval-enhanced reasoning module, and an authenticity verification module. The multimodal statement generation module generates a multimodal statement about the retrieval results. The retrieval-enhanced reasoning module performs multiple rounds and multi-path verification and reasoning on the multimodal statement until a preset maximum number of iterations is reached. The authenticity verification module verifies the authenticity of the iterated multimodal statement, constructs a verification question based on the fact-checking explanation generated by the large language model, generates an answer using retrieval-enhanced technology, and verifies the preliminary verification results through correction check prompts. The trusted response unit integrates the filtered and verified knowledge and evidence to generate a final trusted user response.

Images