Question and answer method and device, electronic equipment and storage medium

Abstract

This invention provides a question-answering method, apparatus, electronic device, and storage medium. The method includes: retrieving multiple documents based on a user query; determining, using a large language model, a target conflict type and corresponding explanatory information for the multiple documents, where the target conflict type represents the type of knowledge conflict between the document content of the multiple documents, and the explanatory information explains the reasons for the knowledge conflict between the multiple documents; obtaining a target answer generation strategy corresponding to the target conflict type from a mapping relationship between conflict types and answer generation strategies; generating an answer corresponding to the user query using the large language model based on first prompt information and the multiple documents, where the first prompt information includes the target answer generation strategy and explanatory information, and the first prompt information provides a way to resolve the knowledge conflict between the multiple documents; and outputting the answer. This invention can improve the accuracy of the generated answer.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a question-and-answer method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, the industry commonly employs Retrieval-Augmented Generation (RAG) technology, which uses an external knowledge base to improve the real-time performance and accuracy of Large Language Models (LLMs) in answering questions. The basic process of RAG is: receiving a user question, retrieving relevant documents, and then feeding both the question and document content to the LLM to generate the answer. However, existing RAG technologies generally suffer from a fatal flaw: they mostly assume that the retrieved information is singular, consistent, and accurate.

[0003] When multiple documents retrieved contain contradictory or inconsistent information, existing RAG systems often fail to provide accurate answers. Specifically, this manifests in several ways: answers containing contradictory information may result in the system simply piecing together conflicting viewpoints to generate logically flawed and unusable answers; answers may be randomly selected or based on conjecture, meaning the system may randomly accept one source without clear evidence or create "illusions" based on contradictory information, leading to a lack of assurance regarding the accuracy and reliability of the output results. Summary of the Invention

[0004] The purpose of this invention is to provide a question-and-answer method, apparatus, electronic device, and storage medium to improve the accuracy of answers. The specific technical solution is as follows: In a first aspect of this invention, a question-and-answer method is provided, comprising: Based on the user's query, multiple documents were retrieved. Based on the user query using a large language model, the target conflict type of the multiple documents and the corresponding explanatory information are determined. The target conflict type represents the type of knowledge conflict between the document content of the multiple documents, and the explanatory information is used to explain the reasons for the knowledge conflict in the multiple documents. From the mapping relationship between conflict types and answer generation strategies, obtain the target answer generation strategy corresponding to the target conflict type; Based on the first prompt information and the multiple documents, a large language model is used to generate an answer corresponding to the user's query. The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents. Output the answer.

[0005] In a second aspect of the invention, a question-and-answer device is also provided, comprising: The search module is used to retrieve multiple documents based on user queries; The conflict classification module is used to determine the target conflict type of the multiple documents and the corresponding explanatory information based on the user query using a large language model. The target conflict type represents the knowledge conflict type between the document content of the multiple documents, and the explanatory information is used to explain the reasons for the knowledge conflict between the multiple documents. The strategy acquisition module is used to acquire the target answer generation strategy corresponding to the target conflict type from the mapping relationship between conflict types and answer generation strategies; The answer generation module is used to generate an answer corresponding to the user query based on the first prompt information and the multiple documents using a large language model. The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents. The answer output module is used to output the answer.

[0006] In another aspect of the present invention, an electronic device is also provided, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described above.

[0007] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform any of the question-and-answer methods described above.

[0008] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the question-and-answer methods described above.

[0009] The question-answering method, apparatus, electronic device, and storage medium provided in this invention, after retrieving multiple documents based on a user query, determine the target conflict type of the multiple documents and the corresponding explanatory information, obtain the target answer generation strategy corresponding to the target conflict type, and then generate an answer corresponding to the user query using a large language model based on the first prompt information including the target answer generation strategy and explanatory information, and the multiple documents. In this way, the target answer generation strategy can resolve knowledge conflicts between different documents, reduce conflicting information in the generated answer, and improve the accuracy of the generated answer. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0011] Figure 1 This is a flowchart of a question-and-answer method provided in an embodiment of the present invention; Figure 2 This is a flowchart of another question-and-answer method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a question-and-answer system provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a question-and-answer device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0013] The embodiments of this invention are primarily applied to the fields of AI-driven content creation, review, and decision support. For example, the embodiments of this invention can be applied to the following scenarios: 1) AI Scriptwriting Assistance System: When screenwriters or planning teams are building a script, they need to consult a large amount of background information, such as events, relationships between characters, social customs, or professional knowledge of a specific industry during a particular historical period. For example, when creating a business drama with a financial theme, the system needs to extract information from a large amount of financial news, research reports, and forum discussions to search for "the most important financial leverage tools in recent years."

[0014] 2) Intelligent Analysis of Film and Television Marketing Strategies: When formulating publicity and distribution strategies, marketing teams need to analyze market trends, audience preferences, and competitor performance. For example, when asking the question "What is the most effective publicity channel for suspense dramas?", the information retrieved by the system may include a variety of different or even outdated viewpoints (such as "WeChat, Weibo, and Douyin are the way to go" versus "Short drama platform advertising is a new blue ocean").

[0015] 3) Fact-checking for documentary / historical content: The accuracy of historical materials is extremely important when producing documentaries or historical dramas. When researching a historical event, different documents, memoirs, or research papers may contain discrepancies in details or even contradictory accounts.

[0016] This invention aims to address the problem that existing retrieval enhancement generation technologies fail to generate accurate, reliable, and logically consistent answers when faced with multi-source, heterogeneous, and even conflicting retrieval information. This invention proposes a multi-stage, strategy-driven knowledge conflict resolution method. This method innovatively introduces two core processing modules into the traditional two-stage "retrieval-generation" model: "knowledge conflict detection and classification" and "conflict resolution strategy selection." First, it automatically identifies contradictions between retrieval information and determines their conflict type (e.g., timeliness conflict, differing viewpoints). Then, based on preset rules, it matches the optimal answer generation strategy for different types of conflicts, ultimately guiding the large language model to produce high-quality, reliable answers based on this strategy.

[0017] Figure 1 This is a flowchart of a question-and-answer method provided in an embodiment of the present invention. This question-and-answer method can be applied to electronic devices such as mobile phones and tablets. Figure 1 As shown, the method may include steps 110 to 140.

[0018] Step 110: Based on the user's query, multiple documents are retrieved.

[0019] It receives user queries and retrieves a set (N) of the most relevant documents from one or more specified knowledge sources. These knowledge sources may include search engines and / or pre-created knowledge bases.

[0020] The retrieval process can include query understanding, source retrieval, and content extraction. Query understanding involves parsing the user's natural language query, extracting keywords, entities, and intent. Intent represents the type of knowledge being retrieved, such as scripts or historical content. Source retrieval involves sending the parsed user query to one or more search engine APIs (such as Google Search) or internal document databases (such as script libraries or market report libraries, implemented through a vector search engine). Content extraction involves extracting the title, summary, key paragraphs, and metadata (such as publication date and source website) from each returned link or document. These N extracted documents are then structured into a set of documents, organized as key-value pairs. Keys can include titles, summaries, key paragraphs, and metadata, while values ​​represent the specific content corresponding to the key. This facilitates subsequent determination of knowledge conflict types.

[0021] Step 120: Based on the user query, determine the target conflict type of the multiple documents and the corresponding explanation information based on the target conflict type using a large language model.

[0022] Among them, the target conflict type represents the type of knowledge conflict between the content of multiple documents, and the explanatory information is used to explain the reasons for the knowledge conflict between the multiple documents.

[0023] A large language model can be used to analyze the content of multiple retrieved documents to determine whether knowledge conflicts exist between them and identify the specific types of knowledge conflicts as the target conflict type. Simultaneously, the large language model can provide explanatory information corresponding to the target conflict type. For example, the large language model can be used as a classifier to categorize the knowledge conflict relationships between the content of multiple documents based on user queries, thereby obtaining the target conflict type for each document.

[0024] Target conflict types can include: no conflict, complementary information, differing viewpoints or research, timeliness conflict, or factual error. The definitions of each conflict type are as follows: no conflict indicates consistency of information across documents; complementary information indicates that documents provide different perspectives that are not contradictory but rather complementary; differing viewpoints or research indicates that documents present different viewpoints on subjective or controversial issues; timeliness conflict indicates inconsistencies in factual descriptions between documents due to outdated information; and factual error indicates that some documents contain generally accepted errors.

[0025] Step 130: Obtain the target answer generation strategy corresponding to the target conflict type from the mapping relationship between conflict types and answer generation strategies.

[0026] The answer generation strategy is a method used to resolve knowledge conflicts between multiple documents when generating answers. Each conflict type corresponds to its own answer generation strategy. For example, the mapping relationship between conflict types and answer generation strategies can be represented as follows: If the conflict type is "no conflict" or "complementary information", the answer generation strategy is "comprehensive statement: comprehensively and objectively integrate all information"; If the conflict type is "disagreement on viewpoints or research", the answer generation strategy is "parallel presentation: neutrally summarize the main viewpoints of each party without favoring any one party"; If the conflict type is "time-sensitive conflict", the answer generation strategy is "adopt the latest: prioritize the information from the document with the most recent publication date, and selectively mention historical information as a timeline"; If the conflict type is "factual error", the answer generation strategy is "falsification and clarification: adopt the information with high credibility and clearly point out the errors in other information".

[0027] After determining the target conflict type of multiple documents, the target answer generation strategy corresponding to the target conflict type can be obtained from the mapping relationship between conflict type and answer generation strategy.

[0028] The aforementioned mapping is configurable, hard-linking the conflict type identified in the previous step to a clear, optimal answer generation paradigm. For example, once "disagreement" is identified, subsequent generation models are forced to adopt a "neutral and declarative" tone. This mapping significantly enhances the controllability, predictability, and consistency of the AI ​​system's output behavior, avoiding the uncertainty in behavior of traditional RAG models when faced with contradictory information. It ensures that the style and structure of the output are always best suited to the current context, thereby significantly improving the usability of the answers and user satisfaction.

[0029] For ambiguous or highly complex conflicts, i.e., situations where a clear answer cannot be obtained through a large language model based on the answer generation strategy, clarifying questions can be posed to the user (e.g., "Two statements about this event have been detected. Which perspective do you prefer to adopt?"), incorporating the user's judgment into the decision-making loop to improve the accuracy of the generated answer.

[0030] Step 140: Using a large language model, generate an answer corresponding to the user's query based on the first prompt information and the multiple documents.

[0031] The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents.

[0032] The initial prompt information and the retrieved multiple documents are input into the large language model. The target answer generation strategy and explanation information in the initial prompt information guide the large language model to resolve knowledge conflicts between multiple documents when generating answers corresponding to user queries based on multiple documents, thereby generating accurate answers and reducing conflicting information.

[0033] Step 150: Output the answer.

[0034] The obtained answers can be displayed on a screen, and / or the text-based answers can be converted into speech and played back via a microphone.

[0035] The question-answering method provided in this invention, after retrieving multiple documents based on a user query, determines the target conflict type and corresponding explanatory information of the multiple documents, obtains the target answer generation strategy corresponding to the target conflict type, and then generates an answer corresponding to the user query using a large language model based on the first prompt information including the target answer generation strategy and explanatory information, and the multiple documents. In this way, the target answer generation strategy can resolve knowledge conflicts between different documents, reduce conflicting information in the generated answer, and improve the accuracy of the generated answer.

[0036] Based on the above technical solution, the step of determining the target conflict type of the multiple documents, i.e., step 120 above, may include: determining the target conflict type of the multiple documents based on the user query using the large language model.

[0037] A large language model can be used as a classifier to classify the knowledge conflict relationships between the content of multiple documents based on user queries, thereby obtaining the target conflict type of multiple documents.

[0038] Determining the target conflict type of multiple documents using a large language model can improve the accuracy and flexibility of conflict type determination.

[0039] In some embodiments of the present invention, a large language model is used to determine the target conflict type of the multiple documents and the corresponding explanatory information based on the user query. That is, step 120 above may include: filling the user query and the multiple documents into a prompt template to obtain second prompt information. The prompt template further includes prompt words, which include definitions of multiple conflict types and examples corresponding to each conflict type. The second prompt information is input into the large language model, and the large language model determines the target conflict type of the multiple documents and the corresponding explanatory information based on the prompt words.

[0040] A large language model can be used as a classifier. In the prompt words of the large language model, multiple knowledge conflict types are clearly defined and examples are provided. That is, the prompt template provides definitions of multiple conflict types and examples corresponding to each conflict type, and specifies the filling positions of the user query and the document. The obtained user query is filled into the corresponding positions in the prompt template, and the content of multiple retrieved documents is filled into the corresponding positions in the prompt template to obtain the second prompt information.

[0041] The second prompt information is input into the large language model. Based on the prompt words, the large language model analyzes the knowledge conflicts between the document content of multiple documents according to the user query, determines the target conflict type of multiple documents, and outputs a structured result including the number of the target conflict type and the explanation information. The structured result can be represented as key-value pairs. The key can include the conflict type, and the corresponding value is the number of the target conflict type. The key can also include the explanation, and the corresponding value is the specific explanation information. For example, the structured result output by the large language model can be in JSON format: {"category": 4, "explanation": "The descriptions of the CEO candidate in documents [1] and [3] are contradictory due to different publication times."}, where category represents the conflict type, the corresponding value 4 is the number of the target conflict type, and explanation represents the explanation, the specific explanation information being "The descriptions of the CEO candidate in documents [1] and [3] are contradictory due to different publication times.".

[0042] By populating the prompt template with user queries and multiple documents, and including the definition and examples of conflict types in the prompt words of the prompt template, it is easier to guide the large language model to classify the knowledge conflicts between multiple documents according to the guidance of the prompt words, and obtain the accurate target conflict type.

[0043] This invention goes beyond simply identifying textual differences. It designs a conflict classification system encompassing multiple predefined types and leverages the contextual understanding capabilities of a large language model to perform deep semantic analysis on retrieved multi-source information. This automatically determines the existence and fundamental nature of conflicts (such as timeliness and opinion). This represents a leap from "information presentation" to "knowledge understanding," enabling the system to, for the first time, understand the underlying causes of information contradictions. This lays the foundation for subsequent targeted and intelligent solutions, fundamentally improving the depth and accuracy of problem-solving.

[0044] Based on the above technical solution, the step 140, which generates an answer corresponding to the user query using a large language model based on the first prompt information and the multiple documents, includes: inputting the first prompt information, the user query, and the multiple documents into the large language model; and using the large language model to generate an answer corresponding to the user query by fusing information from the multiple documents based on the target answer generation strategy and the explanation information.

[0045] A special, conflict-aware cue word structure can be pre-designed. This cue word first explicitly instructs the large language model on its current role and the generation strategy it should follow. Corresponding cue word structures can be pre-set according to the answer generation strategy for each conflict type. For example, "Based on the following information, please prioritize using the latest data to answer the question..." or "Please neutrally summarize the following two opposing viewpoints...".

[0046] Based on the target answer generation strategy corresponding to the type of target conflict, the first hint information corresponding to that strategy can be obtained. This first hint information, the user query, and multiple documents are then input into the large language model. Guided by the target answer generation strategy explicitly stated in the first hint information, the large language model generates a logically clear, focused answer that conforms to the expected style, based on the user query and multiple documents. For example, for timeliness conflicts, the large language model will naturally provide the latest facts; for differing opinions, it will clearly list the arguments for and against.

[0047] By identifying the second hint information corresponding to the answer generation strategy, and then using the guidance of the second hint information, the large language model can generate accurate answers based on user queries and multiple documents, thus solving the problem of information conflict in the answers generated in the prior art and improving the accuracy of the generated answers.

[0048] In some embodiments of the present invention, the target conflict type includes a timeliness conflict, and the target answer generation strategy includes adopting the latest strategy; the step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and generating an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: sorting the multiple documents in descending order according to their publication time; inputting the first prompt information, the user query, and the descendingly sorted multiple documents into the large language model, and generating an answer corresponding to the user query by using information from the first-ranked document among the multiple documents according to the latest strategy and the explanation information.

[0049] After determining the target conflict type, the retrieved documents can be further filtered and sorted according to the target answer generation strategy corresponding to that type to select the core evidence for generating the answer. When the target conflict type is a time-sensitive conflict, the documents can be sorted in descending order of publication time, with the newest document marked as the primary source. The initial prompt and the descendingly sorted documents are input into a large language model. Based on the adoption of the latest strategy and explanatory information, the large language model uses the document ranked first in the descending order—that is, the most recently published document—to generate the final answer.

[0050] In one exemplary embodiment, for other conflict types, such as no conflict, complementary information, differing opinions or research findings, since there is no knowledge conflict between these documents, the information of different documents can be fully displayed without document screening. All retrieved documents are used as core evidence to input into the large language model to generate the final answer. For factual errors, documents with factual errors can be marked, and the marked documents are used as evidence to input into the large language model.

[0051] When the target conflict type is a time-sensitive conflict, multiple documents are sorted in descending order according to their publication time. This makes it easier for the large language model to determine the final answer based on the publication time, thus resolving the time-sensitive conflict problem between different documents.

[0052] In other embodiments of the present invention, the target conflict type includes conflict-free or complementary information, the complementary information indicating that the information in the plurality of documents complements each other, and the target answer generation strategy includes a comprehensive statement strategy; The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and generating an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: inputting the first prompt information, the user query, and the multiple documents into the large language model, and generating an answer corresponding to the user query by fusing information from the multiple documents according to the comprehensive statement strategy and the explanation information.

[0053] Among them, no conflict means that there is no knowledge conflict between multiple documents.

[0054] When the target conflict type is non-conflicting or complementary information, the first prompt information, user query, and multiple documents can be input into the large language model. The large language model comprehensively and objectively integrates the information in all documents based on the comprehensive statement strategy and explanatory information in the first prompt information to generate an answer corresponding to the user query. This can improve the comprehensiveness and accuracy of the generated answer.

[0055] In other embodiments of the present invention, the target conflict type includes disagreement, and the target answer generation strategy includes a parallel display strategy; The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and generating an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: inputting the first prompt information, the user query, and the multiple documents into the large language model, and generating an answer corresponding to the user query by using information from the multiple documents in parallel according to the parallel display strategy and the explanation information.

[0056] The "disagreement" indicator indicates that there are differing viewpoints among multiple documents. The "parallel presentation" strategy indicates that the main viewpoints of each document are neutrally summarized without favoring any one side.

[0057] The initial prompt, user query, and multiple documents are input into a large language model. Based on the parallel display strategy and explanatory information, the large language model neutrally summarizes the information from multiple documents to obtain the answer corresponding to the user query. The generated answer can present the viewpoints of different documents, making it easy for users to refer to.

[0058] In some other embodiments of the present invention, the target conflict type includes factual error, and the target answer generation strategy includes a falsification clarification strategy; The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and using the large language model to fuse information from the multiple documents according to the target answer generation strategy and the explanation information to generate an answer corresponding to the user query, includes: inputting the first prompt information, the user query, and the multiple documents into the large language model; using the large language model to determine the correct information in the multiple documents according to the explanation information; and using the correct information according to the falsification and clarification strategy to generate an answer corresponding to the user query.

[0059] In this framework, factual errors indicate that some documents within a multi-document set contain incorrect information. The falsification and clarification strategy indicates the use of information with high credibility from among the multiple documents. The credibility of different documents can be determined based on the weight of the data source to which they correspond; the higher the weight of the data source, the higher the credibility of the corresponding document. Explanatory information can be based on the weight of the data source to determine whether the information in each document is correct.

[0060] The initial prompt, user query, and multiple documents are input into a large language model. Based on the explanation information, the large language model determines the correct information from the multiple documents. Then, based on the falsification and clarification strategy, the correct information is summarized to obtain the answer corresponding to the user query. In this way, the generated answer no longer borrows from incorrect information, which can improve the accuracy of the generated answer.

[0061] Figure 2 This is a flowchart of another question-and-answer method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment can further mark the document identifier corresponding to the content in the answer, and verify the answer based on the document identifier. For example... Figure 2 As shown, the method may include steps 210 to 280.

[0062] Step 210: Based on the user's query, retrieve multiple documents.

[0063] Step 220: Based on the user query, determine the target conflict type of the multiple documents and the corresponding explanation information based on the target conflict type using a large language model.

[0064] Step 230: Obtain the target answer generation strategy corresponding to the target conflict type from the mapping relationship between conflict types and answer generation strategies.

[0065] Step 240: Based on the first prompt information and the multiple documents, generate an answer corresponding to the user's query using a large language model.

[0066] Step 250: When generating the answer using the large language model, mark the document identifier corresponding to the content in the answer.

[0067] The prompts used to guide the large language model to generate answers may also include prompts that require the model to mark the document identifiers corresponding to the content in the answer. In this way, when guiding the large language model to generate answers based on the answer generation strategy, the source of the content in the answer will also be marked, that is, the document identifiers corresponding to the content will be marked. The document identifiers may be document numbers (such as...[1]).

[0068] Step 260: Extract the answer content corresponding to the document identifier from the answer, and determine the semantic similarity between the answer content and the document content corresponding to the document identifier.

[0069] Based on the document identifier, obtain the document content corresponding to the document identifier, and verify the corresponding content in the answer based on the document content to obtain the verification result.

[0070] The document content corresponding to the document identifier is the document content referenced by the answer content.

[0071] When large language models generate answers based on document content, they need to summarize the information from each document to arrive at the final answer. The generated answer may be consistent with the document content, or some parts of the answer may be unsupported. Therefore, answer validation is necessary. This can be achieved by extracting the answer content corresponding to the document identifier and comparing its semantic similarity with the content of the referenced document.

[0072] Step 270: If the semantic similarity is greater than or equal to the similarity threshold, the answer content verification is determined to be successful; if the semantic similarity is less than the similarity threshold, the answer content verification is determined to be unsuccessful.

[0073] If the semantic similarity between the answer content and the document content is greater than or equal to the similarity threshold, the answer content passes the verification. If the semantic similarity is less than the similarity threshold, meaning the answer content cannot be supported in the document corresponding to the document identifier or contains misinterpretations, the answer content fails the verification. By verifying answers based on semantic similarity, it is possible to accurately determine whether the answer content is consistent with the document content, thereby improving the accuracy of the verification.

[0074] Step 280: Output the answer based on the verification result of the answer content.

[0075] In one optional implementation, the verification result of an answer can be determined based on the verification results of all answer content in the answer. If all answer content in the answer passes verification, the answer is considered verified. If the answer contains both verified and failed answer content, the verification result can be considered partially passed. If all answer content in the answer fails verification, the verification result is considered failed. When an answer passes verification, it can be directly output. When an answer fails verification, the above steps can be repeated to regenerate the answer and perform verification again. When the verification result is partially passed, the answer can be output.

[0076] The question-answering method provided in this embodiment generates answers using a large language model, marks the document identifiers corresponding to the content in the answer, verifies the answer based on the document identifiers, and outputs the answer based on the verification results. In this way, the verification can output more accurate answers, further improving the accuracy of the answers.

[0077] Based on the above technical solution, the step of outputting the answer according to the verification result of the answer, i.e., the above step 280, includes: if the answer content verification fails, supplementing the answer with third prompt information and outputting the answer with the supplemented third prompt information, wherein the third prompt information is used to indicate that the confidence level of the answer content is low.

[0078] In one exemplary embodiment, when the answer content fails verification, a hint indicating low confidence of the answer content can be added to the answer; alternatively, the answer can be regenerated and verified.

[0079] When an answer fails validation, adding information indicating low confidence in the answer can provide users with relevant hints, making it easier for them to make a judgment.

[0080] Based on the above technical solution, the method may further include: generating a citation list corresponding to the document identifier, the citation list including the title and metadata corresponding to the document identifier; and outputting the citation list when outputting the answer.

[0081] The metadata may include publication time, source website, etc.

[0082] When the answer references the content of a document corresponding to a document identifier, the document information corresponding to that document identifier can be obtained, and a citation list corresponding to the document identifier can be generated. When outputting the answer, the citation list is also output, which makes it easier for users to judge the accuracy of the answer based on the citation list.

[0083] In some embodiments of the present invention, generating an answer corresponding to the user query using a large language model based on the first prompt information and the multiple documents includes: using the large language model to fuse information from the multiple documents based on the first prompt information, the multiple documents, and the weights of the data sources corresponding to each document, to generate an answer corresponding to the user query.

[0084] A data source authority assessment mechanism can be introduced to assign different weights to different data. For example, when processing film and television information, official releases and mainstream media reports should have a higher weight than personal blogs or forum posts. Large language models can use the weights of different data sources to make weighted decisions on conflicting information in different documents, further improving the reliability of the answer (e.g., when there is a factual error, the weights of different data sources can be referenced to generate the final answer).

[0085] In some embodiments of the present invention, before generating the answer corresponding to the user query based on the first prompt information and the multiple documents using a large language model, the method may further include: obtaining a prompt word structure including the target answer generation strategy, filling the explanatory information into the prompt word structure, and obtaining the first prompt information.

[0086] The system can obtain a prompt word structure, including the target answer generation strategy, from the pre-set prompt word structures. Explanatory information is then filled into this prompt word structure to obtain the first prompt information. This first prompt information can guide the large language model to integrate information from different documents based on the target answer generation strategy and the explanatory information to generate the final answer, thereby improving the accuracy of the answer.

[0087] This invention does not simply input information into a large model. Instead, it strongly constrains the model's behavior before generation through "strategy-injected prompts (including second prompt information for the answer generation strategy)" and conducts reliability reviews and corrections on the output results after generation through citation management and consistency verification. This constructs a closed-loop quality control system, which not only ensures that the final answer is faithful to reliable sources in terms of content but also guarantees its high traceability in terms of form. For scenarios with strict requirements such as content creation, this mechanism can effectively reduce factual errors, lower compliance risks, and build users' high trust in the AI ​​system.

[0088] The following specific example illustrates the processing procedure of the question-and-answer method provided in this embodiment of the invention.

[0089] For example, to search for "Who is the director of the movie series 'XXXX'?": User query: "Who is the director of the 'XXXX' movie series?" Module 1 (Search): The system retrieved 3 documents: Document [1] (from a certain encyclopedia, updated in 2020): "The director of 'XXXX' is Zhang San." Document [2] (a movie information website, 2023 article): "XXXX2 continues the previous work and is still directed by Zhang San. Document [3] (a forum post, 2019): "There are rumors that the director of 'XXXX2' may be changed, and Li Si expressed his appreciation." Module 2 (Detection and Classification): The system submits the user query and three documents to the classification model. The model analysis found that the information in document [1] and document [2] is consistent, but document [3] presents uncertain information that is inconsistent with subsequent facts. The model determines that this is a mixture of "timeliness conflict" and minor "factual error", but the main contradiction is outdated information, and the final output conflict type is "timeliness conflict" (type four).

[0090] Module 3 (Strategy Selection): The system matches the "Timeliness Conflict" type and activates the "Adopt the Latest" strategy. Based on this strategy, documents [1] and [2] are marked as core evidence because they have consistent information and updated records, while document [3] is ignored as outdated information.

[0091] Module 4 (Answer Generation): The system sends an instruction to the generation model: "Based on the following information, please adopt the latest and most accurate statement to answer who is the director of the 'XXXX' series." Then attach the contents of document [1] and document [2]. The model generates a preliminary answer: "The director of both movies in the 'XXXX' series is Zhang San [1][2]". Module 5 (Verification and Output): The system verifies that the statement "the director is Zhang San" in the answer is strongly supported in documents [1] and [2]. The citation format is correct. Finally, the answer is output to the user: The director of both films in the "XXXX" series is Zhang San[1][2].

[0092] References [1] According to a certain encyclopedia, the entry for "XXXX" was published in 2020.

[0093] [2] A movie information website, “Review of “XXXX2”, 2023.

[0094] Figure 3 This is a schematic diagram of a question-and-answer system provided in an embodiment of the present invention, such as... Figure 3 As shown, this embodiment of the invention proposes a question-answering system comprising five sequentially connected modular processing flows, aiming to solve the knowledge conflict problem encountered by large language models when answering questions using retrieved information. The core idea is as follows: First, the retrieval enhancement query processing module retrieves a set of relevant documents from external knowledge sources (such as the Internet or internal databases) based on user input. Then, the knowledge conflict detection and classification module reviews these documents, automatically identifying any inconsistencies and determining the nature of the conflict based on a predefined conflict classification system (e.g., classifying conflicts as "time-sensitive conflicts," "disagreements," or "factual errors"). Next, the conflict resolution strategy selection module matches and activates an optimal answer generation strategy from a pre-defined strategy library (the mapping relationship mentioned above) based on the conflict type output by the previous module. For example, for "time-sensitive conflicts," a "prioritize the latest information" strategy is used, and for "disagreements," a "neutral presentation of all parties" strategy is used. Then, the conflict-aware answer generation module receives the selected answer generation strategy and filtered original information, using the strategy to guide the large language model to generate the final answer, ensuring that the answer's expression meets the requirements for addressing the current conflict type. Finally, the citation management and quality control module verifies the generated answer, ensuring that the content is faithful to the original text and includes accurate citation sources. Through this series of interconnected processes, the embodiments of the present invention can transform originally chaotic and contradictory original information into a logically clear, focused, reliable, and credible final answer.

[0095] The question-and-answer method provided in this invention can serve as the core engine of an "AI screenwriter / planning assistant." When a creative team is researching background information, the system can quickly provide clear and concise summaries, freeing them from tedious and repetitive information vetting. For example, when researching a historical figure, the system can automatically integrate conflicting accounts from different historical materials and provide the most credible biographical outline. This method directly shortens the early research cycle of a project and reduces labor costs. More importantly, by providing higher-quality information input (including answer generation strategies), it stimulates creative inspiration and reduces the risk of script rework due to factual errors, thereby improving the quality and reputation of the final work.

[0096] Figure 4 This is a schematic diagram of the structure of a question-and-answer device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes: The retrieval module 410 is used to retrieve multiple documents based on the user's query; The conflict classification module 420 is used to determine the target conflict type of the multiple documents and the explanation information corresponding to the target conflict type based on the user query using a large language model. The target conflict type represents the knowledge conflict type between the document content of the multiple documents, and the explanation information is used to explain the reason for the knowledge conflict in the multiple documents. The strategy acquisition module 430 is used to acquire the target answer generation strategy corresponding to the target conflict type from the mapping relationship between conflict types and answer generation strategies; The answer generation module 440 is used to generate an answer corresponding to the user query based on the first prompt information and the multiple documents using a large language model. The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents. The answer output module 450 is used to output the answer.

[0097] Optionally, the conflict classification module includes: The second prompt information generation unit is used to fill the user query and the multiple documents into a prompt template to obtain the second prompt information. The prompt template also includes prompt words, which include definitions of multiple conflict types and examples corresponding to each conflict type. The conflict classification unit is used to input the second prompt information into the large language model, and the large language model determines the target conflict type of the multiple documents and the explanation information corresponding to the target conflict type based on the prompt words.

[0098] Optionally, the answer generation module includes: The first answer generation unit is used to input the first prompt information, the user query, and the multiple documents into the large language model, and generate an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information.

[0099] Optionally, the target conflict type includes timeliness conflict, and the target answer generation strategy includes adopting the latest strategy; The first answer generation unit is specifically used for: Sort the documents in descending order according to their publication time; The first prompt information, the user query, and the multiple documents sorted in descending order are input into the large language model. The large language model then uses information from the first-ranked document among the multiple documents, based on the latest adoption strategy and the explanation information, to generate an answer corresponding to the user query.

[0100] Optionally, the target conflict type includes conflict-free or complementary information, the complementary information indicating that the information in the multiple documents complements each other, and the target answer generation strategy includes a comprehensive statement strategy; The first answer generation unit is specifically used for: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model then integrates the information from the multiple documents based on the comprehensive statement strategy and the explanation information to generate an answer corresponding to the user query.

[0101] Optionally, the target conflict type includes differing opinions, and the target answer generation strategy includes a parallel display strategy; The first answer generation unit is specifically used for: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model then uses information from the multiple documents in parallel according to the parallel display strategy and the explanation information to generate an answer corresponding to the user query.

[0102] Optionally, the target conflict type includes factual error, and the target answer generation strategy includes a falsification clarification strategy; The first answer generation unit is specifically used for: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model determines the correct information in the multiple documents based on the explanation information, and generates an answer corresponding to the user query using the correct information according to the falsification and clarification strategy.

[0103] Optionally, the device further includes: The tagging module is used to tag the document identifiers corresponding to the content in the answer when the answer is generated by the large language model. The answer output module includes: A similarity determination unit is used to extract the answer content corresponding to the document identifier from the answer, and determine the semantic similarity between the answer content and the document content corresponding to the document identifier. The answer verification unit is used to determine that the answer content verification passes if the semantic similarity is greater than or equal to the similarity threshold, and to determine that the answer content verification fails if the semantic similarity is less than the similarity threshold. The answer output unit is used to output the answer based on the verification result of the answer content.

[0104] Optionally, the answer output unit is specifically used for: If the answer content fails the verification, a third hint is added to the answer, and the answer with the added third hint is output. The third hint is used to indicate that the confidence level of the answer content is low.

[0105] Optionally, the device further includes: A citation list generation module is used to generate a citation list corresponding to the document identifier, wherein the citation list includes the title and metadata corresponding to the document identifier; The citation list output module is used to output the citation list when outputting the answer.

[0106] Optionally, the answer generation module includes: The second answer generation unit is used to generate an answer corresponding to the user query by fusing information from the multiple documents based on the first prompt information, the multiple documents, and the weights of the data sources corresponding to each document using the large language model.

[0107] Optionally, the device further includes: The first prompt information generation unit is used to obtain a prompt word structure including the target answer generation strategy, fill the explanation information into the prompt word structure, and obtain the first prompt information.

[0108] The question-answering device provided in this embodiment of the invention, after retrieving multiple documents based on a user query, determines the target conflict type of the multiple documents and the explanatory information corresponding to the target conflict type, obtains the target answer generation strategy corresponding to the target conflict type, and then generates an answer corresponding to the user query based on the first prompt information including the target answer generation strategy and explanatory information and the multiple documents through a large language model. In this way, the knowledge conflict between different documents can be resolved based on the target answer generation strategy, reducing conflicting information in the generated answer and improving the accuracy of the generated answer.

[0109] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. Memory 503 is used to store computer programs; When processor 501 executes the program stored in memory 503, it performs the following steps: Based on the user's query, multiple documents were retrieved. Based on the user query using a large language model, the target conflict type of the multiple documents and the corresponding explanatory information are determined. The target conflict type represents the type of knowledge conflict between the document content of the multiple documents, and the explanatory information is used to explain the reasons for the knowledge conflict in the multiple documents. From the mapping relationship between conflict types and answer generation strategies, obtain the target answer generation strategy corresponding to the target conflict type; Based on the first prompt information and the multiple documents, a large language model is used to generate an answer corresponding to the user's query. The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents. Output the answer.

[0110] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0111] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0112] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0113] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0114] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the question-and-answer methods described in the above embodiments.

[0115] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the question-and-answer methods described in the above embodiments.

[0116] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0119] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A question and answer method, characterized by, include: Based on the user's query, multiple documents were retrieved. Based on the user query using a large language model, the target conflict type of the multiple documents and the corresponding explanatory information are determined. The target conflict type represents the type of knowledge conflict between the document content of the multiple documents, and the explanatory information is used to explain the reasons for the knowledge conflict in the multiple documents. From the mapping relationship between conflict types and answer generation strategies, obtain the target answer generation strategy corresponding to the target conflict type; Based on the first prompt information and the multiple documents, a large language model is used to generate an answer corresponding to the user's query. The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents. Output the answer.

2. The method of claim 1, wherein, Based on the user query using a large language model, the target conflict type of the multiple documents and the corresponding explanatory information are determined, including: The user query and the multiple documents are populated into the prompt template to obtain the second prompt information. The prompt template also includes prompt words, which include definitions of multiple conflict types and examples corresponding to each conflict type. The second prompt information is input into the large language model, and the large language model determines the target conflict type of the multiple documents and the corresponding explanation information based on the prompt words.

3. The method of claim 1, wherein, The step of generating an answer corresponding to the user's query using a large language model based on the first prompt information and the multiple documents includes: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model then integrates the information from the multiple documents based on the target answer generation strategy and the explanation information to generate an answer corresponding to the user query.

4. The method of claim 3, wherein, The target conflict types include timeliness conflicts, and the target answer generation strategy includes adopting the latest strategy; The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and then using the large language model to generate an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: Sort the documents in descending order according to their publication time; The first prompt information, the user query, and the multiple documents sorted in descending order are input into the large language model. The large language model then uses information from the first-ranked document among the multiple documents, based on the latest adoption strategy and the explanation information, to generate an answer corresponding to the user query.

5. The method according to claim 3, characterized in that, The target conflict type includes conflict-free or complementary information, the complementary information indicating that the information in the multiple documents complements each other, and the target answer generation strategy includes a comprehensive statement strategy. The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and then using the large language model to generate an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model then integrates the information from the multiple documents based on the comprehensive statement strategy and the explanation information to generate an answer corresponding to the user query.

6. The method according to claim 3, characterized in that, The target conflict types include differing opinions, and the target answer generation strategy includes a parallel display strategy; The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and then using the large language model to generate an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model then uses information from the multiple documents in parallel according to the parallel display strategy and the explanation information to generate an answer corresponding to the user query.

7. The method according to claim 3, characterized in that, The target conflict type includes factual error, and the target answer generation strategy includes falsification and clarification strategy; The step of inputting the first prompt information, the user query, and the multiple documents into the large language model, and then using the large language model to generate an answer corresponding to the user query by fusing information from the multiple documents according to the target answer generation strategy and the explanation information, includes: The first prompt information, the user query, and the multiple documents are input into the large language model. The large language model determines the correct information in the multiple documents based on the explanation information, and generates an answer corresponding to the user query using the correct information according to the falsification and clarification strategy.

8. The method according to any one of claims 1-7, characterized in that, Before outputting the answer, the following is also included: When generating the answer using the large language model, the document identifier corresponding to the content in the answer is marked; The output of the answer includes: Extract the answer content corresponding to the document identifier from the answer, and determine the semantic similarity between the answer content and the document content corresponding to the document identifier; If the semantic similarity is greater than or equal to the similarity threshold, the answer content verification is determined to be successful; if the semantic similarity is less than the similarity threshold, the answer content verification is determined to be unsuccessful. Based on the verification results of the answer content, output the answer.

9. The method according to claim 8, characterized in that, The step of outputting the answer based on the verification result of the answer content includes: If the answer content fails the verification, a third hint is added to the answer, and the answer with the added third hint is output. The third hint is used to indicate that the confidence level of the answer content is low.

10. The method according to claim 8, characterized in that, Also includes: Generate a citation list corresponding to the document identifier, wherein the citation list includes the title and metadata corresponding to the document identifier; When outputting the answer, the list of citations is also output.

11. The method according to any one of claims 1-7, characterized in that, The step of generating an answer corresponding to the user's query using a large language model based on the first prompt information and the multiple documents includes: The large language model integrates the information from the first prompt, the multiple documents, and the weights of the data sources corresponding to each document to generate an answer that corresponds to the user's query.

12. The method according to any one of claims 17, characterized in that, Before generating the answer corresponding to the user query using a large language model based on the first prompt information and the multiple documents, the method further includes: Obtain the prompt word structure including the target answer generation strategy, and fill the explanatory information into the prompt word structure to obtain the first prompt information.

13. A question-and-answer device, characterized in that, include: The search module is used to retrieve multiple documents based on user queries; The conflict classification module is used to determine the target conflict type of the multiple documents and the corresponding explanatory information based on the user query using a large language model. The target conflict type represents the knowledge conflict type between the document content of the multiple documents, and the explanatory information is used to explain the reasons for the knowledge conflict between the multiple documents. The strategy acquisition module is used to acquire the target answer generation strategy corresponding to the target conflict type from the mapping relationship between conflict types and answer generation strategies; The answer generation module is used to generate an answer corresponding to the user query based on the first prompt information and the multiple documents using a large language model. The first prompt information includes the target answer generation strategy and the explanation information. The first prompt information is used to provide a way to resolve knowledge conflicts between the multiple documents. The answer output module is used to output the answer.

14. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-12.

16. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1-12.

Images