A method and related apparatus for fact-level relation annotation and reliable response generation in the context of knowledge conflict

By extracting self-fact sets and annotating fact-level conflict relationships from large language models, the credibility and interpretability issues of the models in knowledge conflict scenarios are resolved, and more credible response generation is achieved.

CN123115388APending Publication Date: 2026-11-17XI AN JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611387273.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-08
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

When faced with multi-source knowledge conflicts, large language models struggle to explicitly model the factual relationships between different knowledge sources, leading to the generation of unreliable or uninterpretable responses.

Method used

By extracting facts from a set of facts and labeling the conflict relationships between different knowledge sources, a credible response containing reasoning and a final answer is generated.

Benefits of technology

This improves the credibility and interpretability of language models in knowledge conflict scenarios, and reduces the risk of models over-relying on their own knowledge or blindly adopting external context.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115388A_ABST
    Figure CN123115388A_ABST
Patent Text Reader

Abstract

This invention belongs to the fields of artificial intelligence, natural language processing, knowledge conflict detection, and credible response generation. It discloses a method and related apparatus for fact-level relation labeling and credible response generation oriented towards knowledge conflict. The method involves: acquiring a sample to be answered; extracting a fact set from the user's question; preprocessing the retrieval context to obtain a context chunk set; semantically aligning the fact set with the context chunk set to obtain a candidate fact-context pair set; labeling the candidate fact-context pair set with fact-level conflict relations to obtain relation labels, conflict scores, confidence levels, and labeling reasons; constructing a conflict-aware evidence set based on the relation labels, conflict scores, confidence levels, and labeling reasons; and inputting the user's question, the original context, a set of possible candidate answers, and the conflict-aware evidence set into a conflict-aware chain-like thinking generation model to generate a credible response containing reasoning and a final answer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, natural language processing, knowledge conflict detection and trusted response generation, and specifically relates to a method and related apparatus for fact-level relation annotation and trusted response generation for knowledge conflict. Background Technology

[0002] With the widespread application of large language models in intelligent question answering, intelligent customer service, knowledge management, decision support and intelligent agent systems, language models usually need to integrate multiple knowledge sources when generating responses, including knowledge implicit in the model's own parameters, knowledge provided by external knowledge bases, textual evidence returned by retrieval systems, tool call results and historical interaction information.

[0003] However, inconsistencies may exist between different knowledge sources. For example, the model's own parameter knowledge may be outdated, external knowledge bases may contain errors, retrieval systems may return fragments that are relevant to the question but inaccurate in fact, tool call results may be limited by interface or data update time, and historical interaction information may conflict with current real-time information. When language models encounter these knowledge conflicts, two types of problems are prone to occur: first, over-reliance on their own existing knowledge while ignoring external evidence; second, blindly adopting external knowledge and introducing erroneous information.

[0004] Existing methods typically enhance a model's ability to utilize external information through cue word constraints, contextual enhancement, or decoding strategy adjustments. However, most methods lack explicit modeling of factual relationships between different knowledge sources, making it difficult to determine whether a knowledge fragment supports, conflicts with, is irrelevant to, or remains undetermined with another. Therefore, in knowledge conflict scenarios, the model may still generate inconsistent, unreliable, or uninterpretable responses. Thus, there is an urgent need for a method that can explicitly align, label relationships, and perform conflict-aware reasoning on factual units from different knowledge sources before generating a response, thereby improving the credibility, robustness, and interpretability of language models in knowledge conflict scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a method and related apparatus for fact-level relation annotation and credible response generation oriented towards knowledge conflict. This invention extracts content from different knowledge sources into fact units, annotates the relationships between fact units with fact-level conflict relations, and uses structured relation labels and conflict scores as control information in the response generation stage, thereby reducing the risk of language models blindly adopting erroneous knowledge or over-relying on a single knowledge source.

[0006] A method for fact-level relation annotation and trusted response generation in response to knowledge conflicts includes the following process: Obtain a sample to be answered, which includes a user question Q, a search context C related to the user question, and a set of possible candidate answers O; The facts were extracted from the user question Q. ; The search context C is cleaned by removing bracket content, segmenting sentences, normalizing whitespace characters, and performing fixed-length segmentation based on word count or character count to obtain a set of context segments. ; Set of facts With context slice set Perform semantic alignment to obtain a set of candidate fact-context pairs. ; The candidate fact-context pair set is labeled with fact-level conflict relationships to obtain the relationship label y, conflict score s, confidence level p, and labeling reason r; Construct a set of conflict-perceived evidence E based on the relation label y, conflict score s, confidence level p, and annotation reason r; Input the user's question Q, the original context C, the set of possible candidate answers O, and the set of conflict-aware evidence E into the conflict-aware chain thinking generation model to generate a credible response containing reasoning and a final answer.

[0007] Preferably, the facts are extracted from the user question Q. At that time, the user question Q is externalized using the self-fact mining model to obtain a set of self-facts. ; The self-fact mining model includes a self-knowledge extraction submodule, a self-context generation submodule, and a self-fact extraction submodule: The self-knowledge extraction submodule is used to generate a set of high-level factual knowledge needed to answer user questions. ; The context-generating submodule is based on the user question Q and the high-level factual knowledge set. Generate background documents related to the problem. ; The self-fact extraction submodule will extract the background document. Distillation into a fine-grained set of self-facts The self-fact set In the middle, from the facts This includes one or more of the following: factual text, entity, attribute, value, time constraint, source type, and internal confidence level.

[0008] Preferably, when performing bracket content cleaning, sentence segmentation, whitespace character normalization, and fixed-length segmentation based on word count or character count on the search context C, the segmentation size is... Select 20 to 80 words, and use punctuation-based regular expression segmentation as a fallback strategy when the sentence segmenter is unavailable.

[0009] Preferably, the set of facts will be used. With context slice set When performing semantic alignment to obtain a set of candidate fact-context pairs: Using sentence vector model For the set of facts Each self-fact in the data is encoded to obtain a self-fact vector. The sentence vector model Use any one of the following models: SentenceTransformer series, all-MiniLM-L6-v2, TinyBERT-L6-v2, Contriever, or Sentence-T5; Using sentence vector model Context segmentation set Each context segment is encoded to obtain a context segment vector. ; Based on self-fact vector and context slice vector Cosine similarity between For each self-fact, select the top K similarity score-ranked context blocks to obtain candidate facts. Context to set .

[0010] Preferably, when annotating the candidate fact-context pair set with fact-level conflict relationships, it is done by a large language model, a natural language inference model, a rule verification module, or a combination thereof; The labeled inputs include user questions (Q) and self-facts. and context segmentation The annotation output is a JSON structure, which includes at least the relation label y, conflict score s, confidence p, and annotation reason r, where the relation label y belongs to one of the four categories: support, conflict, irrelevant, or uncertain. The conflict score s is located in the interval [0,1]. When the relation label y is supportive, the conflict score s is between 0.0 and 0.3. When the relation label y is irrelevant, the conflict score s is between 0.0 and 0.5. When the relation label y is uncertain, the conflict score s is between 0.3 and 0.7. When the relation label y is conflicting, the conflict score s is between 0.7 and 1.0.

[0011] Preferably, when constructing the conflict-perceived evidence set E based on the relation label y, conflict score s, confidence level p, and annotation reason r: Filter relation labels y to be irrelevant or uncertain low-value fact pairs; The fact pairs with conflicting relation labels y are sorted from high to low according to the conflict score s, and both the fact and context slices are exposed as warning signals in the generated prompts. Relationship labels y are sorted by conflict score from low to high for supported fact pairs, and only contextual chunks are exposed in the generated prompts, not from the fact text.

[0012] Preferably, when the user question Q, the original context C, the set of optional candidate answers O, and the set of conflict-aware evidence E are input into the conflict-aware chain-thinking generation model to generate a credible response containing reasoning and a final answer, the conflict-aware chain-thinking generation model generates a credible response in the order of fact analysis, option matching or answer matching, context checking, conflict verification, and final confirmation; the credible response is output in JSON format and includes at least a Reason field and an Answer field.

[0013] This invention also provides a fact-level relation annotation and trusted response generation system for knowledge conflict, used to implement the fact-level relation annotation and trusted response generation method for knowledge conflict as described above, including: Input parsing unit: used to obtain the sample to be answered, which includes user question Q, retrieval context C related to user question, and a set of possible candidate answers O; Self-fact mining unit: used to extract a set of self-facts from the user question Q. ; Context preprocessing unit: used to perform bracket content cleaning, sentence segmentation, whitespace character normalization, and fixed-length segmentation based on word count or character count on the retrieval context C, to obtain a context segment set. ; Semantic alignment unit: used to align self-fact sets With context slice set Perform semantic alignment to obtain a set of candidate fact-context pairs. ; Fact-level conflict relationship annotation unit: used to annotate the candidate fact-context pair set with fact-level conflict relationships, and obtain the relationship label y, conflict score s, confidence p, and annotation reason r; Conflict-aware evidence organization unit: used to construct a conflict-aware evidence set E based on the relation label y, conflict score s, confidence level p, and annotation reason r; Conflict-aware generation unit: It is used to input the user question Q, the original context C, the set of optional candidate answers O and the set of conflict-aware evidence E into the conflict-aware chain thinking generation model to generate a credible response containing reasoning and final answer.

[0014] The present invention also provides an electronic device, comprising: One or more processors; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the fact-level relation annotation and trusted response generation method for knowledge conflict as described above.

[0015] The present invention also provides a storage medium, characterized in that it stores a computer program thereon, wherein the computer program, when executed by a processor, implements the fact-level relation annotation and trusted response generation method for knowledge conflict as described above.

[0016] The present invention has the following beneficial effects: To address the shortcomings of existing technologies in handling multi-source knowledge conflicts—namely, the inability to explicitly model factual relationships between different knowledge sources, the tendency of large language models to over-rely on their own built-in knowledge or blindly accept external retrieval contexts, resulting in responses lacking credibility and interpretability—this invention addresses the problem of implicit knowledge within the model being implicit and difficult to directly compare with external retrieval contexts in existing solutions. Furthermore, by preprocessing the retrieval context to obtain a set of context segments, semantic alignment is performed to filter out context segments related to the self-facts, forming candidate facts. The context pair set avoids introducing noise from a large amount of irrelevant text in the complete retrieval context, while narrowing the processing scope of subsequent fact-level conflict relationship annotation and reducing the computational cost of the processing. By outputting relationship labels, conflict scores, confidence levels, and annotation reasons through fact-level conflict relationship annotation, the factual relationships between self-facts and context blocks are expressed in a structured way, achieving explicit modeling of factual relationships. This overcomes the shortcomings of existing technologies that rely solely on cue word constraints and context enhancement methods, which cannot identify and characterize support and conflict relationships between facts. Based on the aforementioned annotation results, a conflict-aware evidence set is constructed, enabling the identification of candidate facts. The context is filtered and organized to provide evidence carrying conflict attribute information to the generation stage. This differs from the processing method of directly inputting the original retrieval context, enabling the conflict-aware chain-thinking generation model to identify the conflict risk of the evidence. After the conflict-aware evidence set is input into the conflict-aware chain-thinking generation model, the model's reasoning process will be constrained by the evidence set. It needs to combine conflict-related information to complete the reasoning verification, thereby reducing the situation where the model ignores external evidence and calls its own outdated knowledge, or adopts erroneous fragments of the retrieval context without discrimination. The output credible response comes with reasoning, which improves the credibility and interpretability of the output results in knowledge conflict scenarios. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of the fact-level relation annotation and credible response generation method for knowledge conflict in this embodiment of the invention. The diagram illustrates the system's process of taking a sample to be answered as input and sequentially executing input parsing and context preprocessing, self-fact mining, contextual knowledge alignment, fact-level conflict relation annotation, conflict-aware evidence organization, and conflict-aware chain-like thinking generation. Through this process, the system can convert user questions, retrieval context or external knowledge text, and optional candidate answers into structured evidence with relation labels, conflict scores, confidence levels, and annotation reasons, and generate credible responses containing Reason and Answer fields accordingly.

[0018] Figure 2 This is a schematic diagram of the fact-level conflict relationship annotation module in an embodiment of the present invention. The diagram illustrates the process by which the annotation module takes user questions, self-facts, and contextual segments as input, uses a large language model, a natural language inference model, a rule validation module, or a combination thereof to determine factual relationships, and outputs structured fields such as relation, conflict_score, confidence, and reason. Here, relation represents four types of factual relationships: support, conflict, irrelevant, or uncertain; conflict_score characterizes the degree of inconsistency between fact pairs; confidence characterizes the reliability of the annotation; and reason records traceable annotation justifications.

[0019] Figure 3This is a schematic diagram of the conflict-aware evidence organization and credible response generation module in an embodiment of the present invention. The diagram illustrates how the system filters low-value fact pairs based on fact-level annotation results, sorts them according to conflict score pairs (support and conflict fact pairs), and constructs a conflict-aware evidence set using a differentiated evidence exposure strategy. Subsequently, the user question, original context, set of possible candidate answers, and conflict-aware evidence are input into the conflict-aware chain-thinking generation model, causing the model to output a credible response in the order of fact analysis, answer matching, context checking, conflict verification, and final confirmation. Detailed Implementation

[0020] The present invention will be further described clearly and in detail below with reference to specific embodiments and the accompanying drawings. Those skilled in the art will be able to implement the present invention based on these descriptions. Furthermore, the embodiments of the present invention described below are generally only some, not all, of the embodiments of the present invention. Therefore, all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0021] See Figures 1-3 This invention relates to a method for fact-level relation annotation and credible response generation in the context of knowledge conflict, comprising the following steps: Step 1: Obtain the sample to be answered, x = (id, Q, C, O, a), where id is the sample identifier, Q represents the user question, C represents the retrieval context related to the user question Q, O represents the set of possible candidate answers, and a represents the reference answer during the training or evaluation phase. In open-ended question-answering scenarios, O can be empty; in multiple-choice question-answering scenarios, ... The goal of this invention is to learn or infer the following reliable answer distribution without modifying the parameters of a large language model. :

[0022] in, E is the set of conflict-perceived evidence constructed from the results of fact-level conflict relationship annotation.

[0023] This invention introduces As an intermediate structure, it enables the generative model to perceive which contextual fragments are reliable, which fragments are conflicting, and which fragments should be downweighted before it can respond.

[0024] Step 2: Externalize the user question Q using the self-fact mining model to obtain a set of self-facts. Self-fact mining is used to externalize the model's internal understanding of the user's question, rather than directly asking the model to answer the question. This step includes three sub-processes: self-knowledge extraction, self-context generation, and self-fact extraction, which are described in detail below: First, self-knowledge extraction: calling the large language model. Perform problem analysis on user question Q and output the set of high-level knowledge required to answer the question. :

[0025] in, The model suggests that the conceptual facts, domain rules, or key entity relationships required to answer question Q are necessary.

[0026] Second, self-context generation: combining Q and Common input self-contextual generative model Generate a background document of a specified length. This background document is used to transform abstract knowledge into more complete natural language fact descriptions, avoiding reliance on fragmented conceptual terms for subsequent alignment.

[0027]

[0028] Self-context is not external evidence, but rather an externalized representation of knowledge about model parameters.

[0029] Third, self-fact extraction: calling the fact extraction model ,from Extract concise and comparable factual assertions from the data to obtain a set of facts. :

[0030] Each, It can be represented as a factual sentence in natural language, or it can be further transformed into an entity-attribute-value structure.

[0031] In engineering implementation, the large language model can be implemented using Qwen2.5-7B-Instruct. The sampling parameters for the fact mining phase are preferably... , , To obtain reproducible deterministic output.

[0032] Based on the above scheme, the self-fact mining model includes a self-knowledge extraction submodule, a self-context generation submodule, and a self-fact extraction submodule; wherein, the self-knowledge extraction submodule is used to generate a set of high-level factual knowledge required to answer the user's question Q. The self-context generation submodule generates data based on the user question Q and the high-level fact knowledge set. Generate background documents related to user question Q. The self-fact extraction submodule will Distillation into a fine-grained set of self-facts Among them, the set of facts Self-facts in At least include factual texts ,entity e ,property Values Time constraints Source type and internal confidence One or more of the following; when using a semi-structured representation. Represented as .

[0033] Step 3: In the engineering implementation, the input data is organized as a JSON list, and each sample contains at least the fields of id, question, context, and answer; when the task is a multiple-choice question, it also includes the choices field. For the original retrieval context C, this invention performs bracket content cleaning, sentence segmentation, whitespace character normalization, and fixed-length segmentation based on word count or character count to obtain a context segment set. Sentence segmentation is preferentially performed using the NLTK sentence segmenter; when the segmenter is unavailable in the runtime environment, regular expressions are used for backtracking segmentation based on punctuation marks such as periods, question marks, and exclamation marks. Let the retrieval context C be segmented into several sentences s1, s2, ..., s... n The system calculates the number of words according to a preset number. By concatenating the components, a set of context segments is obtained. :

[0034] In a preferred embodiment, the size of the cut pieces Select 20 to 80 words, preferably 20 words; simultaneously set , This is to control the computational load and cue word length during the alignment phase.

[0035] When the sentence segmenter is unavailable, punctuation-based regular expression segmentation is used as a fallback strategy.

[0036] Step 4, set the self-facts With context slice set Perform semantic alignment to obtain a set of candidate fact-context pairs. (Also referred to as candidate fact pairs). The contextual knowledge alignment is used to find the relevant fact set from the long context C. Self-facts in The most relevant segments are selected to reduce noise introduced by inputting the entire context into the generative model.

[0037] Specifically: the semantic alignment is achieved through a sentence vector model. For the set of facts Each self-fact in the data is encoded to obtain a self-fact vector. ; through sentence vector model Context segmentation set Each context segment is encoded to obtain a context segment vector. ; wherein, the sentence vector model The model can be any one of the SentenceTransformer series, all-MiniLM-L6-v2, TinyBERT-L6-v2, Contriever, or Sentence-T5; in a preferred embodiment, sentence-transformers / all-MiniLM-L6-v2 is used as the default similarity model.

[0038] Then based on the self-fact vector and context slice vector Cosine similarity between For each self-fact, select the Top-K context slices (i.e., the context slices ranked Kth by similarity score) to obtain candidate facts. Context to set .

[0039] In a preferred embodiment, the sentence vector model uses sentence-transformers / all-MiniLM-L6-v2. This model has a small number of parameters, making it suitable for rapid batch encoding in CPU or single-GPU environments; for higher precision requirements, it can also be replaced with TinyBERT-L6-v2, Contriever, or Sentence-T5.

[0040] Step 5, for the candidate fact-context pair set Perform fact-level conflict relationship annotation to obtain relationship label y, conflict score s, confidence level p, and annotation reason r. The fact-level conflict relationship annotation is implemented by a large language model, a natural language inference model, a rule validation module, or a combination thereof; the annotation input includes the user question Q and self-facts. and context segmentation The annotation output is a JSON structure, including at least the fields `relation` (relation label y), `conflict_score` (conflict score s), `confidence` (confidence level p), and `reason` (annotation reason r). Here, `support` indicates that the context slice supports or is consistent with the self-fact; `conflict` indicates that the context slice contradicts the self-fact in terms of entity, attribute, value, time, or logical relationship; `irrelevant` indicates that the context slice lacks substantial relevance to the self-fact or user question; and `uncertain` indicates insufficient evidence or ambiguity, making it impossible to determine the relationship. In this invention, `relation` falls into one of the four categories: `support`, `conflict`, `irrelevant`, or `uncertain`.

[0041] The conflict score is located in the interval [0,1]. When the relation is support, the conflict score is between 0.0 and 0.3; when the relation is irrelevant, the conflict score is between 0.0 and 0.5; when the relation is uncertain, the conflict score is between 0.3 and 0.7; and when the relation is conflict, the conflict score is between 0.7 and 1.0.

[0042] The fact-level conflict annotation of this invention is the key intermediate layer of this invention. For each candidate fact-context pair ( , The system constructs annotation hints that include Question, Self-Fact, and Context Chunk, and calls the relation annotation model. Output structured JSON:

[0043] in, For relationship tags, For conflict scores, For confidence level, The reason is indicated here.

[0044] The JSON format of the annotation output is as follows: {"relation":"support / conflict / irrelevant / uncertain","conflict_score":0.0,"confidence":0.0,"reason":"brief reason"} The above outputs mean: `relation` represents the relation label, which can be one of `support`, `conflict`, `irrelevant`, or `uncertain`; `conflict_score` represents the conflict score, with a higher value indicating a greater degree of inconsistency between the facts; `confidence` represents the model's confidence in the current relation judgment; and `reason` represents a brief explanation. Both the conflict score and confidence score range from 0 to 1.

[0045] In one implementation, the conflict score is directly output by the model and validated by intervals; in another implementation, it can be obtained by weighting multiple discrepancy features.

[0046] in, Indicates the degree of inconsistency between entities or attributes. Indicates numerical differences, Indicates differences in time constraints. to The weights are determined for the preset weights or the validation set. Wherein, to The weights are assigned sequentially to six factors: conflict labels, semantic differences, entity or attribute differences, numerical differences, time constraint differences, and annotation uncertainty. Each factor is first normalized to 0-1, with all weights being non-negative and summing to 1. As an optional configuration, the six weights can be set to 0.70, 0.06, 0.08, 0.06, 0.05, and 0.05 respectively, with conflict labels as the primary basis and the remaining factors used to refine the scores. In practical applications, adjustments can be made based on business characteristics; for example, increasing the weight of numerical differences for numerical verification tasks and increasing the weight of time differences for time-sensitive tasks. Alternatively, a weight combination with smaller errors can be selected through manually labeled validation sets. The weighted results are then cropped according to the score range specified by the corresponding relation labels.

[0047] To ensure engineering robustness, the model output undergoes secure parsing: if the output contains Markdown code blocks, the JSON body is extracted first; if a field is missing or invalid, the relation is rolled back to uncertain, the conflict_score is set to 0.5, and the con... Set dence to 0.0 and record the reason for parsing failure.

[0048] Step 6: After obtaining the fact-level relation annotations, the candidate fact-context pairs are transformed into a conflict-aware evidence set E usable in the generation stage. The core principle is: supporting evidence is used to enhance contextual support, conflict evidence is used to prompt the generation model to handle cases cautiously, and low-confidence irrelevant evidence is used for noise reduction and filtering. Specifically, this invention can construct the conflict-aware evidence set E based on the relation label y, conflict score s, confidence level p, and annotation reason r. When constructing the conflict-aware evidence set E: filtering low-value fact pairs where the relation label y is irrelevant or uncertain; sorting conflicting fact pairs with relation label y in descending order of conflict score s, and simultaneously exposing them from both the fact and context segments as warning signals in the generation prompt; sorting supporting fact pairs with relation label y in ascending order of conflict score, and only exposing context segments in the generation prompt without exposing them from the fact text, to reduce the risk of knowledge noise injection into model parameters. To control the length of the prompt words, each sample retains at most the first N fact annotations, preferably 3.

[0049] In the above-mentioned scheme of the present invention, a differentiated evidence exposure strategy is adopted: for support relations, only the context slices are exposed in the generated prompts, and the self-fact text is not exposed, so as to avoid the model parameter knowledge being injected into the generation stage again through the self-facts; for conflict relations, both the self-facts and the context slices are exposed in the generated prompts, so that the model knows clearly that there is a contradiction between the two and performs conflict verification.

[0050] Step 7: Input the user question Q, the original context C, the set of possible candidate answers O, and the set of conflict-aware evidence E into the Conflict-Aware CoT Generation model to generate a credible response containing reasoning and the final answer.

[0051] Specifically, in this step, the process of generating a credible response using the conflict-perception chain-like thinking generation model includes the following steps: First, analyze the facts to determine whether the supporting evidence directly answers the question; Second, option matching or answer matching maps evidence to candidate options in multiple-choice tasks; Third, context check: review the original context when supporting evidence is insufficient; Fourth, conflict check: check whether the context of conflict evidence may be contaminated, mismatched with the issue entity, or contradicts common sense. Fifth, finally confirm and output the answer with a concise reason. The credible response should be output in JSON format and should include at least the Reason and Answer fields.

[0052] Unlike ordinary chain thinking, conflict-aware chain thinking does not simply require the model to "think step by step," but rather provides relational labels and conflict scores for each key piece of evidence, thereby constraining generative reasoning to fact-level labeled results.

[0053] Furthermore, embodiments of the present invention also provide a fact-level relation annotation and trusted response generation system for knowledge conflict, used to implement the fact-level relation annotation and trusted response generation method for knowledge conflict as described above. The system includes: Input parsing unit: used to obtain the sample to be answered, which includes user question Q, retrieval context C related to user question, and a set of possible candidate answers O; Self-fact mining unit: used to extract a set of self-facts from the user question Q. ; Context preprocessing unit: used to perform bracket content cleaning, sentence segmentation, whitespace character normalization, and fixed-length segmentation based on word count or character count on the retrieval context C, to obtain a context segment set. ; Semantic alignment unit: used to align self-fact sets With context slice set Perform semantic alignment to obtain a set of candidate fact-context pairs. ; Fact-level conflict relationship annotation unit: used to annotate the candidate fact-context pair set with fact-level conflict relationships, and obtain the relationship label y, conflict score s, confidence p, and annotation reason r; Conflict-aware evidence organization unit: used to construct a conflict-aware evidence set E based on the relation label y, conflict score s, confidence level p, and annotation reason r; Conflict-aware generation unit: It is used to input the user question Q, the original context C, the set of optional candidate answers O and the set of conflict-aware evidence E into the conflict-aware chain thinking generation model to generate a credible response containing reasoning and final answer.

[0054] The fact-level relation annotation and trusted response generation system for knowledge conflict also includes an evaluation and logging unit: used to store self_facts (i.e., the set of self-facts) and topk_chunks (i.e., the Topk chunks). The intermediate results include K-context segments, annotated_facts (i.e., labeled fact pairs), predictions (i.e., prediction results), eval (i.e., evaluation data / evaluation metric files), and summary (i.e., runtime summary). Here, self_facts refers to all self-facts output by the self-fact mining unit, i.e. Each sample mining yields externalized factual assertions from the model; topk_chunks refers to the semantic alignment stage, where each self-factual retrieval similarity Top... K context slice set, candidate facts The original fragment of the context chunk in the context pair; annotated_facts refers to the facts after fact-level conflict relationship annotation. The context pair includes a complete set of annotated structured results for relation, conflict_score, confidence, and reason; predictions refer to the credible response results output by the conflict-aware chain-thinking generation model, including the reasoning for reason and the final answer; eval refers to the evaluation output obtained after the model runs, including evaluation data such as accuracy and matching metrics, used to quantify the question-answering effect; summary refers to the overall summary log of the full sample run, including statistics on the number of conflict samples, the number of anomalies, the statistics of filtered samples, and the overall task summary information.

[0055] The embodiments of the present invention also provide corresponding electronic devices and computer-readable storage media for implementing the solutions provided in the embodiments of the present invention.

[0056] The electronic device includes a memory and one or more processors. The memory is used to store one or more programs, and the one or more processors are used to execute the one or more programs to enable the electronic device to perform the knowledge conflict-oriented fact-level relation annotation and trusted response generation method according to any embodiment of this application.

[0057] The storage medium stores a computer program, which, when executed by a processor, implements the knowledge conflict-oriented fact-level relation annotation and trusted response generation method described in any embodiment of this application.

[0058] Example 1 See Figure 1 This embodiment addresses the method for fact-level relation annotation and trusted response generation in the context of knowledge conflicts, and includes the following process: First, a sample to be answered is obtained, which includes a sample identifier, a user question, a retrieval context, and a set of possible candidate answers. For open question-answering tasks, the set of possible candidate answers can be empty; for multiple-choice question-answering tasks, the set of possible candidate answers contains several candidate options. Instead of directly inputting the user question and the complete long context into the generation model, the model's internal knowledge related to the question is first externalized into self-facts. Then, the self-facts are semantically aligned with the external context and fact-level conflict relationships are labeled, thereby providing structured control information for the subsequent generation of credible responses.

[0059] During input parsing and context preprocessing, the system reads fields such as id, question, context, choices, and answer, and performs bracket content cleaning, whitespace normalization, sentence segmentation, and fixed-length chunking on the original context. Sentence segmentation is preferentially implemented using a sentence segmenter; when the sentence segmenter is unavailable, a regular expression-based segmentation method based on punctuation marks such as periods, question marks, and exclamation marks is used as a fallback strategy. In the preferred method, the chunk size is set to 20 words, and send_topk and chunk_topk are set to control the subsequent alignment computation and generated prompt length while covering relevant evidence.

[0060] In the self-fact mining process, the self-fact mining model externalizes the model's internal understanding of the user's question. Its purpose is not to directly generate the final answer, but rather to form fact anchors that can be compared with the external context. This process includes three consecutive steps: self-knowledge extraction, self-context generation, and self-fact extraction. The self-knowledge extraction submodule identifies the high-level factual knowledge, domain concepts, entity relationships, or reasoning premises required to answer the user's question. The self-context generation submodule organizes the aforementioned high-level knowledge into a background document relevant to the question. The self-fact extraction submodule further extracts concise, explicit, and comparable factual assertions from this background document. Each self-fact can be represented as a natural language fact sentence or as a semi-structured field containing factual text, entities, attributes, values, time constraints, source type, and internal confidence. In a preferred implementation, the self-fact mining model can fine-tune the large language model using Qwen2.5-7B-Instruct or other instructions, setting the generation sampling parameters to temperature = 0.0, top_p = 1.0, and max_tokens = 1000 to improve the stability and reproducibility of the self-fact output. It should be noted that self-facts are not the same as external evidence, but rather an externalized representation of model parameter knowledge and problem understanding. Their main function is to serve as a reference anchor for subsequent context alignment and conflict judgment.

[0061] During the context knowledge alignment process, the system inputs self-facts and context chunks into the sentence vector model, obtaining self-fact vectors and context chunk vectors respectively, and calculates the cosine similarity between them. For each self-fact, the system selects several context chunks with the highest similarity to form candidate fact-context pairs. This step avoids inputting complete long contexts into the generation model without differentiation, instead focusing subsequent conflict judgments on context fragments highly related to the self-facts, thereby improving the efficiency and interpretability of fact-level relation annotation. In the preferred approach, the sentence vector model uses sentence-transformers / all-MiniLM-L6-v2; in other approaches, TinyBERT-L6-v2, Contriever, Sentence-T5, or other encoding models with semantic representation capabilities can also be used.

[0062] See Figure 2 For each candidate fact-context pair, the system constructs a tagging hint containing Question, Self-Fact, and Context Chunk, and calls the fact-level conflict relationship tagging module to output structured JSON. The tagging module can be implemented by a large language model, a natural language inference model, a rule validation module, or a combination thereof. The tagging output includes at least the fields relation, conflict_score, confidence, and reason, where relation takes one of four categories: support, conflict, irrelevant, or uncertain: support indicates that the context chunk supports or is consistent with the self-fact; conflict indicates that the context chunk contradicts the self-fact in terms of entity, attribute, value, time, or logical relationship; irrelevant indicates that the context chunk lacks substantial connection with the self-fact or user question; and uncertain indicates that it cannot be determined due to insufficient evidence or ambiguity.

[0063] The conflict score, ranging from 0 to 1, describes the degree of inconsistency between fact pairs. In one range constraint mode, when the relation is supportive, the conflict score is between 0.0 and 0.3; when the relation is irrelevant, it is between 0.0 and 0.5; when the relation is uncertain, it is between 0.3 and 0.7; and when the relation is conflict, it is between 0.7 and 1.0. Confidence indicates the labeling model's confidence in the relation's judgment, while reason records a brief annotation basis for system auditing, log tracking, and manual review. To improve engineering robustness, the system also performs secure parsing on the model output; if the relation field, conflict score field, or confidence field is missing, out of bounds, or has an invalid format, it is pruned to a valid range or set to a default value, and the reason for parsing failure is recorded.

[0064] See Figure 3 After obtaining the fact-level conflict relationship annotation results, the system constructs a conflict-aware evidence set based on relation, conflict_score, and confidence. Specifically, the system filters low-value fact pairs with relations of irrelevant or uncertain to reduce interference from irrelevant context and uncertain information in the generation process; for fact pairs with relations of conflict, they are sorted from high to low conflict scores, allowing the generation model to prioritize high-risk conflicts; for fact pairs with relations of support, they are sorted from low to high conflict scores, allowing evidence with stronger consistency to be included in the generation prompts first. In one preferred approach, each sample retains at most the first 3 fact annotations to balance prompt length and evidence coverage.

[0065] Regarding the method of evidence exposure, this invention adopts a differentiated strategy. For support relationships, the generation prompt only exposes contextual chunks, not self-factual text, to reduce the risk of model parameter knowledge being re-injected into the generation stage through self-facts. For conflict relationships, the generation prompt exposes both self-facts and contextual chunks, making the model explicitly aware of the inconsistency between the two and performing conflict verification before generation. Through this evidence organization method, the system can not only identify conflicts but also transform conflict annotation results into evidence adoption, weighting, review, and prompt control information during the generation stage.

[0066] The conflict-aware chain-thinking generation model receives a user question, original context, a set of alternative candidate answers, and a set of conflict-aware evidence. It then generates a response in the following order: fact analysis, option or answer matching, context checking, conflict verification, and final confirmation. For multiple-choice question-answering tasks, the model maps evidence to candidate options and outputs the most appropriate option; for open-ended question-answering tasks, the model generates a text answer. The final credible response is output in JSON format, containing at least Reason and Answer fields, where Reason is a concise reasoning explanation and Answer is the final answer. Compared to ordinary chain-thinking generation, this invention does not merely require the model to reason step-by-step; instead, it provides fact-level relational labels, conflict scores, and confidence levels for key evidence before reasoning, subjecting the generation process to structured evidence constraints. This reduces the risk of over-reliance on the model's intrinsic knowledge or blindly adopting external context.

[0067] From a system composition perspective, this embodiment may include an input parsing unit, a self-fact mining unit, a context preprocessing unit, a semantic alignment unit, a fact-level conflict relationship annotation unit, a conflict-aware evidence organization unit, a conflict-aware generation unit, and an evaluation and logging unit. The system engineering modules are divided as shown in Table 1. The input parsing unit reads fields such as sample identifier, question, context, candidate answer, and reference answer; the self-fact mining unit generates self-knowledge, self-context, and self-facts; the context preprocessing unit performs text cleaning, sentence segmentation, and fixed-length chunking; the semantic alignment unit calculates the similarity between self-facts and context chunks; the fact-level conflict relationship annotation unit outputs relationship labels, conflict scores, confidence levels, and annotation reasons; the conflict-aware evidence organization unit filters, sorts, and organizes evidence; the conflict-aware generation unit generates credible responses; and the evaluation and logging unit saves intermediate results such as self_facts, topk_chunks, annotated_facts, predictions, eval, and summary.

[0068] Table 1

[0069] This embodiment does not limit the specific source of external knowledge. The external knowledge can be text returned by the retrieval system, knowledge base fragments, tool call results, database records, historical interaction information, domain rule documents, or other knowledge sources that can be converted into text or factual assertions. The method also does not require retraining a large language model and can be integrated with existing question-answering systems, knowledge management systems, or intelligent application systems through prompt templates, semantic encoding models, relation annotation modules, and evidence organization strategies. Correspondingly, this invention can also be implemented by electronic devices, computer-readable storage media, and computer program products. The electronic device includes a processor and a memory, and the computer program stored in the memory, when executed by the processor, implements the above-described fact-level relation annotation and trusted response generation method.

[0070] This embodiment selects two commonly used comparison methods in public knowledge conflict question-answering experiments as reference baselines and supplements the verification results of the current implementation of this invention. The ordinary context input method directly concatenates the user question and the original context before inputting it into the large language model, without performing self-fact mining, context chunk alignment, or fact-level conflict relationship annotation. The Context-Aware Decoding (CAD) method compares the output probabilities under contextual and non-contextual conditions during the decoding stage to enhance the model's dependence on context; the comparison results are shown in Table 2. The method in this embodiment explicitly constructs candidate fact-context pairs before generation and annotates them with support, conflict, irrelevant, or uncertain tags before inputting conflict-aware evidence into the generation model.

[0071] Table 2

[0072] The accuracy figures in Table 2 use Qwen2.5-7B-Instruct as the backbone model and the SQuAD knowledge conflict question answering dataset as a representative test scenario. The above results are only for reference when selecting a baseline for implementation of this invention. In actual deployment, the accuracy, perfect match rate, and F1 score can be recalculated based on the selected large language model, vector encoding model, chunk length, and business dataset. As shown in Table 2, on the SQuAD knowledge conflict question answering task, the current implementation of this invention achieves an ACC index of 74.3, which is higher than the representative results of ordinary context input methods and CAD methods.

[0073] As can be seen from the above scheme, the present invention can explicitly identify fact-level conflicts between different knowledge sources before response generation, reduce the risk of the model blindly adopting incorrect knowledge or over-relying on a single knowledge source, and improve the credibility, robustness and interpretability of the language model in knowledge conflict scenarios.

[0074] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A method for fact-level relation annotation and reliable response generation for knowledge conflict, characterized in that, The process includes the following: Obtain a sample to be answered, which includes the user question, the search context related to the user question, and a set of possible candidate answers; Extracted from the set of facts in the user question; The search context is cleaned by removing bracket content, segmenting sentences, normalizing whitespace characters, and dividing into fixed-length blocks based on word count or character count to obtain a set of context blocks; Semantically align the self-fact set with the context chunk set to obtain a candidate fact-context pair set; Fact-level conflict relationship annotation is performed on the candidate fact-context pair set to obtain relationship labels, conflict scores, confidence levels, and annotation reasons; Construct a set of conflict-perceived evidence based on the relationship labels, conflict scores, confidence levels, and annotation reasons; The user's question, the original context, the set of possible candidate answers, and the set of conflict-aware evidence are input into the conflict-aware chain-thinking generation model to generate a credible response containing reasoning and a final answer.

2. The method for fact-level relation annotation and reliable response generation for knowledge conflict as described in claim 1, characterized in that, When extracting the self-fact set from the user question, the user question is externalized using the self-fact mining model to obtain the self-fact set; The self-fact mining model includes a self-knowledge extraction submodule, a self-context generation submodule, and a self-fact extraction submodule: The self-knowledge extraction submodule is used to generate a set of high-level factual knowledge required to answer user questions; The self-context generation submodule generates a background document related to the question based on the user's question and a set of high-level factual knowledge. The self-fact extraction submodule distills the background document into a fine-grained set of self-facts, in which self-facts... This includes one or more of the following: factual text, entity, attribute, value, time constraint, source type, and internal confidence level.

3. The method for fact-level relation annotation and reliable response generation for knowledge conflict as described in claim 1, characterized in that, When performing bracket content cleaning, sentence segmentation, whitespace character normalization, and fixed-length segmentation based on word count or character count on the search context, the segment size is 20 to 80 words. When the sentence segmenter is unavailable, regular expression segmentation based on punctuation is used as a fallback strategy.

4. The method for fact-level relation annotation and reliable response generation for knowledge conflict as described in claim 1, characterized in that, When semantically aligning the set of self-facts with the set of context segments to obtain the candidate set of fact-context pairs: Each self-fact in the self-fact set is encoded using a sentence vector model to obtain a self-fact vector; the sentence vector model can be any one of the SentenceTransformer series models, all-MiniLM-L6-v2, TinyBERT-L6-v2, Contriever or Sentence-T5. Each context segment in the context segment set is encoded using a sentence vector model to obtain a context segment vector; Based on the cosine similarity between the self-fact vector and the context slice vector, the top-ranked context slice for each self-fact is selected to obtain candidate facts. Context pairs.

5. The method for fact-level relation annotation and reliable response generation for knowledge conflict as described in claim 1, characterized in that, When annotating the candidate fact-context pair set with fact-level conflict relationships, it is achieved by a large language model, a natural language inference model, a rule verification module, or a combination thereof. The annotation input includes user questions, self-facts, and contextual chunks. The annotation output is a JSON structure, which includes at least relational labels, conflict scores, confidence levels, and annotation reasons. The relational labels belong to one of four categories: supportive, conflicting, irrelevant, or uncertain. The conflict score is located in the range [0,1]. When the relation label is supportive, the conflict score is between 0.0 and 0.

3. When the relation label is irrelevant, the conflict score is between 0.0 and 0.

5. When the relation label is uncertain, the conflict score is between 0.3 and 0.

7. When the relation label is conflicting, the conflict score is between 0.7 and 1.

0.

6. The method for fact-level relation annotation and reliable response generation for knowledge conflict as described in claim 5, characterized in that, When constructing a set of conflict-aware evidence based on the aforementioned relation labels, conflict scores, confidence levels, and annotation reasons: Filter relationship labels as irrelevant or uncertain low-value fact pairs; The fact pairs labeled as conflicting relationships are sorted from high to low according to their conflict scores, and both fact and contextual chunks are exposed as warning signals in the generated prompts. Relationships are labeled as supporting fact pairs, sorted by conflict score from low to high, and only contextual chunks are exposed in the generated prompts, not from the fact text.

7. The method for fact-level relation annotation and reliable response generation for knowledge conflict as described in claim 5, characterized in that, When a user's question, original context, set of possible candidate answers, and set of conflict-aware evidence are input into a conflict-aware chain-thinking generation model, and a credible response containing reasoning and a final answer is generated, the conflict-aware chain-thinking generation model generates a credible response in the order of fact analysis, option matching or answer matching, context checking, conflict verification, and final confirmation; the credible response is output in JSON format and includes at least a reason field and an answer field.

8. A fact-level relation annotation and reliable response generation system for knowledge conflict, characterized in that, The method for implementing the fact-level relation annotation and trusted response generation method for knowledge conflict as described in any one of claims 1-7 includes: Input parsing unit: used to obtain the sample to be answered, which includes the user question, the retrieval context related to the user question, and a set of possible candidate answers; Self-fact mining unit: used to extract a set of facts from the user's question; Context preprocessing unit: used to perform bracket content cleaning, sentence segmentation, whitespace character normalization, and fixed-length segmentation based on word count or character count on the search context to obtain a context segmentation set; Semantic alignment unit: used to semantically align the self-fact set with the context chunk set to obtain a candidate fact-context pair set; Fact-level conflict relationship annotation unit: used to annotate the candidate fact-context pair set with fact-level conflict relationships, and obtain relationship labels, conflict scores, confidence scores and annotation reasons; Conflict-aware evidence organization unit: used to construct a conflict-aware evidence set based on the relationship labels, conflict scores, confidence levels, and annotation reasons; Conflict-aware generation unit: It is used to input the user's question, the original context, the set of possible candidate answers, and the set of conflict-aware evidence into the conflict-aware chain thinking generation model to generate a credible response containing reasoning and a final answer.

9. An electronic device, characterized in that, include: One or more processors; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the knowledge conflict-oriented fact-level relation annotation and trusted response generation method as described in any one of claims 1-7.

10. A storage medium, characterized in that, It stores a computer program, wherein when the computer program is executed by a processor, it implements the knowledge conflict-oriented fact-level relation annotation and trusted response generation method as described in any one of claims 1-7.