A method and system for assessing the credibility of a text inference chain

Abstract

The application discloses a text reasoning chain credibility evaluation method and system, which comprises receiving a text reasoning chain to be evaluated; constructing a logical dependency graph based on the text reasoning chain; performing verification interaction based on the graph; and generating an evaluation result based on feedback, so that deep analysis of the internal structure of the reasoning chain is realized through structured logical dependency and multi-round verification interaction, and problems such as insufficient verification depth, dependence on an external knowledge base, lack of robustness testing, poor interpretability and the like in the prior art are solved, and the credibility evaluation can be deep, fine-grained, interpretable and robust.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for evaluating the credibility of text reasoning chains. Background Technology

[0002] In recent years, with the widespread application of Large Language Models (LLMs) in question answering, reasoning, and content generation, automatic reasoning methods based on text reasoning chains have become one of the core directions of artificial intelligence research and application. By generating a series of intermediate reasoning steps, the model can handle more complex tasks and improve the logicality and interpretability of the output.

[0003] However, existing technologies still have significant technical shortcomings when verifying and assessing the credibility of the generated text reasoning chains:

[0004] First, the verification mechanism lacks the ability to make structured logical judgments. Existing methods mostly rely on semantic similarity calculation, text matching, or attention weight analysis to assess the relevance or importance of reasoning steps. These methods struggle to formally determine the logical truth relationships between steps at the system level, thus failing to effectively identify reasoning results that are "semantically similar but logically flawed," resulting in insufficient accuracy and reliability of the evaluation.

[0005] Secondly, there is a lack of interpretable and computable structured intermediate representations. The reasoning chains output by existing systems are usually continuous natural language text, and their internal reasoning structures, claim units, and dependencies are not represented and stored in an explicit, structured data format. This makes it difficult for the system to accurately parse, audit, reuse, or implement targeted logical interventions on reasoning nodes, limiting the depth and interpretability of the evaluation process.

[0006] Secondly, it lacks the ability to perform counterfactual robustness testing based on the reasoning structure. Existing solutions typically only generate and output a single reasoning path under fixed inputs, lacking a systematic mechanism for conditional intervention, multi-path comparison, and consistency testing of the reasoning structure itself. Therefore, it is difficult to assess whether the reasoning conclusion maintains logical robustness when key premises change, and it is impossible to discover hidden structural vulnerabilities.

[0007] Although some studies have attempted to introduce external knowledge, such as knowledge graphs, to enhance the interpretability of models, or to use counterfactual analysis, meta-learning, and other techniques to optimize the generation and selection of reasoning paths, these solutions either focus on knowledge enhancement and attention interpretation or on path generation optimization and feedback learning, and are neither quantifiable nor auditable.

[0008] Therefore, overcoming the above-mentioned shortcomings and providing a systematic method and apparatus capable of conducting in-depth structured analysis, multi-dimensional verification, and robustness assessment of text reasoning chains has become a key technical issue in improving the credibility and practical value of artificial intelligence reasoning results. Summary of the Invention

[0009] The purpose of this invention is to provide a credibility evaluation method and system for text reasoning chains, in order to solve the problem that existing technologies do not construct an end-to-end dedicated evaluation system with a structured logical dependency graph as the core intermediate representation and systematically integrate multi-round question-and-answer verification and counterfactual stress testing.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A method for evaluating the credibility of a text reasoning chain, comprising:

[0012] Receive the text reasoning chain to be evaluated; construct a logical dependency graph representing its internal logical structure based on the text reasoning chain; perform multiple rounds of verification interaction on the logical elements in the graph based on the logical dependency graph through a verification model to obtain verification feedback; generate a credibility evaluation result for the text reasoning chain based on the verification feedback.

[0013] Furthermore, the present invention also provides a credibility evaluation system for text reasoning chains, comprising:

[0014] The graph construction module is used to receive the text reasoning chain to be evaluated and construct a logical dependency graph representing its internal logical structure based on the text reasoning chain; the verification interaction module is used to perform multiple rounds of verification interaction on the logical elements in the graph based on the logical dependency graph and obtain verification feedback; the arbitration output module is used to generate a credibility evaluation result of the text reasoning chain based on the verification feedback.

[0015] Beneficial effects

[0016] This invention provides a method and system for credibility assessment of text reasoning chains, comprising receiving a text reasoning chain to be assessed; constructing a logical dependency graph based on the text reasoning chain; performing verification interactions based on the graph; and generating an assessment result based on feedback. This achieves in-depth analysis of the internal structure of the reasoning chain through structured logical dependencies and multi-round verification interactions, solving problems such as insufficient verification depth, reliance on external knowledge bases, lack of robustness testing, and poor interpretability in existing technologies. It possesses the capability to perform in-depth, fine-grained, interpretable, and robustness testing-based credibility assessment of text reasoning chains.

[0017] This invention, by introducing a logical dependency graph as the core intermediate representation, deconstructs unstructured textual reasoning chains into computable, traceable discrete logical units (nodes) and their dependencies (directed edges) for the first time. This structural transformation provides the foundation for deep, fine-grained analysis: it enables the system to independently identify, verify, and intervene in each atomic claim and its logical connections within the reasoning chain. Based on this, by initiating multiple rounds of verification interactions for each logical element in the graph, this invention overcomes the shortcomings of existing methods that rely on overall semantic similarity and lack structured logical judgment. This graph-based element-by-element verification mechanism ensures that the examination of the reasoning chain can penetrate to every logical transition, thereby achieving a substantial improvement in verification depth. Furthermore, utilizing the explicit structure of the logical dependency graph, this invention can systematically implement counterfactual reasoning tests, that is, by simulating premise changes and observing the propagation of effects in the graph, to test the logical robustness of the reasoning chain. This solves the problem that existing technologies only evaluate on fixed paths and cannot expose structural vulnerabilities. Ultimately, the entire evaluation process (construction, verification, and testing) unfolds based on an interpretable intermediate structure, with the conclusions of each step traceable back to specific elements in the graph, thus naturally generating interpretable evaluation results. In summary, this invention comprehensively addresses the problems identified in the background art through the systematic integration of three mechanisms: 'structured representation', 'graph-based fine-grained verification', and 'structure-driven robustness testing'. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the text reasoning chain credibility evaluation method according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of a text reasoning chain credibility assessment method according to another embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of the credibility assessment system for text reasoning chains according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] For ease of understanding, some key terms in the embodiments of the present invention are explained below:

[0025] A textual reasoning chain is a sequence of texts consisting of a series of intermediate reasoning steps or logical claims, which aims to derive a final conclusion from given premises. It is often used in large language models to generate answers or explanations for complex problems.

[0026] A logical dependency graph is a directed graph that contains the directionality of reasoning to support structured verification. It is used to structurally represent the reasoning dependencies between multiple claims in a textual reasoning chain.

[0027] A verification model is a computational model used to check and verify the logical elements in a logical dependency graph. This model can understand and respond to inquiries regarding the truthfulness of logical claims or the strength of logical relationships, and provide corresponding verification information or judgments.

[0028] Logical elements refer to the basic components that make up a logical dependency graph.

[0029] Verification feedback refers to the response information generated by the verification model after multiple rounds of verification interactions with the logical elements in the logical dependency graph. This feedback information can be a judgment on the truthfulness of claims, an assessment of the strength of logical relationships, or other information that helps to evaluate the credibility of the reasoning chain.

[0030] Credibility assessment results refer to a quantitative or qualitative judgment of the overall or partial credibility of a text's reasoning chain. These results can be expressed as scores, grades, reports, etc., and are used to indicate the reliability, accuracy, and logical robustness of the reasoning chain.

[0031] like Figure 1 As shown, this application proposes a method for evaluating the credibility of a text reasoning chain, including the following steps: S101: receiving the text reasoning chain to be evaluated;

[0032] In this step, the text reasoning chain to be evaluated can be provided in a variety of ways, such as through manual input or pasting via a user interface, reading from a locally stored file, or receiving it from other automated systems via a standardized application programming interface (API).

[0033] In practical applications, the input to be evaluated is a text inference chain. This inference chain is typically a series of coherent inference steps generated by large language models, such as GPT-4 and Claude, based on an initial question or statement. For example, for the initial question "Why is the risk of traffic accidents increased today?", the model might generate the inference chain: "1. It rained today; 2. Rain makes the roads slippery; 3. Slippery roads increase the risk of traffic accidents."

[0034] The inference chain can be a string, and its input source can be direct user input or the output of upstream tasks, such as automated question answering systems or report generation systems. During initialization, necessary parameters are configured, such as loading the Language Model (LLM) application programming interface (API) for subsequent processing, setting the weight parameters for each scoring module, such as question-answering validation weights and counterfactual analysis weights, and loading preset prompt word templates.

[0035] S102: Based on this text reasoning chain, construct a logical dependency graph that represents its internal logical structure;

[0036] In this step, a set of grammatical or semantic rules can be pre-defined. Pattern matching and keyword recognition are used to extract logical claims from the text's reasoning chain, identifying causal, parallel, or adversative relationships between these claims. These claims and relationships are then mapped to nodes and directed edges in a graph structure. Alternatively, a specialized statistical model, such as a sequence labeling model or a relation extraction model, can be trained. This model automatically identifies key entities in the text as nodes in the graph and the logical relationships between these entities as directed edges, ultimately organizing this information into a complete logical dependency graph.

[0037] In practical applications, this logical dependency graph G can be formally defined as a directed graph G=(V, E). Where:

[0038] Node set V: Each node v_i represents an independent logical claim in the inference chain, such as "It rained today". A node can be defined as a triple (id_i, text_i, confidence_i), representing a unique identifier, claim text, and initial confidence level, respectively.

[0039] The set of directed edges E: Each directed edge e_k represents a logical dependency between two claims, defined as a triple (from_k, to_k, type_k), representing the source node (premise), the target node (conclusion), and the relation type, respectively. The relation type type_k includes: "Supports", "Contradicts", and "RelatedTo".

[0040] The logical dependency graph constructed through the above steps fundamentally changes the representation of textual reasoning chains. It maps coherent natural language text into a graph data structure containing discrete claims (nodes) and explicit logical relationships (directed edges). This transformation has the following key implications: First, it provides a computable, structured intermediate representation, enabling all subsequent analyses (such as verification and testing) to be built upon programmable, queryable objects, overcoming the parsing and auditing difficulties caused by relying on unstructured text in the background technology. Second, the explicit structure of the graph reveals the inherent logical thread of reasoning, visualizing and formalizing the originally implicit premises, conclusions, and supporting relationships, laying an indispensable foundation for in-depth logical analysis. In other words, the construction of the logical dependency graph is not the end goal itself, but rather a prerequisite and core enabling technology created for achieving all subsequent in-depth verification and robustness testing.

[0041] S103: Based on this logical dependency graph, the logical elements in the graph are validated through multiple rounds of validation interaction using a validation model to obtain validation feedback;

[0042] In this step, after obtaining the logical dependency graph, a deep verification process is initiated to independently verify the factuality and logical strength of each logical element in the graph. In practical applications, a simple yes / no question can be generated for each logical claim or relationship in the logical dependency graph and submitted to the verification model. The verification model responds to the question based on its internal knowledge or reasoning ability, returning judgments such as "yes," "no," or "uncertain," and these responses constitute the verification feedback. Another possible implementation involves using a pre-defined query template to generate structured query statements for the nodes and directed edges in the logical dependency graph, sending them to the verification model, and collecting the text or structured data returned by the model as verification feedback.

[0043] The aforementioned verification interaction process based on the logical dependency graph achieves a paradigm shift in evaluation mode from "overall fuzzy judgment" to "local precise verification." Since each logical element (node ​​and edge) is independently identified in the graph, the system can generate precise verification questions for them. This independent verification of each element in the graph ensures that the examination of the reasoning chain can penetrate to the finest logical unit, avoiding the problem that traditional methods may overlook or mask local errors due to overall evaluation. Especially for the strength verification of directed edges, it directly assesses whether the logical dependency relationship between claims is valid and how strong it is. This is the core manifestation of "structured logical judgment" at the system level, effectively identifying reasoning leaps that are "semantically similar but logically flawed." Therefore, the depth of multi-round verification interactions is directly guaranteed by the fine-grained structure of the logical dependency graph.

[0044] In another embodiment, the verification model is a second language model, and the second language model is different from the generative model used to generate the text reasoning chain.

[0045] The verification model, as the core verification component, functions to deeply analyze and verify the internal logical structure of the text reasoning chain. Designating it as a second language model implies that it possesses powerful natural language understanding, generation, and reasoning capabilities, able to handle complex semantic information, understand logical relationships, and generate corresponding verification questions or respond to verification questions based on verification requirements. For example, this second language model can be a large pre-trained language model (LLM), such as a Transformer-based model like the GPT series, Llama series, or PaLM series. These models are trained on massive amounts of text data, possessing rich world knowledge and powerful language understanding capabilities. Furthermore, this second language model can also be a language model fine-tuned for a specific domain or verification task. For instance, by training on datasets containing fact-checking, logical reasoning, or debate, it can exhibit higher accuracy and robustness on verification tasks.

[0046] Meanwhile, the second language model differs from the generative model used to generate the text inference chain, a feature that emphasizes the independence and objectivity of the verification process. By ensuring a difference between the verification model and the generative model that generates the text inference chain, the inherent biases, knowledge boundaries, or systematic errors inherited by the generative model can be effectively avoided. Specifically, the generative model and the second language model can have completely different model architectures, or they can be models with the same architecture but different versions or different training datasets. For example, the generative model might be a specific version of the GPT model, while the second language model might be another different version of the GPT model or a GPT model trained on a different dataset, or a language model developed by a completely different vendor with different internal mechanisms.

[0047] S104: Based on the verification feedback, generate a credibility assessment result for the reasoning chain of the text.

[0048] In this step, a simple count can be performed on the logical elements indicated as "true" or "logically correct" in the verification feedback, and the result can be used as a preliminary credibility score for the textual reasoning chain. Alternatively, the number of inconsistent or erroneous logical elements identified in the verification feedback can be compared with a preset threshold; if the number of errors exceeds the threshold, the reasoning chain is deemed unreliable. Furthermore, the detailed textual information in the verification feedback can be summarized and organized to generate a qualitative report that describes potential specific problems or weaknesses in the reasoning chain.

[0049] In practical applications, by deconstructing the text inference chain into a structured logical dependency graph and using a verification model to perform multi-round interactive checks on the logical elements in this graph, a deep and fine-grained evaluation of the inference chain is achieved, effectively solving the problem of insufficient verification depth in existing technologies. This method reduces the strong dependence on external structured knowledge bases and improves its versatility in open domain scenarios.

[0050] In other implementations, the logical dependency graph includes a set of nodes and a set of directed edges, wherein each node corresponds to a logical claim and each directed edge corresponds to a logical dependency between claims;

[0051] Step S102 includes: providing prompts to the first language model to drive it to extract the node and the directed edge from the text reasoning chain. Specifically, the logical dependency graph, as a structured representation, deconstructs the complex text reasoning chain into discrete logical elements and their interrelationships. That is, each node in the node set corresponds to a logical claim in the text reasoning chain, which can be a factual statement, hypothesis, intermediate conclusion, or final conclusion, etc. Each directed edge in the set of directed edges represents the logical dependency relationship between these logical claims, such as causal relationship, support relationship, implication relationship, or premise-conclusion relationship, etc. This structured representation makes the internal logic of the reasoning chain clearly presented, laying the foundation for subsequent fine-grained verification.

[0052] In practical applications, a node can be defined as containing the text content of the claim, a unique identifier, and its position information in the original text, while a directed edge can be defined as containing the type of dependency, such as "support", "refute", "cause", etc., and the direction.

[0053] In this implementation, a node represents each independent claim in the reasoning process. A claim is the smallest unit of judgment that can be cited, supported, or refuted during the reasoning process, such as "It rained today" or "The road is slippery." Depending on its position in the reasoning structure, a claim can play different roles in the logical dependency graph:

[0054] Initial claim: A claim node that exists as the starting point of reasoning in the current reasoning chain without relying on the support of other claims. Its discrete state mapping is regarded as a given premise in the modeling stage of this graph.

[0055] Conclusion: The claim node located at the end of the reasoning chain and no longer serving as a premise for other claims is the target output of the reasoning process.

[0056] It should be noted that the conclusion is not an independent semantic type, but rather a structural role claimed in the logical dependency graph.

[0057] Edges represent the logical relationships between claims, used to characterize the dependency of claims in the reasoning structure, and are divided into three categories:

[0058] Supports: Premise supports conclusion, indicating that the source claim provides reasoning support or premises for the target claim;

[0059] Contradictions: Premise and conclusion contradict each other, indicating a logical conflict or negation between the source claim and the target claim;

[0060] RelatedTo: Related but with unclear logical direction, indicating that there is a correlation between the claims, but the direction of reasoning or the strength of dependence cannot be clearly determined.

[0061] In some possible implementations, the node can be further associated with a confidence score, representing the certainty of the claim when it is extracted; the directed edge can be associated with a weight, representing the strength of its logical dependency.

[0062] To automate the construction of the logical dependency graph, this embodiment employs an extraction mechanism based on a first language model. This first language model can be, but is not limited to, a large pre-trained language model (LLM), such as the GPT series, BERT series, or Llama series, which possesses powerful natural language understanding and generation capabilities. By providing prompts to this first language model, it can be effectively guided to perform specific tasks. These prompts typically contain explicit instructions for the task, such as "Extract all logical claims and their logical dependencies from the following text and output them as triples," examples (few-shot examples), and strict requirements for the output format. Upon receiving these prompts, the first language model performs a deep analysis of the text's reasoning chain, identifying each logical claim it contains, and further determining the logical relationships between these claims, thereby automatically extracting the node and the directed edge. For example, the first language model can be trained or fine-tuned to identify propositional units in the text as nodes and to identify causal connectors, transition words, etc., connecting these propositional units as indicators of directed edges. In another possible implementation, the first language model can be designed to directly identify and structure output nodes and directed edges from the text through sequence labeling or relation extraction tasks, for example, by identifying subject-verb-object structures in the text as potential nodes and identifying verbs or conjunctions as directed edges.

[0063] In practical applications, this first language model can extract the set of nodes and the set of directed edges from the unstructured text reasoning chain, avoiding the need for manually defining complex rules. It utilizes the semantic understanding capabilities of a large language model to complete the extraction task under structured prompt constraints. The specific implementation process is as follows:

[0064] A prompt word template is predefined or dynamically generated. This template is a structured instruction that explicitly requires the first language model to perform a specific task. For example, the template might contain the following elements: role instruction: such as "You are a logic analysis assistant, and your task is to identify claims and their logical relationships in the text."

[0065] The model is required to output results in a specified structured format, such as JSON, and to explicitly define output fields, such as lists of claims and replies, and their subfields, such as id, text, from, to, and type.

[0066] Explain the definitions of "claim" and "relationship type" and provide a brief example.

[0067] Fill the corresponding position in the template with the text reasoning chain to generate a complete prompt message containing specific tasks and data.

[0068] The constructed prompt information is sent to the first language model via the application programming interface (API). The first language model, as a controlled data processing unit, understands and analyzes the text reasoning chain according to the prompt instructions and generates a structured response data R_json that meets the requirements.

[0069] After receiving R_json, start the parser:

[0070] Successful path: The parser attempts to parse R_json into a predefined data structure, such as a Python dictionary. If successful, it iterates through the R_json["claims"] list, creates a node object for each element, fills in the id and text, and adds it to the node collection;

[0071] Iterate through the R_json["relations"] list, create an edge object for each element, fill in the from, to, and type, and add it to the directed edge collection;

[0072] Exception handling: If parsing fails, such as due to formatting errors, a retry mechanism is activated, allowing the first language model to be called up to 3 times. If the process still fails after multiple retries, a rule backup process is triggered. This rule backup process uses a deterministic rule, such as splitting the inference chain into nodes by sentence and adding RelatedTo edges in sequence to generate a simplified logical dependency graph to ensure the continuity of the system flow.

[0073] For example, given the text reasoning chain "It rained today; therefore the road is slippery; therefore the risk of traffic accidents increases.", after processing in this embodiment, the following logical dependency graph may be constructed:

[0074] Node: V={v1(id:“C1”,text:“It’s raining today”),v2(id:“C2”,text:“The road is slippery”),v3(id:“C3”,text:“Increased risk of traffic accidents”)};

[0075] Directed edges: E = {e1(from:“C1”,to:“C2”,type:“Supports”) e2(from:“C2”,to:“C3”,type:“Supports”)}. This graph clearly shows the logical dependency structure that “rain” supports “slippery road”, and “slippery road” supports “increased accident risk”.

[0076] Through the above technical solution, this application can efficiently and accurately extract the internal logical structure from complex text reasoning chains, making the extraction target of logical elements clear and avoiding ambiguity.

[0077] In other implementations, such as Figure 2 As shown, step S103 includes:

[0078] S301: For the nodes in the logical dependency graph, generate a first type of verification question to verify the authenticity of the claims, and determine the verification status of the node based on the response of the verification model.

[0079] This step aims to perform factual or truth verification on the logical claim represented by each node in the logical dependency graph. Specifically, the verification model can be provided with the text content contained in the node and asked to determine the truthfulness of the claim. For example, the verification model can be directly asked: "Please verify the truthfulness of the claim '[node content]'." Alternatively, the node content and its contextual information in the original text reasoning chain can be provided, and the verification model can be asked: "Based on the provided context, is the claim '[node content]' true?" In this way, the objective truthfulness of each atomic claim is ensured to be independently evaluated.

[0080] The verification status of a node is determined based on the response of the verification model, transforming the model's response to the first type of verification question into a structured and quantifiable node verification status. One possible implementation is to label the node status as a discrete category such as "true," "false," "unverifiable," or "partially true" based on the verification model's output, for example, explicit "yes," "no," "uncertain," or a confidence score. Another possible implementation is to parse the verification model's response into a continuous numerical value, such as a confidence score between 0 and 1, where a higher score indicates a higher degree of assertion. Furthermore, supporting or refutating evidence provided by the verification model can be incorporated into the node status to enhance traceability.

[0081] S302: For the directed edges in the logical dependency graph, generate a second type of verification question to check the logical strength of the relationship, and determine the strength state of the directed edge based on the response of the verification model.

[0082] This step aims to evaluate the validity or strength of the logical dependencies represented by directed edges in the logical dependency graph. Specifically, the validation model can be provided with two logical claims represented by the start and end nodes of a directed edge, and the strength of the logical connection between them can be queried. For example, one could ask: "Given claim '[start node content]', to what extent is claim '[end node content]' logically valid?" Or, the validation model can be asked to evaluate: "Does claim '[start node content]' sufficiently support claim '[end node content]'?" In this way, the logical coherence and reasoning rationality between the steps in the reasoning chain can be analyzed in depth.

[0083] The strength states of directed edges are determined based on the response of the verification model, transforming the model's response to the second type of verification problem into structured and quantifiable directed edge strength states. One possible implementation is to label the directed edge states as discrete categories such as "strong dependency," "weak dependency," "no dependency," or "contradiction" based on the verification model's output. Another possible implementation is to parse the verification model's response into a continuous numerical value, such as a logic strength score between 0 and 1, where a higher score indicates a stronger logical dependency. Furthermore, explanations or counterexamples provided by the verification model regarding the rationality of the logical deduction can be incorporated into the directed edge states to provide richer evaluation information.

[0084] In practical applications, step S301 aims to evaluate the veracity or factuality of the statements made by each claim node in the logical dependency graph. By constructing a first-type verification question, the verification model is guided to perform independent fact-checking on each claim:

[0085] To ensure the independence of verification and avoid contextual interference, a completely new and clean dialogue session is created for each claim node v_i to be verified. During the initialization phase of each session, a preset system prompt is injected into the verification model, such as Verifier_LLM, for example: "You are a rigorous fact checker, making judgments on given claims only based on known facts." This fixes the behavior pattern of the verification model.

[0086] Based on the text content text_i of node v_i, a verification instruction is automatically generated using a predefined fact-checking question template. For example, the template might be "Please determine whether the following claim is true: '[text_i]'", and filling in text_i yields the specific question Q_factual(v_i), which aims to guide the verification model to output a judgment that can be mapped to a finite discrete state.

[0087] In a separate session, Q_factual(v_i) is sent to the validation model, and its natural language response A_i is obtained. This A_i is then parsed using keyword matching, intent recognition, or other semantic analysis methods. The goal is to classify the model's free-text response into one of three predefined discrete validation states: "Supported," "Refuted," or "Uncertain." After parsing, the validation state Verdict_i of that node is recorded.

[0088] To facilitate subsequent quantization and aggregation, the discrete verification state Verdict_i is mapped to a numerical claim verification score S(v_i). In a preferred embodiment, the mapping rule is as follows:

[0089] If Verdict_i = “Supported”, then S(v_i) = 1.0;

[0090] If Verdict_i = “Refuted”, then S(v_i) = 0.0;

[0091] If Verdict_i = “Uncertain”, then S(v_i) = 0.5. This score S(v_i) objectively reflects the single

[0092] The credibility of a claim after fact-checking.

[0093] Step S302 aims to evaluate whether the logical relation represented by the directed edge connecting two claims in the logical dependency graph holds and its strength. By constructing a second type of verification question, the verification model is guided to logically evaluate the relation:

[0094] For each directed edge e_k = (v_i, v_j, type_k) in the logical dependency graph, select the corresponding relation validation template based on its relation type_k:

[0095] If type_k is "Supports": the generated question aims to ask "To what extent does claim '[text_i]' support claim '[text_j]'?";

[0096] If type_k is "Contradicts": the generated question aims to ask "To what extent does claim '[text_i]' contradict claim '[text_j]'?";

[0097] If type_k is “RelatedTo”: deep strength verification can usually be skipped, or a default correlation score can be directly assigned.

[0098] In a separate, clean session, the generated relation verification problem Q_relation(e_k) is sent to the verification model, and a response A_k is obtained. A_k is parsed and mapped to a predefined set of relation strength levels. For Supports relations, the strength levels may include: "Strong_Support", "Weak_Support", and "Uncertain_Support". For Contradicts relations, the strength levels may include: "Strong_Contradiction", "Weak_Contradiction", and "Uncertain_Contradiction". After parsing, the strength state Verdict_k of the edge is recorded.

[0099] The relation strength state Verdict_k is mapped to a numerical relation strength score R(e_k), which can be positive or negative to distinguish between support and contradiction. In a preferred embodiment, the mapping rule is as follows:

[0100] For the Supports edge:

[0101] R(e_k) = 1.0 if Verdict_k = “Strong_Support”; R(e_k) = 0.2 if Verdict_k = “Weak_Support”; R(e_k) = 0.5 if Verdict_k = “Uncertain_Support” (neutral with a positive bias).

[0102] For Contradicts edges:

[0103] R(e_k) = -1.0 if Verdict_k = “Strong_Contradiction”; R(e_k) = -0.2 if Verdict_k = “Weak_Contradiction”; R(e_k) = 0.0 if Verdict_k = “Uncertain_Contradiction” (considered as no explicit contradiction).

[0104] For the RelatedTo edge: a small positive value can be assigned, such as R(e_k) = +0.1, to indicate a low-strength positive association.

[0105] By executing steps S301 and S302 in parallel in this embodiment, a deep, structured verification of all core logical elements (claims and relationships) in the logical dependency graph is completed. The final verification feedback is a structured dataset containing the following information:

[0106] The verification status {Verdict_i} and claim score {S(v_i)} of all nodes;

[0107] The strength states {Verdict_k} and relation strength scores {R(e_k)} of all directed edges;

[0108] Optional original question-and-answer records {A_i, A_k} are used for tracing and interpretation.

[0109] Through the above technical solution, this application effectively solves the problem of insufficient verification depth in existing technologies. Specifically, by performing differentiated verification on nodes and directed edges in the logical dependency graph, fine-grained examination of the internal logical structure of the text reasoning chain is achieved. Generating a first type of verification question for each node and determining its verification state ensures that the factual truthfulness of each atomic claim is independently verified, avoiding ambiguity in the overall evaluation, thereby accurately identifying factual errors in the reasoning chain. Simultaneously, generating a second type of verification question for directed edges and determining their strength state specifically evaluates the logical dependencies between claims, addressing the lack of logical coherence evaluation, and thus effectively identifying logical weaknesses in the reasoning chain.

[0110] In another embodiment, step S103 is followed by the following step:

[0111] S113: Perform counterfactual reasoning tests on the logical dependency graph to obtain a counterfactual consistency measure; Step S104 includes:

[0112] S401a: Based on the verification feedback and the counterfactual consistency measure, generate the credibility assessment result.

[0113] Specifically, step S113, added after step S103 and before S104, aims to simulate whether the conclusion will logically change if the premise changes, thereby verifying the robustness and consistency of the logical structure upon which the reasoning chain depends. In this step, a counterfactual reasoning test is performed on the logical dependency graph, aiming to explore the inherent logical robustness of the textual reasoning chain by simulating hypothetical scenarios. This can include systematically selecting one or more nodes in the logical dependency graph and hypothetically changing their states—for example, changing the truth value of a premise from "true" to "false," or modifying the strength of a causal relationship—and then observing how these changes affect the states of other nodes in the graph or the conclusion of the entire reasoning chain. Another possible implementation is to introduce an external, hypothetical event or condition that logically affects certain nodes in the graph, thereby evaluating the logical coherence of the reasoning chain and the validity of the conclusion under these new conditions.

[0114] After completing the counterfactual reasoning test, a counterfactual consistency metric is required. This metric quantifies the degree to which the logical dependency graph maintains consistency in its logical structure or conforms to expected behavior when faced with hypothetical changes. For example, a numerical score can be calculated based on the degree of match between the observed logical changes in the graph and the expected logical derivation after a counterfactual intervention. A higher consistency metric indicates a high degree of consistency. Alternatively, the metric can be a qualitative assessment, such as classifying consistency into "high," "medium," and "low" levels, depending on the number and severity of detected logical contradictions or unexpected deviations. The feasibility and effectiveness of the counterfactual reasoning test demonstrated in this step depend heavily on the integrity of the aforementioned logical dependency graph. It is precisely because the reasoning chain has been modeled as a directed graph containing causal / supporting directions that the system can erroneously select intervention nodes and deduce the expected impact along the directed edges. This graph-based, systematic "if-then" inquiry is the only rigorous way to conduct robustness testing. It proactively exposes "fragile reasoning chains" that seem reasonable under the original premises but fail to maintain consistency in logical conclusions when key premises change. This is fundamentally different from existing techniques that lack structured representations and can only perform arbitrary or superficial counterfactual inquiries. Therefore, counterfactual testing is a key module in this method for evaluating logical robustness, and its foundation lies in the reasoning structure formalized by the logical dependency graph.

[0115] In step S401a, the credibility assessment result is generated based on the verification feedback and the counterfactual consistency measure. This means integrating two different types of assessment information. The verification feedback mainly focuses on the factual correctness or direct validity of the logical relationships of each logical element in the reasoning chain, while the counterfactual consistency measure focuses on the logical robustness of the reasoning chain under hypothetical conditions. This integration can be achieved in various ways. For example, the verification feedback and the counterfactual consistency measure can be quantified separately and then fused using methods such as weighted averaging, multi-criteria decision analysis, or machine learning models to obtain a comprehensive credibility assessment score or level.

[0116] Through the above technical solution, this application can systematically examine the robustness and consistency of the internal logic of a textual reasoning chain. This solves the problem that relying solely on verification feedback cannot expose logical vulnerabilities, because verification feedback mainly focuses on checking the current facts, while counterfactual testing can actively explore the response of the reasoning chain when the premises change. By combining verification feedback (reflecting factual correctness) with counterfactual consistency measures (reflecting logical robustness), this application can generate more comprehensive and reliable credibility assessment results.

[0117] In other embodiments, step S113 includes the following steps:

[0118] S1131: Select an intervention node from the logical dependency graph; S1132: For the intervention node, generate a counterfactual query to explore the impact on other nodes in the graph when its state changes; S1133: Based on the response of the verification model to the counterfactual query, compare it with the expected impact derived from the structure of the logical dependency graph to obtain the counterfactual consistency measure.

[0119] Specifically, in step S1131, intervention nodes are selected from the logical dependency graph to provide a clear starting point for counterfactual reasoning testing. Intervention nodes are specific nodes in the logical dependency graph whose states—e.g., truth value, degree, or whether a condition is satisfied—will be hypothetically changed to observe the impact of this change on the entire reasoning chain. By selectively choosing these nodes, the testing can be ensured to focus on the core claims or key logical links in the reasoning chain, thereby improving the efficiency and relevance of the testing and avoiding the ineffectiveness that may result from blind testing.

[0120] The selection of intervention nodes can be based on various strategies. For example, nodes can be selected based on their structural importance in the logical dependency graph, such as choosing nodes with high centrality, for example, nodes with high in-degree or out-degree, which typically play a crucial role in the inference chain. Alternatively, nodes can be selected according to pre-defined rules, such as selecting all nodes that serve as initial premises of the inference chain or those that exhibit low confidence in the initial verification. Another possible approach is to randomly select a certain proportion of nodes from the graph as intervention nodes to comprehensively explore the robustness of the logical dependency graph and discover non-obvious vulnerabilities.

[0121] Step S1132 is the core step of counterfactual reasoning. This step constructs a hypothetical scenario, simulates a change in the state of the intervention node, and uses this as a basis to pose questions to the verification model to assess how this change affects the states of other relevant nodes in the logical dependency graph. This dynamic, hypothetical approach reveals the behavioral patterns of the reasoning chain under different preconditions, thus compensating for the inability of static evaluation to capture dynamic changes and helping to discover potential logical flaws. Specifically, counterfactual questions can be generated in various forms. For example, for an intervention node representing a factual claim, if its original state is "true," the question "If this claim is false, how will other claims dependent on it change?" can be generated; and vice versa. For claims involving quantification or degree, the degree can be modified, for example, changing "significantly increased" to "slightly increased," and then the impact on subsequent nodes can be inquired. Furthermore, specific counterfactual questions can be automatically generated using a pre-set counterfactual question template, combined with the content of the intervention node and its contextual information in the logical dependency graph, to ensure the accuracy and effectiveness of the questions.

[0122] Step S1133 aims to quantify the robustness of the inference chain. This step compares the actual response of the verification model under counterfactual conditions with the theoretically expected result derived from the given structure of the logical dependency graph. This comparison objectively assesses whether the internal logic of the inference chain remains consistent when the hypothetical premises change. If the response of the verification model closely matches the expected impact, it indicates that the inference chain has strong logical robustness; otherwise, there may be logical inconsistency or fragility. The method for obtaining counterfactual consistency measures can be varied. For example, for binary judgments returned by the verification model, such as "true" or "false," they can be directly compared one by one with the expected truth values ​​derived from the logical dependency graph structure to calculate the consistency ratio. If the verification model returns a response in natural language form, natural language processing techniques, such as semantic embedding or natural language inference (NLI) models, can be used to calculate the semantic similarity between the verification model's response and the description of the expected impact to quantify consistency. Furthermore, if the validation model can output probability predictions of node states, these probability distributions can be compared with conditional probability distributions derived from the logical dependency graph structure, for example, using statistical measures such as KL divergence, to obtain a more refined counterfactual consistency measure.

[0123] In practical applications, potential intervention nodes are first identified from the logical dependency graph G. Preferably, the set of nodes V_root with an in-degree of zero in the logical dependency graph, i.e., the initial premises that do not depend on other claims, are selected as intervention points. If no such nodes exist, the top k most important nodes can be selected based on a node centrality metric, such as the PageRank value.

[0124] For each selected intervention node v_x ∈ V_root, define an intervention operation do(v_x = false). This does not mean modifying reality, but rather performing a pre-defined value substitution operation on the state variable of that node in the computational model, such as setting its claimed truth value to "false", to generate a contrasting input condition different from the original state.

[0125] For each intervention node v_x and each downstream target node v_y reachable in the graph, construct a specific counterfactual question Q_cf based on a predefined counterfactual question template. For example: "In the original reasoning, claim 'A' supports claim 'B'. Now, assuming 'A' is false, how do you think the probability of 'B' being true will change? A) Decrease B) Remain unchanged C) Increase D) Cannot be determined."

[0126] In a new, independent session, Q_cf is sent to the validation model, which acts as a controlled response generation unit and outputs its choice, such as "A".

[0127] Simultaneously, based on the structure of the original logical dependency graph G, the theoretical expected impact from the intervention node v_x to the target node v_y is automatically derived. This is the key difference between this invention and simple causal queries; it is based on graph structure for computation: a. Path discovery: finding one or more directed paths from v_x to v_y in the graph; b. Symbol function definition: mapping the relation types of edges to logical symbols. For example, define sign(“Supports”) = +1, sign(“Contradicts”) = -1, sign(“RelatedTo”) = 0; c. Path synthesis relation calculation: For a path p = [e1, e2, ..., e_n], its synthesis logical effect R_path is defined as the product of all edge relation symbols on the path: R_path = Π sign(e.type). If R_path > 0, it indicates a net support effect; if R_path < 0, it indicates a net contradiction effect; if R_path = 0, the path contains RelatedTo edges or the symbol product is zero, indicating an unclear logical direction; d. Determine the expected effect: Based on R_path, determine the expected effect Expected_Effect: If R_path > 0, then Expected_Effect = "reduction" (the premise is not true, weakening the net support for the conclusion); if R_path < 0, then Expected_Effect = "increase" (the premise is not true, reducing the net contradiction for the conclusion, equivalent to relative enhancement); if R_path = If the value is 0, then Expected_Effect = "unchanged".

[0128] The actual response of the verification model to counterfactual inquiries is analyzed as "decrease", "increase", "no change" or "cannot be determined", and compared with the theoretical expected effect Expected_Effect calculated in the previous step.

[0129] According to the preset consistency mapping rules, a consistency score Consistency(v_x, v_y) is calculated for each pair (v_x, v_y).

[0130] For example: if the actual response is exactly the same as Expected_Effect, then Consistency = 1.0;

[0131] If the actual response is "cannot be determined", then Consistency = 0.5;

[0132] If the actual response is inconsistent with Expected_Effect, then Consistency = 0.0.

[0133] The average consistency score of all intervention-target pairs (v_x, v_y) across all tests is used to obtain the overall counterfactual robustness score (CF_Score), which is the aforementioned counterfactual consistency measure. A higher CF_Score indicates that the logical structure of the reasoning chain is more robust and consistent under counterfactual pressure.

[0134] Through the above technical solution, this application can systematically execute counterfactual reasoning tests, solving the problems of traditional methods lacking specific implementation mechanisms and failing to effectively expose logical vulnerabilities. By selectively choosing intervention nodes, the efficiency and focus of the test are ensured. By generating diverse counterfactual inquiries, changes in preconditions in the real world are simulated, allowing for a dynamic exploration of the robustness of the reasoning chain. More importantly, by comparing the response of the verification model with the expected impact of the logical dependency graph structure derivation, this application provides an objective and quantifiable counterfactual consistency measure, making the assessment of the internal logical robustness of the reasoning chain no longer a vague judgment, but a scientifically based quantitative result. This measurement result can serve as an important basis for generating credibility assessment results for textual reasoning chains, significantly improving the comprehensiveness, operability, and interpretability of the assessment. This allows for a more accurate identification of seemingly reasonable but potentially fragile reasoning that collapses under changing preconditions, providing solid technical support for the application of trustworthy artificial intelligence.

[0135] In other embodiments, step S102 is followed by the step:

[0136] S112: Evaluate the construction quality of the logical dependency graph and obtain a graph construction quality score;

[0137] Step S104 includes:

[0138] S401b: Based on the graph, construct a quality score and the verification feedback to generate a credibility assessment result for the reasoning chain of the text.

[0139] Specifically, after successfully constructing the logical dependency graph, a graph construction quality assessment step is added. The graph construction quality score obtained from the assessment is used as an important piece of evidence, along with subsequent verification feedback, to generate the final credibility assessment result. This aims to quantify the reliability and fidelity of the transformation process from textual reasoning chains to structured logical dependency graphs, thereby ensuring that subsequent in-depth verification and analysis are based on accurate and complete intermediate representations. In step S112, the construction quality of the logical dependency graph is assessed. This aims to measure the success or failure and quality of the structured transformation through multi-dimensional indicators. This may include: assessing whether the structured extraction instructions based on the language model are successfully executed and parsed, and calculating the parsing success rate; checking whether there are circular dependency structures in the generated directed graph that violate logical decidability, and calculating the graph structure consistency; and analyzing whether the abstraction granularity from the original text to the claim nodes is reasonable, and calculating the extraction granularity matching degree.

[0140] After completing the graph construction quality assessment, a graph construction quality score needs to be obtained. This score quantifies the reliability of the logical dependency graph as the basis for subsequent analysis. For example, a comprehensive numerical score can be calculated by weighting and fusing the three core indicators mentioned above—parsing success rate, graph structure consistency, and extraction granularity matching—using preset weights. A higher quality score is achieved if the extraction is successful, the structure is clear, and the granularity is reasonable. Alternatively, the score can also be a segmented or categorized assessment, such as assigning different baseline quality levels based on the source of the extraction results, such as successful parsing, parsing after retrying, or rule backup generation.

[0141] In step S401b, the credibility assessment result is generated based on the verification feedback and the graph construction quality score. This means integrating verification information reflecting the correctness of the content with quality information reflecting the reliability of the representation. The verification feedback mainly focuses on the factual and logical strength of each claim and relationship in the reasoning chain, while the graph construction quality score focuses on the soundness of the intermediate data structure itself supporting these verifications. This integration can be achieved in various ways. For example, the verification feedback can be quantified into a first score, such as QA_Score, and then fused with the graph construction quality score through weighted averaging, product adjustment, or as a decision threshold condition to obtain a more robust overall credibility score or level. For example, even if the verification score is high, if the graph construction quality score is low, the final credibility will be lowered because the verification feedback on which deep verification depends may be inaccurate.

[0142] By employing the aforementioned technical solution, this application adds a crucial pre-assessment quality control step to the evaluation process. This addresses the problem of decreased confidence in evaluation results caused by directly using potentially distorted or incomplete structural representations for deep analysis. By combining verification feedback—reflecting content correctness—with graph construction quality scores—reflecting fundamental reliability—more rigorous and robust confidence assessment results can be generated.

[0143] In another embodiment, step S401a includes,

[0144] S411a: Quantify the verification feedback into a first score and quantify the counterfactual consistency measure into a second score; S421a: Weight and fuse the first score and the second score to obtain a comprehensive credibility score; S431a: Determine the credibility level of the text reasoning chain based on the comprehensive credibility score.

[0145] Specifically, in step S411a, the verification feedback is quantified into a first score. This aims to transform the verification feedback obtained from multiple rounds of verification interactions between the verification model and logical elements into a quantitative form that can be numerically calculated and compared. This verification feedback may include the results of checking the veracity of logical claims, such as true, false, or uncertain, and the assessment of the strength of logical dependencies between claims, such as strong, weak, or contradictory. One possible implementation is to map different types of verification results to specific numerical ranges according to preset scoring rules. For example, claims confirmed as true or strong logical dependencies by the verification model are assigned higher scores; claims confirmed as false or contradictory logical dependencies are assigned lower scores; and uncertain or weak verification results are assigned intermediate scores. Subsequently, the first score representing the overall verification quality can be obtained by averaging, weighted summing, or calculating based on a specific aggregation function of the scores of all logical elements. Another possible implementation is to utilize a machine learning model, such as a regression model, using the raw responses output by the verification model, such as confidence scores, semantic similarity scores, or classification labels, as input features, and directly outputting a normalized first score through a trained model. The model can be trained using expert-annotated data to ensure the accuracy and objectivity of the quantification results.

[0146] Simultaneously, quantifying the counterfactual consistency measure into a second score aims to transform the counterfactual consistency measure obtained from counterfactual reasoning tests on logical dependency graphs into a quantifiable form that can be numerically calculated and compared. The counterfactual consistency measure reflects the robustness and consistency of the logical deduction of the textual reasoning chain when the underlying assumptions change. One possible implementation is to calculate it by comparing the degree of matching between the verification model's response to counterfactual inquiries and the expected impact derived from the logical dependency graph structure. For example, if the verification model's response is completely consistent with the expected impact, a high score is assigned; if there is partial inconsistency, points are deducted according to the degree of inconsistency; if there is complete inconsistency, a low score is assigned. This degree of matching can be evaluated through semantic similarity calculation, logical implication judgment, or expert rule systems, and ultimately aggregated into a second score representing overall counterfactual consistency. Another possible implementation is to set a series of counterfactual test scenarios and define an expected result for each scenario. After executing the test, a local consistency score is assigned to each scenario based on the degree of deviation between the actual result and the expected result. Then, the final second score is obtained by averaging or weighted averaging all local consistency scores.

[0147] Building upon this, step S421a involves a weighted fusion of the first and second scores to obtain a comprehensive credibility score. The aim is to integrate the first and second scores, representing verification feedback and counterfactual consistency respectively, into a single, comprehensive credibility score through a weighted combination. One implementation method is to use a linear weighted summation: Comprehensive Credibility Score = w1 * First Score + w2 * Second Score, where w1 and w2 are preset weight coefficients, typically satisfying w1 + w2 = 1. These weights can be adjusted based on the specific application scenario's emphasis on factual accuracy versus logical robustness. For example, in scenarios with higher requirements for factual accuracy, the value of w1 can be appropriately increased. Another possible implementation is to construct a non-linear fusion model, such as a model based on neural networks or decision trees. This model uses the first and second scores as input features and automatically learns and outputs a comprehensive credibility score by learning the comprehensive credibility assessment results of experts on the text reasoning chain from historical data. This method can capture more complex interaction relationships and provide a more refined fusion effect.

[0148] Finally, in step S431a, the credibility level of the text reasoning chain is determined based on the comprehensive credibility score, aiming to convert the continuous comprehensive credibility score into a discrete, easily understood, and manipulable credibility level. One possible implementation is to preset a series of score thresholds to divide the comprehensive credibility score into different level ranges. For example, the score can be divided into multiple levels such as "high credibility," "medium credibility," "low credibility," and "untrustworthy." These thresholds can be adjusted according to actual needs and domain expert experience. Another possible implementation is to combine natural language generation technology to automatically generate a structured report containing the credibility level, score details, and key evaluation criteria based on the numerical range of the comprehensive credibility score. For example, when the score is in the "medium credibility" range, the report can further indicate that it is due to certain weaknesses in the verification feedback or certain instabilities in the counterfactual consistency test.

[0149] Through the above technical solution, this application effectively solves the technical problem that the lack of a standardized quantification mechanism leads to subjective and inconsistent evaluation results when generating credibility assessment results based on verification feedback and counterfactual consistency measures, and makes it difficult to provide interpretable and auditable outputs. Specifically, quantifying verification feedback as a first score allows the verification results of the authenticity and logical strength of logical elements to be transformed into an objective and comparable numerical form, avoiding bias caused by subjective judgment and laying the foundation for subsequent integration. Simultaneously, quantifying the counterfactual consistency measure as a second score provides a numerical measure of the robustness of the reasoning chain in counterfactual testing, enhancing the comprehensiveness of the assessment and ensuring a systematic consideration of the inherent logical robustness of the reasoning chain. Based on this, the first and second scores are weighted and integrated to obtain a comprehensive credibility score. This not only achieves effective integration of evaluation results from different dimensions, but also allows for adaptation to the evaluation focus requirements of different application scenarios by adjusting the weights, ensuring the balance, consistency, and flexibility of the evaluation results. Ultimately, the credibility level of the text reasoning chain is determined based on the comprehensive credibility score, mapping the complex numerical evaluation results to an easily understandable grading system. This results in a structured evaluation report, significantly improving the interpretability and traceability of the evaluation results and meeting the stringent requirements for auditable decision-making processes in high-risk application scenarios. This quantitative, integrated, and tiered mechanism transforms the credibility assessment of text reasoning chains from qualitative to quantitative, and from fragmented to unified, significantly enhancing the objectivity, comprehensiveness, and practicality of the evaluation.

[0150] In other embodiments, step S401b includes:

[0151] S411b: Quantify the verification feedback as a first score;

[0152] S421b: Weight and fuse the first score and the graph construction quality score to obtain a comprehensive credibility score; S431b: Based on the comprehensive credibility score, determine the credibility level of the text reasoning chain.

[0153] Specifically, step S411b aims to transform the verification feedback obtained from multiple rounds of verification interactions between the verification model and logical elements into a quantitative form that can be numerically calculated and compared. This verification feedback includes the results of verifying the truthfulness of logical claims and the assessment of the strength of logical dependencies between claims. One possible implementation is to map the judgment responses returned by the verification model, such as "Supported," "Refuted," and "Uncertain," and the relationship strength levels, such as "Strong_Support" and "Weak_Contradiction," into numerical scores according to a preset mapping rule. Subsequently, using a preset aggregation function, such as normalizing the original total score to the theoretical extreme range, the scores of all logical elements are calculated to obtain the first score representing the overall verification quality, i.e., the question-and-answer verification score (QA_Score). Another possible implementation is to use a statistical model to comprehensively analyze the results of multiple independent verification sessions, or to calibrate based on the confidence information contained in the verification model's output response, to output a more robust QA_Score.

[0154] Step S421b aims to integrate the verification score, representing content accuracy, and the quality score, representing the reliability of the intermediate representation, into a single, comprehensive overall trust score through a weighted combination. One implementation is to use a linear weighted summation method, i.e., the overall trust score (TrustScore) = w_qa * QA_Score + w_q * Q_graph, where w_qa and w_q are preset weight coefficients, and typically satisfy w_qa + w_q = 1. These weights can be adjusted according to the different emphases on content verification accuracy and data structure reliability based on the specific application scenario. For example, in high-risk scenarios where the evaluation basis must be absolutely reliable, the value of w_q can be appropriately increased; while in general scenarios where more attention is paid to the accuracy of the final content, w_qa is given a higher weight. A typical value of w_q is small, such as 0.1, used to fine-tune or correct QA_Score. Another possible implementation is to use a non-linear fusion or conditional adjustment model. For example, Q_graph can be used as a confidence coefficient, and a penalty factor can be applied to TrustScore or its highest level can be directly limited when Q_graph falls below a certain threshold; or, Q_graph can be used as a feature input into a more complex decision model that can learn a more refined interaction between weights and scores.

[0155] Finally, step S431b aims to convert the continuous composite confidence score into discrete, easily understood, and manipulable confidence levels. One possible implementation is to pre-define a series of score thresholds or composite rules associated with graph construction quality, mapping the composite confidence score and / or the value of Q_graph itself to different level intervals. These rules can more flexibly handle the uncertainty brought about by graph construction quality. Another possible implementation is to automatically add descriptive text about the confidence level of the assessment results based on the value of Q_graph when generating a structured report containing confidence levels. For example, when Q_graph is low, the report could state, "This assessment is based on an intermediate structure representation with lower confidence; the conclusions are for reference only."

[0156] In practical applications, firstly, two core quantization inputs are received from the preceding module:

[0157] First score (Question-Answer Validation Score, QA_Score): As described in the previous embodiment, this score is calculated based on the aggregated validation feedback of all nodes and edges in the logical dependency graph and normalized to the interval [0, 1].

[0158] Graph Construction Quality Score (Q_graph): As described in the previous embodiments, this score is based on the evaluation of the logical dependency graph construction process, for example, combining parsing success rate, graph structure consistency and extraction granularity matching degree, and is also normalized to the [0, 1] interval.

[0159] Secondly, a weighted fusion model is pre-defined. This model assigns a weight coefficient to each input score and calculates the final comprehensive trust score through linear weighted summation. For example:

[0160] TrustScore = w_qa * QA_Score + w_q * Q_graph

[0161] Where w_qa + w_q = 1. The weighting coefficients w_qa and w_q are configurable parameters of the system. In a typical setting, w_qa = 0.9 and w_q = 0.1 can be set, which reflects the basic principle of evaluation focusing on content verification while using structure to fine-tune the quality. The value range of TrustScore is also [0, 1].

[0162] Next, the calculated TrustScore is mapped to a preset threshold and a rule set incorporating Q-graph to obtain the trust level. This decision rule set considers both the total score and the Q-graph, for example:

[0163] Conditions for being deemed "highly trustworthy" (all must be met):

[0164] TrustScore > θ_high, for example, θ_high = 0.7

[0165] QA_Score > θ_qa, for example, θ_qa = 0.6

[0166] There are no claims that are explicitly refuted;

[0167] Q_graph > θ_q, for example, θ_q = 0.5, which ensures that the quality of the structural representation on which the evaluation depends is up to standard itself.

[0168] Conditions for triggering "partially trusted" or downgrading (meeting any of the following):

[0169] θ_low < TrustScore ≤ θ_high, for example, θ_low = 0.4;

[0170] There are refuted claims.

[0171] Q_graph is low (for example, Q_graph < θ_q). Especially when Q_graph is generated from the rule fallback, even if TrustScore is high, the system may actively determine it as "partially trusted".

[0172] Conditions for being judged "untrusted" or triggering exception handling:

[0173] TrustScore ≤ θ_low.

[0174] Q_graph is extremely low or invalid (for example, Q_graph is close to 0 or corresponding Parse_Success = 0.0). In this case, the system may output "evaluation failed" instead of the normal grade.

[0175] Finally, output a structured verification report. This report is a complete manifestation of the above credibility evaluation results. In addition to including the final grade, details of TrustScore, QA_Score, etc., it especially newly adds:

[0176] Detailed sub - scores: Clearly list graph_construction_confidence, that is, Q_graph.

[0177] Details of graph construction quality (optional): Can record in detail sub - item scores such as Parse_Success, Graph_Coherence, Granularity_Ratio and their calculation weights.

[0178] System Notes / Warnings: If Q_graph is low or originates from a backup rule, the report will automatically add a corresponding explanation to remind users to pay attention to the basic reliability of the evaluation results.

[0179] For example, in a high-quality graph foundation: Q_graph = 0.95, QA_Score = 0.88 for a certain inference chain. TrustScore = 0.9 * 0.88 + 0.1 * 0.95 = 0.887. Since all criteria meet the "highly trustworthy" requirement, it is classified as this level.

[0180] Low-quality graph correction: Due to ambiguity in the description of a certain inference chain, the backup rule was triggered. Q_graph = 0.35, QA_Score = 0.80, TrustScore = 0.9*0.80 + 0.1*0.35 = 0.755. Although the total TrustScore > 0.7, because Q_graph < 0.5, the system judges it as "partially reliable, requiring caution," and notes "Evaluate the logic graph generated based on simplified rules."

[0181] Through the above technical solution, this application effectively addresses the technical problem that relying solely on content verification feedback while neglecting the quality of the intermediate data structures upon which it depends in credibility assessment can lead to unreliable results. Specifically, by quantifying verification feedback into a first score (QA_Score) and quantifying the quality assessment of the graph construction process into an independent score (Q_graph), the "foundation" and "superstructure" of the assessment are considered separately. Based on this, a weighted fusion of the two scores yields a comprehensive credibility score (TrustScore), ensuring that the final result reflects both the correctness of the content and is corrected for by the reliability of the underlying representation, thus guaranteeing the robustness of the assessment. Finally, the credibility level is determined based on the fused score and composite rules including the Q_graph, generating a report containing quality details. This provides crucial "meta-assessment" information for the assessment conclusion—information on the reliability of the assessment itself. This mechanism significantly enhances the transparency, interpretability, and robustness to poor input of the entire system, making the assessment results more valuable and supportive of decision-making.

[0182] In summary, in other possible implementations, step S104 includes:

[0183] S401: Based on the verification feedback, the quality score constructed from the graph, and the counterfactual consistency measure, generate a credibility assessment result for the reasoning chain of the text.

[0184] Step S401 includes:

[0185] S411: Quantify the verification feedback into a first score and the counterfactual consistency measure into a second score; S412: Weight and fuse the first score, the second score, and the graph construction quality score to obtain a comprehensive credibility score; S413: Determine the credibility level of the text reasoning chain based on the comprehensive credibility score.

[0186] Specifically, this embodiment aims to establish a more complete and robust credibility assessment framework. This framework simultaneously considers verification feedback reflecting the correctness of the content, counterfactual test results reflecting the robustness of the logic, and graph construction quality reflecting the basic reliability, achieving a comprehensive assessment through the weighted fusion of these three factors.

[0187] In step S411, the verification feedback is quantified into a first score (i.e., the question-and-answer verification score, QA_Score), and the counterfactual consistency measure is quantified into a second score (i.e., the counterfactual robustness score, CF_Score). The quantification method can refer to the aforementioned embodiment. For example, QA_Score is obtained by normalizing and aggregating the verification scores of all claims and relations; CF_Score is obtained by calculating the average consistency between the observed effect and the theoretical expected effect in the counterfactual test, and both are normalized to the interval [0, 1].

[0188] In step S421, the first score (QA_Score), the second score (CF_Score), and the graph construction quality score (Q_graph) are weighted and fused to obtain the comprehensive trust score (TrustScore). This is the core innovation of this embodiment, aiming to balance the three dimensions of content, logic, and basic representation. A standard implementation method is to use a linear weighted fusion model:

[0189] TrustScore = w_qa * QA_Score + w_cf * CF_Score + w_q * Q_graph, where w_qa, w_cf, and w_q are the preset weighting coefficients of QA_Score, CF_Score, and Q_graph, respectively, and satisfy w_qa + w_cf + w_q = 1. The weighting coefficients are configurable system parameters that can be flexibly adjusted according to the application scenario. One possible configuration is: w_qa = 0.6, w_cf = 0.3, w_q = 0.1. This configuration reflects the principle of prioritizing factual and direct logical verification, while also placing great emphasis on the robustness of the logical structure under stress, and maintaining prudent consideration of the fundamental quality of the evaluation process itself. Q_graph serves as a global confidence adjustment factor.

[0190] In step S431, based on this comprehensive credibility score, determine the credibility level of this text inference chain. This process combines the total score of TrustScore and the performance of each sub-item score, and applies a more refined composite decision rule set. For example:

[0191] Conditions for being judged as "highly credible" (all need to be met):

[0192] TrustScore > θ_high, for example, θ_high = 0.75)

[0193] QA_Score > θ_qa, for example, θ_qa = 0.65

[0194] CF_Score > θ_cf, for example, θ_cf = 0.6

[0195] Q_graph > θ_q, for example, θ_q = 0.5

[0196] There are no explicitly refuted claims;

[0197] Conditions for being judged as "partially credible, proceed with caution" (meeting any of the following):

[0198] θ_low < TrustScore ≤ θ_high, for example, θ_low = 0.45

[0199] Any one of the scores of QA_Score, CF_Score or Q_graph is lower than its corresponding single-item threshold (θ_qa, θ_cf, θ_q)

[0200] There are refuted claims;

[0201] Conditions for being judged as "uncredible" or "evaluation invalid":

[0202] TrustScore ≤ θ_low

[0203] Q_graph is extremely low, for example, Q_graph < 0.2 or Parse_Success = 0.0, indicating that the basic representation fails

[0204] The key premise is refuted and CF_Score is extremely low.

[0205] For example: Suppose the system obtains the following after evaluating an inference chain:

[0206] QA_Score = 0.88 (the content is basically correct)

[0207] CF_Score = 0.75 (the logic is relatively sound)

[0208] Q_graph = 0.70 (Graph construction quality is good)

[0209] Weight configuration: w_qa=0.6, w_cf=0.3, w_q=0.1

[0210] Threshold configuration: θ_high=0.75, θ_qa=0.65, θ_cf=0.6, θ_q=0.5

[0211] Calculate the overall score: TrustScore = 0.6*0.88 + 0.3*0.75 + 0.1*0.70 = 0.528 + 0.225 + 0.07 = 0.823

[0212] Application rule determination:

[0213] TrustScore (0.823)>θ_high (0.75)

[0214] QA_Score (0.88) > θ_qa (0.65)

[0215] CF_Score (0.75) > θ_cf (0.6)

[0216] Q_graph (0.70) > θ_q (0.5)

[0217] No refutation claims; Conclusion: All conditions for "highly credible" are met, therefore it is deemed "highly credible".

[0218] Structured reports reflect:

[0219] In the final evaluation report, the `detailed_breakdown` section will simultaneously display the scores and their weighted contributions across all three dimensions. The explanation in `final_verdict` will explicitly mention the three criteria, such as: "The overall credibility score of this reasoning chain is 0.823, exceeding the threshold. It performs excellently in the three dimensions of fact-checking (0.88), logical robustness test (0.75), and intermediate structure construction quality (0.70), therefore it is judged to be highly credible." The `weakest_links` analysis will also comprehensively examine potential weaknesses from all three dimensions.

[0220] Through this embodiment, the present invention constructs a three-dimensional integrated credibility assessment model: based on a high-quality intermediate representation (Q_graph), with rigorous factual and direct logical verification (QA_Score) as the main body, and with in-depth stress testing (CF_Score) to verify its robustness. The final arbitration (TrustScore and rating) is the comprehensive output of this pyramid structure. This design maximizes the comprehensiveness, robustness, and interpretability of the assessment results, and can adapt to the needs of various complex application scenarios, from general question answering to high-risk decision support.

[0221] like Figure 3 As shown, this application proposes a credibility evaluation system 500 for text reasoning chains, including: a graph construction module 501, used to receive the text reasoning chain to be evaluated and construct a logical dependency graph representing its internal logical structure based on the text reasoning chain; a verification interaction module 502, used to perform multiple rounds of verification interaction on the logical elements in the graph based on the logical dependency graph and obtain verification feedback through a verification model; and an arbitration output module 503, used to generate a credibility evaluation result for the text reasoning chain based on the verification feedback.

[0222] Specifically, in one implementation, the graph construction module 501 extracts logical claims as nodes from the text inference chain using preset grammatical or semantic rules, and identifies causal, parallel, or adversative relationships between claims as directed edges, thereby constructing a logical dependency graph. Alternatively, the graph construction module 501 uses a trained sequence labeling model or relation extraction model to automatically identify key entities as nodes and parse the logical dependencies between entities as directed edges, generating a structured logical dependency graph. In other possible implementations, the graph construction module 501 is configured to map the input inference chain text into a structured extraction request that satisfies preset field constraints based on a predefined prompt word template, and drive the language model to output candidate results containing claim identification information and logical relationship information between claims. The prompt word template is used to constrain the output structure and semantic roles; its specific text form does not affect the technical effect of the present invention.

[0223] The verification interaction module 502 generates structured query statements for each node (representing a logical claim) and directed edge (representing a logical dependency) in the logical dependency graph and sends them to the verification model. For example, for a logical claim, the verification interaction module generates a yes / no question, and the verification model responds with "yes," "no," or "uncertain." For logical dependencies, the verification interaction module generates a logical strength assessment request and collects the text or structured data returned by the verification model as verification feedback. This process ensures independent verification of logical elements through multiple rounds of interaction, avoiding dependence on a single external knowledge base.

[0224] The arbitration output module 503 summarizes and analyzes the verification feedback, such as statistically analyzing the proportion of logical elements that passed verification or identifying inconsistencies in the verification feedback, and finally generates an evaluation report containing a credibility score, error location information, and a detailed chain of evidence. This report can be presented as a quantitative score, qualitative description, or visual graph to indicate the specific performance of the reasoning chain in terms of factuality, logical coherence, and robustness.

[0225] The core innovation of this embodiment lies in combining the structured representation of the logical dependency graph with a multi-round verification interaction mechanism. This enables the step-by-step verification of the logical elements within the text reasoning chain, solving the problems of insufficient verification depth and dependence on external knowledge bases, and achieving a deep, fine-grained, and interpretable credibility assessment effect. Specifically, the graph construction module 501 directly extracts the internal logical structure from the reasoning chain, avoiding strong dependence on external knowledge bases; the verification interaction module 502 performs independent fact-checking and logical strength testing on logical elements through multi-round interactions, ensuring the depth and fine-grainedness of verification; and the arbitration output module 503 integrates verification feedback to generate a structured report, meeting the stringent requirements of traceability and auditability of the decision-making process in high-risk scenarios.

[0226] Through the above technical solution, this application can perform in-depth and fine-grained credibility assessment of text reasoning chains, reduce dependence on external structured knowledge bases, and provide interpretable assessment results. It effectively solves the technical problems of insufficient verification depth, strong dependence on knowledge bases, lack of robustness testing, and poor interpretability in the prior art.

[0227] In other embodiments, this application further proposes a credibility assessment system 500 for a text reasoning chain, which, based on the above system, also includes a counterfactual testing module 504 for performing counterfactual reasoning tests on the logical dependency graph to obtain a counterfactual consistency measure; wherein, the arbitration output module 503 is used to generate a credibility assessment result based on the verification feedback and the counterfactual consistency measure.

[0228] The counterfactual testing module 504 aims to evaluate the behavioral consistency and logical robustness of a logical dependency graph under different preconditions by simulating hypothetical scenarios. It can be a standalone software component, such as an engine specifically designed for causal reasoning, capable of receiving structural information and node states of the logical dependency graph and generating counterfactual queries based on predefined intervention rules. In another implementation, the counterfactual testing module 504 can integrate a large language model (LLM)-based reasoning agent, trained or fine-tuned to perform counterfactual reasoning tasks, for example, by receiving intervention instructions and a serialized representation of the logical dependency graph, and outputting the expected outcome after intervention.

[0229] The purpose of counterfactual reasoning tests is to systematically examine the inherent logical robustness of a logical dependency graph. Specifically, this test involves selecting specific nodes in the logical dependency graph as intervention points, assuming that the states of these intervention points change, and then observing how this change affects the states or logical relationships of other nodes in the graph. One possible implementation is to systematically traverse the key nodes in the logical dependency graph, generating a series of counterfactual queries for each selected intervention node. These queries aim to explore "how a conclusion (i.e., the states of other nodes) would change if a premise (i.e., the state of the intervention node) were different." For example, a specialized causal reasoning algorithm, such as do-calculus based on structural causal models (SCM), can be used to simulate interventions to specific variables and calculate their causal effects on other variables. Another implementation is to provide an intervention node and its hypothetical new state to a validation model, such as a large language model, and ask it to predict the state of other relevant nodes in the graph under this new assumption. This process can be repeated multiple times to cover different intervention nodes and intervention scenarios, thereby comprehensively evaluating the robustness of the logical dependency graph.

[0230] The purpose of obtaining a counterfactual consistency metric is to quantify the logical robustness of a logical dependency graph under counterfactual scenarios. This metric reflects whether the internal logical deductions of the reasoning chain remain consistent and reasonable when faced with changes in premises. One approach is to compare the model's response to counterfactual inquiries with the expected impact derived from the structure of the logical dependency graph and known causal relationships. For example, a semantic similarity score, logical implication score, or direct factual match between the model's response and the expected outcome can be calculated. These scores can be aggregated to form a comprehensive counterfactual consistency metric, for example, by calculating the average consistency score across all test cases. Another approach is to statistically analyze the proportion of times the structure or node states of the logical dependency graph remain unchanged or change as expected under different counterfactual interventions. For example, if a change to a premise leads to a conclusion completely contrary to the expectation, the consistency metric decreases; conversely, if the conclusion changes as expected, the consistency metric is higher. This metric can be a value between 0 and 1, where 1 represents complete consistency.

[0231] The role of the arbitration output module 503 here is to integrate the verification feedback from multiple rounds of verification interactions and the counterfactual consistency measure obtained from the counterfactual reasoning test, thereby generating a more comprehensive and reliable credibility assessment result for the text reasoning chain. One implementation is that the arbitration output module 503 can employ a multi-criteria decision analysis method, for example, quantifying the verification feedback as a first score and the counterfactual consistency measure as a second score, and then fusing these two scores through weighted summation, analytic hierarchy process (AHP), or fuzzy comprehensive evaluation to obtain a comprehensive credibility score. Another implementation is that the arbitration output module 503 can utilize a machine learning model, such as a support vector machine, neural network, or decision tree, which takes the feature representation of the verification feedback and the counterfactual consistency measure as input and outputs the final credibility level or score of the text reasoning chain. This model can learn how to optimally combine this information by training on a corpus with labeled data.

[0232] Specifically, the arbitration output module 503 performs the following operations:

[0233] Score Aggregation: First, based on all claim scores {S(v_i)} and relation strength scores {R(e_k)} obtained in the aforementioned embodiments, a comprehensive question-answer verification score (QA_Score) is calculated. For example, by normalizing the calculation using the formula QA_Score = (ΣS(v_i) + ΣR(e_k) + |E|) / (|V| + 2*|E|), a value between 0 and 1 is obtained, which intuitively reflects the degree of factual and logical support obtained by the reasoning chain.

[0234] Final ruling: The arbitrator uses QA_Score as the core criterion, combined with preset thresholds, such as the high credibility threshold θ_high=0.7, and rules, such as whether there are key refuted claims, to determine the credibility of the reasoning chain as a discrete level, such as "highly credible", "partially credible, but should be treated with caution" or "unreliable".

[0235] Report Generation: Finally, the credibility assessment system 500 generates a structured assessment report for this textual reasoning chain. This report includes at least: the original reasoning chain text, the constructed logical dependency graph structure, a detailed list of verification feedback, the calculated scores for each item, the final credibility level determination, and a textual explanation of the basis for that determination.

[0236] Through the above technical solutions, the text reasoning chain credibility assessment system 500 of this application introduces verification feedback assessment and systematic testing of the logical robustness of the text reasoning chain. The counterfactual testing module 504, by performing counterfactual reasoning tests on the logical dependency graph, can proactively explore whether the internal logical derivation remains consistent when the premises in the reasoning chain undergo hypothetical changes. This allows the system to not only verify the factual truth and logical strength of each logical element in the reasoning chain, but also further assess the entire reasoning chain's resistance to interference and its internal consistency in the face of uncertainty or changes in premises. The arbitration output module 503 integrates verification feedback and counterfactual consistency measurement, thereby generating a credibility assessment result that more comprehensively reflects the reliability of the text reasoning chain and effectively identifies those reasonings that appear reasonable on the surface but are actually logically weak. This dual assessment mechanism, combining fact-checking and logical robustness testing, significantly improves the depth, granularity, interpretability, and reliability of the assessment results. Especially in high-risk application scenarios, it can provide more convincing audit evidence and avoid erroneous decisions due to ignoring potential logical vulnerabilities.

[0237] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0238] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A method for evaluating the credibility of a text reasoning chain, characterized in that, include: Receive the text reasoning chain to be evaluated; based on the text reasoning chain, construct a logical dependency graph representing its internal logical structure; wherein, the logical dependency graph includes a set of nodes and a set of directed edges, the nodes correspond to logical claims, and the directed edges correspond to logical dependencies between claims; based on the logical dependency graph, perform multiple rounds of verification interaction on the logical elements in the graph through a verification model to obtain verification feedback; wherein, the multiple rounds of verification interaction include: For each node in the logical dependency graph, a first type of verification question is generated to verify the authenticity of the claim, and the verification status of the node is determined based on the response of the verification model. For the directed edges in the logical dependency graph, a second type of verification question is generated to check the logical strength of the relationship, and the strength state of the directed edge is determined according to the response of the verification model; counterfactual reasoning test is performed on the logical dependency graph to obtain a counterfactual consistency measure; based on the verification feedback and the counterfactual consistency measure, a credibility evaluation result for the text reasoning chain is generated.

2. The credibility assessment method for text reasoning chains according to claim 1, characterized in that, The construction of a logical dependency graph representing the internal logical structure based on the text inference chain includes: providing prompt information to the first language model to drive the first language model to extract the nodes and directed edges from the text inference chain.

3. The credibility assessment method for text reasoning chains according to claim 1, characterized in that, The step of performing counterfactual reasoning tests on the logical dependency graph to obtain a counterfactual consistency measure includes: selecting intervention nodes from the logical dependency graph; generating counterfactual queries for the intervention nodes to explore the impact on other nodes in the graph when their state changes; and comparing the response of the verification model to the counterfactual queries with the expected impact derived from the structure of the logical dependency graph to obtain the counterfactual consistency measure.

4. The credibility assessment method for text reasoning chains according to claim 1, characterized in that, After constructing the logical dependency graph that represents its internal logical structure, the method further includes evaluating the construction quality of the logical dependency graph to obtain a graph construction quality score. The process of generating a credibility assessment result for the text inference chain based on the verification feedback includes: Based on the graph, a quality score is constructed and the verification feedback is used to generate a credibility assessment result for the text reasoning chain.

5. The credibility assessment method for text reasoning chains according to claim 1, characterized in that, The step of generating a credibility assessment result for the text reasoning chain based on the verification feedback and the counterfactual consistency measure includes: quantifying the verification feedback into a first score and quantifying the counterfactual consistency measure into a second score; weighting and fusing the first score and the second score to obtain a comprehensive credibility score; and determining the credibility level of the text reasoning chain based on the comprehensive credibility score.

6. The credibility assessment method for text reasoning chains according to claim 4, characterized in that, The process of generating a credibility assessment result for the text inference chain based on the quality score constructed from the graph and the verification feedback includes: The verification feedback is quantified into a first score; The first score and the graph construction quality score are weighted and fused to obtain a comprehensive credibility score; based on the comprehensive credibility score, the credibility level of the text reasoning chain is determined.

7. A credibility evaluation system for text reasoning chains, characterized in that, include: A graph construction module is used to receive the text reasoning chain to be evaluated and construct a logical dependency graph representing its internal logical structure based on the text reasoning chain; wherein, the logical dependency graph includes a set of nodes and a set of directed edges, the nodes correspond to logical claims, and the directed edges correspond to logical dependencies between claims; a verification interaction module is used to perform multiple rounds of verification interaction on the logical elements in the graph based on the logical dependency graph and obtain verification feedback; wherein, the multiple rounds of verification interaction include: for the nodes in the logical dependency graph, generating a first type of verification question to verify the authenticity of the claims and determining the verification status of the nodes based on the response of the verification model; for the directed edges in the logical dependency graph, generating a second type of verification question to verify the logical strength of the relationships and determining the strength status of the directed edges based on the response of the verification model; The counterfactual testing module is used to perform counterfactual reasoning tests on the logical dependency graph to obtain a counterfactual consistency measure; the arbitration output module is used to generate the credibility assessment result based on the verification feedback and the counterfactual consistency measure.

Images