Traceable information chain output method, electronic device, and program product

Abstract

The present disclosure provides a traceable information chain output method, an electronic device and a program product. The method of the present disclosure comprises: obtaining a question input by a user; determining a plurality of evidence segments related to the question; based on the question and the evidence segments, performing an inference process comprising at least one inference step to generate a final answer; in the inference process, recording a dependency relationship between each inference step and the cited evidence segments; and constructing a reverse traceable chain containing the final answer, the inference steps and the cited evidence segments according to the dependency relationship.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a method, electronic device, and program product for outputting a traceable information chain. Background Technology

[0002] In the process of complex task processing and content generation by intelligent agents, large language models and external knowledge base retrieval enhancement are important ways to perform semantic understanding and logical reasoning (such as decision support and compliance review).

[0003] In existing intelligent agent generation scenarios, end-to-end answer generation often relies directly on large language models, or simply splices and displays retrieved reference fragments after generation. When faced with complex, multi-step reasoning requirements demanding high reliability, both approaches are prone to various problems. For example, end-to-end generation, due to the black-box nature of the model, lacks real-time capture of the dependencies between reasoning steps and evidence fragments, making it impossible to trace the source of the generated results. Conversely, simply splicing and displaying reference fragments leads to redundant explanations and a lack of emphasis on core evidence. Furthermore, the absence of version management and quantitative evaluation mechanisms makes it difficult to audit, verify, and reproduce the generated content. Therefore, an answer generation solution that balances traceability and verifiability is needed. Summary of the Invention

[0004] This disclosure provides methods, electronic devices, and program products for outputting traceable information chains.

[0005] According to a first aspect of this disclosure, a method for outputting a traceable information chain is provided. The method includes: acquiring a question input by a user; identifying multiple pieces of evidence related to the question; performing a reasoning process including at least one reasoning step based on the question and the evidence pieces to generate a final answer; recording the dependency relationship between each reasoning step and the cited evidence piece during the reasoning process; and constructing a reverse traceability chain containing the final answer, the reasoning steps, and the cited evidence pieces, starting from the final answer, based on the dependency relationship.

[0006] In at least one embodiment of this disclosure, determining multiple evidence fragments related to the question includes: retrieving multiple candidate contents related to the semantics of the question from a knowledge base; segmenting the candidate contents into at least one evidence fragment; and assigning a unique evidence identifier to each evidence fragment.

[0007] In at least one embodiment of this disclosure, a reasoning process including at least one reasoning step is performed based on a problem and evidence fragments, including: identifying the intent and decomposing the problem to generate at least one sub-problem; generating a corresponding reasoning step for each sub-problem; and assigning a step identifier to each reasoning step.

[0008] In at least one embodiment of this disclosure, recording the dependency relationship between each reasoning step and the cited evidence fragment includes: when generating the content of the reasoning step, outputting the evidence identifier corresponding to the cited evidence fragment and recording the dependency weight of the cited evidence fragment on the reasoning step; and establishing the dependency relationship between the step identifier of the reasoning step and the evidence identifier.

[0009] In at least one embodiment of this disclosure, establishing a dependency relationship between the step identifier of the reasoning step and the evidence identifier includes: reading the internal state data of the language model when generating the reasoning step through the internal state reading interface of the language model; determining the cited evidence identifier based on the attention intensity of each evidence fragment in the internal state data; and establishing a dependency relationship between the step identifier of the reasoning step and the cited evidence identifier.

[0010] In at least one embodiment of this disclosure, a reverse tracing chain containing the final answer, reasoning steps, and cited evidence fragments is constructed according to the dependency relationship, including: assigning an output identifier to the final answer; generating the reverse tracing chain by using a hierarchical mapping structure with the output identifier as the root node, the step identifiers of each reasoning step as intermediate nodes, and the evidence identifiers of the cited evidence fragments as leaf nodes.

[0011] In at least one embodiment of this disclosure, after constructing the reverse tracing chain, the method further includes: aggregating the dependency weights of the same evidence identifier in at least one reasoning step to obtain the global importance of the evidence fragment to the final answer; sorting the evidence fragments according to their global importance and then accumulating the global importance of each evidence fragment; when the proportion of the accumulated value to the total global importance of all evidence fragments exceeds a proportion threshold, constructing the evidence fragments that have participated in the accumulation into a minimum interpretation set; and generating an interpretation summary based on the minimum interpretation set.

[0012] In at least one embodiment of this disclosure, the dependency weights are determined by: reading the internal state data of the language model during the generation of the inference step through the internal state reading interface of the language model; determining the dependency weights based on the intensity of attention of each evidence fragment in the internal state data; or, parsing the citation markers in the language model output and combining the confidence, citation frequency and text matching degree of the language model output to determine the dependency weights.

[0013] According to a second aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs the method described in the first aspect of any embodiment of this disclosure.

[0014] According to a third aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the method described in the first aspect of any embodiment of this disclosure.

[0015] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect of any embodiment of this disclosure. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 This is a flowchart illustrating the traceable information chain output method provided in this embodiment of the disclosure.

[0018] Figure 2 This is a flowchart illustrating the evidence fragment generation method provided in the embodiments of this disclosure.

[0019] Figure 3 This is a flowchart illustrating the reasoning step generation process provided in an embodiment of the present disclosure.

[0020] Figure 4 This is a flowchart illustrating the dependency construction method provided in the embodiments of this disclosure.

[0021] Figure 5 A flowchart illustrating the dependency establishment method provided in this embodiment of the disclosure.

[0022] Figure 6 This is a flowchart illustrating the reverse tracing chain construction method provided in this embodiment of the disclosure.

[0023] Figure 7 A flowchart illustrating the minimum interpretation set generation method provided in this embodiment of the disclosure.

[0024] Figure 8 This is a flowchart illustrating the dependency weight determination method provided in an embodiment of this disclosure.

[0025] Figure 9 This is a schematic block diagram of a traceable information chain output device according to one embodiment of the present disclosure.

[0026] Figure 10 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0027] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0028] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 This is a flowchart illustrating the traceable information chain output method provided in an embodiment of this disclosure. Figure 1 The method shown includes steps S101 to S105. This method can be applied to the server side (e.g., local server, cloud server, etc.).

[0030] Specifically, Figure 1 The method shown includes step S101: obtaining user input for a question.

[0031] The question discussed here is the starting point of the entire processing flow, and its content determines the scope of subsequent evidence retrieval and the direction of reasoning. Since user questions may be presented in various forms such as text, voice, or images, the system can convert non-textual questions into text representations after acquisition, so that subsequent modules can process them uniformly.

[0032] Step S102: Identify multiple pieces of evidence related to the problem.

[0033] By explicitly retrieving information and using the search results as candidate evidence for reasoning, the materials referenced in subsequent reasoning steps have a clear source, providing foundational data for constructing a traceable information chain. In practical applications, the content in the knowledge base has been pre-segmented into small-granularity evidence fragments, each with corresponding semantic feature representations. Then, the similarity between the question's semantic vector and the semantic vectors of each evidence fragment is calculated, selecting the most relevant evidence fragments as candidate evidence.

[0034] Step S103: Based on the question and the evidence fragment, perform a reasoning process including at least one reasoning step to generate a final answer.

[0035] The user's question and identified evidence fragments are input into a language model, guiding the model to deduce the answer based on the content of the evidence fragments. Unlike the end-to-end black-box generation mode in existing technologies, this approach explicitly breaks down the overall answer generation process into a reasoning process that includes at least one reasoning step. End-to-end generation methods leave intermediate logical states invisible, making fine-grained logical tracing impossible; however, by breaking down the reasoning process into multiple reasoning steps, each logical reasoning step becomes an independent traceable node. This multi-step decomposition principle provides a structured foundation for fine-grained tracing and link construction. When the reasoning process is refined into discrete steps, a precise mapping between each step and its corresponding evidence can be established, thereby organizing scattered data into a clear logical link, enabling accurate recording and auditing of the entire deduction process.

[0036] Step S104: During the reasoning process, record the dependency relationship between each reasoning step and the cited evidence fragment.

[0037] Unlike existing explainable solutions that often employ post-hoc attribution—that is, guessing the connection between the answer and evidence through text similarity matching after the final answer has been generated (a post-analysis method prone to logical gaps and errors)—this approach aims to achieve real-time source tracing. During the real-time execution of the reasoning generation task by the language model, it synchronously acquires and records which evidence fragments are referenced in the generated content of each reasoning step, establishing the dependency relationship between reasoning steps and evidence fragments. For example, a specific prompt template can be used to require the language model to output citation tags synchronously when generating the text for each reasoning step. The system, while streaming the model output, parses these citation tags in real time, thus establishing and recording the dependency relationship between the current reasoning step and the corresponding evidence fragment immediately during generation. This real-time dependency capture mechanism during generation ensures the timeliness and accuracy of source tracing, avoiding errors caused by post-hoc backtesting. Furthermore, recording each reasoning step enables fine-grained source tracing, allowing precise location of supporting evidence for any local logic in the derivation process.

[0038] Step S105: Based on the dependency relationship, construct a reverse tracing chain starting from the final answer, which includes the final answer, the reasoning steps, and the cited evidence fragments.

[0039] In real-world business review scenarios, users or auditors typically start from the final conclusion and trace back to the source of that conclusion. However, existing reasoning processes are mostly presented in a forward-flowing, stream-like manner, lacking an output-oriented traceability structure. The dependencies recorded in the aforementioned manner are scattered across various reasoning steps. These scattered dependencies need to be organized into a structured information chain according to the hierarchical logic of output, reasoning steps, and evidence to achieve complete backtracking from any conclusion. In practical applications, after all reasoning steps are executed, a unique output identifier (Output ID, OID) is assigned to the generated final answer. The output identifier of the final answer is used as the root node, the step identifiers of each reasoning step are used as intermediate nodes, and the evidence identifiers of the cited evidence fragments are used as leaf nodes, forming a complete backtracking path from output, reasoning steps, to evidence fragments. Through this reverse traceability chain, the system achieves the ability to trace back to the specific source of evidence layer by layer from any output conclusion, through the reasoning steps it depends on, thus providing a structured data foundation for the interpretability, verifiability, and reproducibility of the answer.

[0040] Based on the aforementioned publicly available solutions, by explicitly decomposing the reasoning process into at least one reasoning step, a structured foundation is provided for fine-grained tracing and link construction. By recording the dependencies between each reasoning step and the cited evidence fragments in real time during the reasoning process, real-time tracing during the generation phase is achieved, ensuring the timeliness and accuracy of tracing, realizing fine-grained tracing, and avoiding errors from retrospective reasoning. Furthermore, based on these dependencies, a reverse tracing chain is constructed from the final answer, through the reasoning steps, to the evidence fragments, giving the generated results an output-oriented reverse tracing path. This provides a clear, fine-grained data tracing link for the final answer, solving the problems of lack of transparency in generated content and inability to accurately trace back and verify in existing technologies, significantly improving the interpretability and auditability of the agent's generated results.

[0041] In one or more embodiments of this disclosure, such as Figure 2 This is a flowchart illustrating the evidence fragment generation method provided in this embodiment of the disclosure. Figure 2 As shown, identifying multiple evidence fragments related to the question includes: Step S201: Retrieving multiple candidate contents semantically related to the question from the knowledge base. Step S202: Dividing the candidate contents into at least one evidence fragment and assigning a unique evidence identifier to each evidence fragment.

[0042] In practical applications, the user-input question is used as the query condition to perform semantic retrieval on the knowledge base. Specifically, the user-input question is first converted into a semantic vector representation, which represents the core semantic information of the question. The content of each document in the knowledge base has been pre-converted into corresponding semantic vectors using the same semantic feature extraction method and indexed. The similarity between the question's semantic vector and the semantic vectors of each document in the knowledge base is calculated, and the documents or paragraphs with the highest similarity are selected as candidate content, sorted from highest to lowest similarity. Semantic similarity can be calculated using metrics such as cosine similarity or Euclidean distance. The number of candidate contents can be preset to a fixed value based on the actual application scenario, or a similarity threshold can be set, including all content with a similarity exceeding the threshold as candidate content. If no candidate content related to the question's semantics exists in the knowledge base, meaning the retrieved content's semantic similarity to the question is below the preset threshold, a prompt message can be returned to the user explaining that no relevant evidence was found, or the search scope can be expanded to obtain more candidate content.

[0043] Furthermore, since a complete document or paragraph in the knowledge base may contain multiple independent semantic units, if the entire paragraph or document is used as the reference unit, the subsequent reverse tracing chain can only locate at the coarse-grained document level, failing to precisely point to the specific statements or data supporting the conclusion, thus affecting the accuracy of the tracing. By segmenting candidate content into the smallest unit that can independently express a complete semantic information, subsequent reasoning steps can accurately reference evidence fragments, thus providing a foundation for fine-grained dependency recording and reverse tracing chain construction. In specific implementation, the system can segment based on natural paragraph boundaries, sentence boundaries, or semantic completeness in the candidate content. For example, for policy documents, they can be segmented according to article number, with each article serving as an evidence fragment; for contract texts, they can be segmented according to clauses; for narrative documents, they can be segmented according to semantic paragraphs, ensuring that each evidence fragment contains a relatively complete and independent semantic information.

[0044] Furthermore, the system generates a unique evidence identifier for each segmented evidence fragment. This identifier can be encoded in various ways, such as a combination of timestamp and sequence number, a digest value generated by a hash algorithm, or a universally unique identifier. Each evidence identifier is bound to its corresponding evidence fragment content for storage. This allows the system to accurately locate the corresponding evidence fragment content when the language model references it during subsequent reasoning. Simultaneously, this evidence identifier also serves as the source of identification for leaf nodes in the subsequent reverse tracing chain, providing a crucial index for precise backtracking from the final answer to the evidence fragment. If the same evidence fragment is repeatedly referenced in multiple reasoning steps or generation tasks, the system records each instance using this unified evidence identifier, ensuring the consistency and traceability of the reference relationship.

[0045] Based on the publicly available solutions described above, by retrieving candidate content from the knowledge base and segmenting it into fine-grained evidence fragments, and assigning a unique evidence identifier to each fragment, subsequent reasoning, dependency recording, and reverse tracing chain construction can be based on precise and indexable evidence units. By pinpointing the source of evidence to the smallest semantic unit and implementing full lifecycle management of evidence through unique identifiers, the problems of ambiguous source location and inability to accurately trace evidence are solved, providing support for accurate verification of generated conclusions in high-reliability business scenarios.

[0046] In one or more embodiments of this disclosure, such as Figure 3 This is a flowchart illustrating the reasoning step generation process provided in an embodiment of this disclosure. Figure 3 As shown, based on a question and evidence fragments, a reasoning process including at least one reasoning step is performed, including: Step S301: Intent identification and task decomposition of the question to generate at least one sub-question. Step S302: Generate corresponding reasoning steps for each sub-question. And, Step S303: Assign a step identifier to each reasoning step.

[0047] First, the intent recognition module performs semantic parsing on the user's input question, extracting key semantic elements and categorizing the question into a preset intent category. Intent recognition can be achieved using a natural language understanding method based on a classification model, mapping the input text to predefined intent labels. After intent recognition, the system further determines whether the question needs to be broken down based on the intent category and the complexity of the question. If the question structure is complex, containing multiple logical judgment levels or multiple independent points that need to be processed separately, the system breaks down the question into multiple sub-questions, each corresponding to a specific judgment objective or information requirement. For example, for the input question "Does the employee's social security need to be retroactively paid?", the intent recognition result is "Social security retroactive payment judgment," which the system breaks down into two sub-questions: "Judgment whether there is a situation where it is postponed to the next month" and "Judgment whether the difference needs to be retroactively paid." If the question structure is simple, involving only a single judgment or a single information query, the system generates a single sub-question, which is the original question itself.

[0048] Furthermore, based on the decomposed sub-problems, a corresponding reasoning step is created for each sub-problem. Each reasoning step is an independent processing unit for a sub-problem, responsible for logically deriving the sub-problem based on the input evidence fragments and generating intermediate conclusions. The generation of reasoning steps can be achieved by constructing reasoning prompts, inputting the sub-problem and relevant evidence fragments into the language model, and instructing the language model to execute the reasoning operation corresponding to that sub-problem. If multiple sub-problems exist, the system can determine the execution order of the reasoning steps according to the logical dependencies between them. For example, after executing the reasoning step that determines whether a deferral mechanism exists, the reasoning step that determines whether reissue is needed is executed, so that the output of the previous reasoning step can serve as context information for the next reasoning step. By breaking down the reasoning process into reasoning steps that correspond one-to-one with the sub-problems, the system transforms the originally black-box end-to-end generation process into a series of independently identifiable and observable processing steps, providing a structured reasoning flow for achieving fine-grained dependency recording and the construction of a traceable information chain.

[0049] Furthermore, while creating inference steps, a unique step identifier is provided for each inference step. This step identifier is bound to the inference step in this generation task and is used to uniquely mark the inference step in subsequent dependency recording and reverse trace chain construction. The step identifier can be generated using methods such as sequence numbering or timestamp-based strings. By assigning a step identifier to each inference step, the system establishes an index bridge between the inference step and the dependencies of subsequent records, enabling each dependency to accurately correspond to a specific inference step, thereby providing a basis for identifying intermediate nodes in the final constructed reverse trace chain.

[0050] Based on the publicly available solutions described above, by identifying the intent of the problem and breaking it down into sub-problems, and generating corresponding reasoning steps for each sub-problem and assigning step identifiers, the system explicitly transforms the reasoning process of complex problems into a series of independently traceable nodes. This structured decomposition method allows subsequent dependency recording to be performed at the granularity of reasoning steps. The resulting reverse traceability chain accurately reflects which reasoning steps constitute the final answer and which evidence each reasoning step cites, thus achieving higher granular interpretability and traceability than existing end-to-end generation methods. Through this step-by-step identification mechanism, a technical foundation is provided for the transparency and auditing of the generation process, solving the problem that existing solutions cannot provide structured traceability of the reasoning process.

[0051] In one or more embodiments of this disclosure, such as Figure 4 This is a flowchart illustrating the dependency construction method provided in an embodiment of this disclosure. Figure 4 As shown, the dependency relationship between each reasoning step and the cited evidence fragment is recorded, including: Step S401: When generating the content of the reasoning step, the evidence identifier corresponding to the cited evidence fragment is output, and the dependency weight of the cited evidence fragment on the reasoning step is recorded. Step S402: The dependency relationship between the step identifier of the reasoning step and the evidence identifier is established.

[0052] During the content generation process of each reasoning step in the language model, the system simultaneously acquires the evidence identifiers and dependency weights of the evidence fragments referenced in that reasoning step. By requiring the output of the referenced evidence identifiers and recording of dependency weights at the same time as generating the content of the reasoning step, the system achieves precise capture and quantitative recording of the generation basis, ensuring the accuracy and completeness of the traceability information.

[0053] In practical implementation, the system can constrain the output behavior of the language model through a prompting instruction mechanism. This allows the language model to proactively annotate the evidence fragments it references and the degree to which those evidence fragments are relevant to the reasoning step when generating the text content for the reasoning step, using a preset format. The preset format could be, for example, annotating the generated text with phrases like "According to the provisions of [evidence identifier]..." or "Refer to [evidence identifier]...". The system extracts the referenced evidence identifiers from the language model's output through text parsing and calculates the dependency weight based on factors such as the confidence parameter attached to the language model when generating relevant content and the frequency of the evidence identifier's citation in the same reasoning step.

[0054] For example, if the same evidence identifier is cited multiple times in a reasoning step, it indicates that the evidence fragment is of high importance to the formation of the conclusion of the reasoning step, and its dependency weight is correspondingly high. If the language model has low confidence when generating content that cites a certain evidence fragment, the dependency weight of that evidence fragment will be reduced accordingly. If the language model does not output any evidence identifier when generating the content of the reasoning step, that is, it does not explicitly cite any evidence fragment, the system records the dependency relationship of the reasoning step as empty, indicating that the conclusion of the reasoning step is derived based on the model's internal knowledge, rather than based on specific external evidence.

[0055] Furthermore, in establishing the dependency relationship between the step identifier and the evidence identifier of the reasoning step, the system associates the step identifier of the reasoning step recorded in the aforementioned steps with the evidence identifier of the cited evidence fragment, forming a structured dependency relationship record. This dependency relationship record clearly indicates which reasoning step references which evidence fragment. If a reasoning step references multiple evidence fragments, the system establishes multiple dependency relationships between the step identifier of that reasoning step and the evidence identifier of each cited evidence fragment. If an evidence fragment is referenced by multiple reasoning steps, the system establishes multiple dependency relationships between the step identifier of each reasoning step and the evidence identifier of that evidence fragment. This many-to-many association mechanism enables the system to comprehensively and accurately depict the complete topological structure of evidence citation during the reasoning process. The dependency relationships established in this way constitute the basic connecting unit of the reverse tracing chain. Subsequent reverse tracing chain construction steps will use these dependency relationships as edges and output identifiers, step identifiers, and evidence identifiers as nodes to assemble a complete hierarchical mapping structure.

[0056] Based on the aforementioned publicly available solutions, by synchronously outputting the evidence identifiers cited and recording their dependency weights during the generation of reasoning steps, and establishing the dependency relationship between step identifiers and evidence identifiers, precise, real-time, and quantitative capture of the reasoning basis is achieved. This solution can accurately answer the questions of which piece of evidence was referenced in which reasoning step and to what extent, ensuring that each reasoning step in generating the conclusion has a clear and quantifiable source of evidence. This solves the problem that existing solutions cannot accurately establish the mapping relationship between reasoning steps and sources of evidence.

[0057] In one or more embodiments of this disclosure, such as Figure 5 This is a flowchart illustrating the dependency establishment method provided in an embodiment of this disclosure. Figure 5As shown, establishing the dependency relationship between the step identifier and the evidence identifier of the reasoning step includes: Step S501: Reading the internal state data of the language model when generating the reasoning step through the internal state reading interface of the language model. Step S502: Determining the cited evidence identifier based on the attention intensity of each evidence fragment in the internal state data. And, Step S503: Establishing the dependency relationship between the step identifier of the reasoning step and the cited evidence identifier.

[0058] It should be noted that the internal state data mentioned here refers to the numerical information generated by each layer of the internal neural network of the language model during the generation of inference steps, reflecting the model's processing of input information. Internal state data includes, but is not limited to: attention matrix, used to represent the degree of attention the model pays to each position in the input sequence when generating the output at the current position; gradient information, used to represent the sensitivity of the model's output relative to the input; and hidden state, used to represent the model's intermediate representation of the input. The intensity of attention mentioned here refers to the numerical intensity of attention to a specific piece of evidence within the internal state data. This indicator reflects the degree to which the model depends on that piece of evidence when generating the content of the current inference step. The intensity of attention can be determined through the attention weight values ​​at the corresponding positions in the attention matrix, the gradient magnitude corresponding to the input in the gradient information, or other quantified values ​​in the relevant internal state data.

[0059] During the process of reading the internal state data of the language model when generating a certain inference step through the internal state reading interface, the system synchronously calls the internal state reading interface provided by the language model when the language model begins to execute the content generation task of a certain inference step, requesting to obtain the internal state data during the generation process of that inference step. The system can combine the user-input question, the sub-question corresponding to the inference step, and multiple retrieved evidence fragments into the input sequence of the language model. As the language model gradually generates the content of the inference step, its internal neural network generates corresponding internal state values ​​for each position in the input sequence. The system reads this internal state data through the internal state reading interface. For example, it reads the attention matrix generated by each layer and each attention head in the multi-head attention mechanism of the language model. This attention matrix is ​​a matrix composed of the attention weight values ​​between any two positions in the input sequence. The system can also read the attention distribution corresponding to each output position in the generated content of the inference step to obtain the model's attention to the positions of each evidence fragment in the input sequence at the current output position.

[0060] Furthermore, the read internal state data is aggregated to extract the attention intensity values ​​corresponding to each evidence fragment. In specific implementation, the system can aggregate the attention weights corresponding to the positions of the evidence fragments in the input sequence based on their location intervals. This aggregation can be achieved by weighted averaging of multiple attention heads and multi-layered attention to obtain the overall attention intensity of each evidence fragment. For example, if the model assigns a high attention weight to the position corresponding to a certain evidence fragment in the input sequence when generating the key conclusive statement of the reasoning step, it indicates that the evidence fragment has a high attention intensity in this reasoning calculation. The system identifies at least one evidence fragment with the highest attention intensity as the evidence fragment cited in this reasoning step and obtains its corresponding evidence identifier. The system can set an attention intensity threshold, identifying only evidence fragments with attention intensity exceeding this threshold as cited evidence identifiers; if the attention intensity of all evidence fragments is below the threshold, it is determined that the reasoning step does not cite any specific evidence fragment, meaning its conclusion is derived based on the model's internal knowledge.

[0061] The system associates the cited evidence identifiers identified in the preceding steps with the step identifiers of the reasoning steps, forming a dependency record. If a reasoning step is determined to cite multiple evidence fragments, the system establishes multiple dependency relationships between the step identifier of the reasoning step and the evidence identifier of each cited evidence fragment. The dependency record can synchronously record the attention intensity corresponding to each cited evidence fragment, which can be directly used or normalized as the dependency weight of that evidence fragment on that reasoning step.

[0062] Based on the aforementioned publicly available solutions, by directly reading the internal state data of the language model during the inference generation step using its internal state reading interface, and determining citation relationships based on the intensity of attention given to evidence fragments, a non-intrusive and objective quantitative capture of dependencies is achieved. This method does not require the language model to adhere to a specific output format, does not alter the original presentation of the model's output content, and does not affect the naturalness or fluency of the generated text. Furthermore, by utilizing the model's finely detailed numerical attention information, it can capture implicit citations of evidence that are not explicitly marked in the text. This dependency determination method based on internal state data further ensures the authenticity and objectivity of the tracing results.

[0063] In one or more embodiments of this disclosure, such as Figure 6 This is a flowchart illustrating the reverse tracing chain construction method provided in this embodiment of the disclosure. Figure 6As shown, based on the dependencies, a reverse tracing chain is constructed, including the final answer, reasoning steps, and referenced evidence fragments. This includes: Step S601: Assigning an output identifier to the final answer. Step S602: Generating the reverse tracing chain using a hierarchical mapping structure with the output identifier as the root node, the step identifiers of each reasoning step as intermediate nodes, and the evidence identifiers of the referenced evidence fragments as leaf nodes.

[0064] It should be noted that the hierarchical mapping structure mentioned here refers to a set of nodes organized in a tree-like or graphical data structure. Nodes in this structure are connected according to a hierarchical relationship, with upper-level nodes establishing links to lower-level nodes through intermediate nodes. In an optional embodiment, the hierarchical mapping structure can be a three-layer structure: the root node layer corresponds to the output identifier of the final answer, the intermediate node layer corresponds to the step identifiers of each reasoning step, and the leaf node layer corresponds to the evidence identifiers of the cited evidence fragments.

[0065] In practical applications, after all reasoning steps are completed and the final answer is generated, a unique output identifier is assigned to the output of this generation task. This output identifier is the root node identifier of the reverse tracing chain and also the unified entry point for backtracking queries starting from the final answer. The output identifier is stored in association with metadata such as the text content of the final answer, the generation time, and a list of step identifiers for each reasoning step used in this task, so that subsequent queries and auditing operations can quickly locate the corresponding generation task instance through this output identifier.

[0066] Furthermore, based on the dependencies recorded in the preceding steps, a three-tiered back-mapping structure starting from the final answer is constructed. The dependencies recorded earlier are scattered, linked records stored as units of reasoning steps. Each dependency isolatedly indicates that a certain reasoning step references a certain piece of evidence, but does not form a complete path from the final answer to the evidence piece. Therefore, these scattered dependencies are further structured and integrated according to the hierarchical logic of output, reasoning steps, and evidence, forming one or more reachable paths from the root node to the leaf node, thereby achieving complete backtracking from the final answer.

[0067] In the specific construction process, the system first creates the output identifier as the root node. Then, the system obtains a list of step identifiers for all reasoning steps executed in this generation task, creates each step identifier as an intermediate node, and establishes a connection between the root node and each intermediate node. This connection indicates that the final answer is derived from the synthesis of these reasoning steps. Next, the system traverses each reasoning step, obtains the evidence identifiers for all evidence fragments referenced by that reasoning step based on the recorded dependencies, creates each evidence identifier as a leaf node, and establishes a connection between the intermediate node corresponding to that reasoning step and the leaf node corresponding to each referenced evidence identifier. This connection indicates that the conclusion of that reasoning step is based on that evidence fragment. Thus, the system constructs a hierarchical mapping structure containing three layers of nodes: output identifier, step identifier, and evidence identifier. This structure is the reverse tracing chain.

[0068] Based on the aforementioned publicly available solutions, by assigning output identifiers to the final answer and constructing a hierarchical mapping structure with output identifiers as root nodes, reasoning step identifiers as intermediate nodes, and evidence identifiers as leaf nodes, the originally scattered dependencies are organized into an output-oriented structured reverse tracing chain. Compared to existing tracing methods centered on inputs or internal model mechanisms, this solution adopts a structured tracing mechanism that starts from the output conclusion and traces back layer by layer according to the output, reasoning steps, and evidence direction. This structure not only ensures that each generated conclusion has a clear and structured source path, but also provides a unified data structure foundation for the implementation of subsequent auditing functions such as version binding and coverage calculation, solving the problem of the lack of an output-oriented structured explanation chain in existing technologies, which leads to difficulties in tracing and auditing.

[0069] In one or more embodiments of this disclosure, such as Figure 7 This is a flowchart illustrating a method for generating a minimal interpretation set according to an embodiment of this disclosure. Figure 7 As shown, after constructing the reverse tracing chain, the method further includes: Step S701: Aggregating the dependency weights of the same evidence identifier in at least one reasoning step to obtain the global importance of the evidence fragments to the final answer. Step S702: After sorting the evidence fragments according to their global importance, the global importance of each evidence fragment is accumulated. Step S703: When the proportion of the accumulated value to the total global importance of all evidence fragments exceeds a proportion threshold, the evidence fragments that have participated in the accumulation are constructed into a minimum interpretation set. Step S704: Generating an interpretation summary based on the minimum interpretation set.

[0070] It should be noted that the Minimal Explanation Set (MES) mentioned here refers to the core set of evidence selected from all cited evidence fragments that can fully explain the final answer with the fewest possible pieces of evidence. The evidence fragments constituting the Minimal Explanation Set are the key pieces of evidence that have the greatest impact on the final answer. Users can understand the main sources of the conclusion by referring to just a few pieces of evidence in the Minimal Explanation Set, without having to go through all the cited records.

[0071] The system iterates through all leaf nodes in the reverse tracing chain to obtain all dependency weight records associated with each evidence identifier. Since the same piece of evidence may be referenced in multiple different reasoning steps, and each step assigns a corresponding dependency weight, these distributed dependency weights reflect the local influence of the evidence piece on different reasoning stages. The system aggregates the dependency weights of the same evidence identifier across all reasoning steps to obtain the global importance of the evidence piece to the final answer. Aggregation calculation can employ methods such as summation, weighted summation, or weighted average. In the case of weighted calculation, the weights of different reasoning steps can be allocated based on their importance in the overall reasoning process or their execution order. For example, key reasoning steps closer to the final conclusion can be assigned higher weights, while auxiliary or preparatory reasoning steps can be assigned lower weights, thus giving higher global importance to evidence pieces frequently referenced in key reasoning steps. If an evidence piece is only referenced in one reasoning step and has a low dependency weight, its global importance is correspondingly low, indicating that the evidence piece's overall contribution to the final answer is small.

[0072] In the process of sorting evidence fragments by global importance and then summing them one by one in descending order of global importance to determine the minimum explanatory set, the system first calculates the sum of the global importance of all cited evidence fragments to obtain the total importance. The total importance serves as the benchmark value for judging the coverage of the minimum explanatory set. If only one evidence fragment is cited in this generation task, the total importance is the global importance of that evidence fragment. Then, the system sorts all cited evidence fragments in descending order of their global importance, with the evidence fragment with the highest global importance at the beginning and the evidence fragment with the lowest global importance at the end. If multiple evidence fragments have the same global importance, their order can be determined according to the lexicographical order of the evidence identifiers or other rules.

[0073] Next, starting from the first element of the sorted evidence sequence, the system adds the global importance of each evidence fragment to the current accumulated value. The first accumulation is the global importance of the first evidence fragment in the sequence; the second accumulation is the sum of the global importance of the first and second evidence fragments; and so on, with the current accumulated value increasing after each accumulation. After each accumulation, the system calculates the ratio of the current accumulated value to the total importance. This ratio reflects the coverage of the accumulated evidence fragments to the global importance of the final answer. The system compares this ratio to a preset ratio threshold. If the ratio is less than the preset ratio threshold, it indicates that the sum of the global importance of the accumulated evidence fragments has not yet reached a level sufficient to explain the final answer. The system continues to accumulate the global importance of the next evidence fragment in the sequence and calculates the ratio again for comparison. If the ratio reaches or exceeds the preset ratio threshold, it indicates that the accumulated evidence fragments are sufficient to explain the source of the final answer to a preset level of sufficiency. The system stops the accumulation operation and no longer selects the remaining evidence fragments in the sequence. The preset ratio threshold can be configured according to the application scenario's requirements for sufficiency of explanation; for example, it can be set to 85%. If the ratio still does not reach the preset ratio threshold after all evidence fragments have been accumulated, the system will include all evidence fragments in the minimum explanation set, indicating that the basis for the final answer is relatively scattered and requires more evidence to be fully explained.

[0074] The set of all evidence fragments that participated in the accumulation before the accumulation stops is determined as the minimum explanatory set for this generation task. The evidence fragments in this minimum explanatory set are the basis for influencing the final answer, and their sum of global importance covers a proportion above a preset threshold of total importance, enabling a sufficient explanation of the main sources of the final answer with a relatively small amount of evidence. After determining the minimum explanatory set, the system can further generate an explanatory summary based on this minimum explanatory set.

[0075] Furthermore, taking the content of each evidence fragment in the minimal explanatory set and its corresponding evidence identifier as input, a language model or text summarization module is invoked to semantically summarize and integrate the content of these core evidences, generating a natural language explanatory summary. The explanatory summary concisely explains which evidence fragments and their key content are primarily based on the final answer, allowing users to quickly understand the core basis of the conclusion without having to review each original evidence fragment individually. The generation of the explanatory summary can include: splicing together key information from each evidence fragment based on a preset template; or generating one or more fluent inductive statements after understanding the content of each evidence fragment in the minimal explanatory set through a language model.

[0076] Based on the publicly available solutions described above, by automatically aggregating the dependency weights of each piece of evidence across multiple reasoning steps after constructing the reverse tracing chain to obtain global importance, and then dynamically determining the minimum explanatory set by accumulating the global importance percentages to reach a preset threshold after sorting by importance, an explanatory summary is generated. This achieves the ability to automatically extract the most core evidence from a large amount of cited evidence. The size of the minimum explanatory set can be automatically adjusted according to the actual distribution of evidence citations in different generation tasks. When evidence citations are relatively concentrated and a few core pieces of evidence are sufficient to fully explain the conclusion, the size of the minimum explanatory set automatically shrinks; when evidence citations are relatively dispersed and more evidence is needed to fully explain the conclusion, the size of the minimum explanatory set automatically expands. This dynamic determination mechanism ensures that the minimum explanatory set always maintains the most concise size while ensuring sufficient explanation, avoiding the problems of missing core evidence or introducing redundant evidence that may occur with a fixed N value.

[0077] In some alternative approaches, after constructing the reverse tracing chain, the method further includes: aggregating the dependency weights of the same evidence identifier in at least one reasoning step to obtain the global importance of that evidence fragment to the final answer. The evidence fragments are then sorted by global importance, and the top N evidence fragments with the highest importance are selected to form a minimal explanatory set; where N is a positive integer. An explanatory summary is generated based on the minimal explanatory set.

[0078] N can be determined by a preset fixed threshold. The evidence fragments constituting the minimum explanatory set are the core basis supporting the final answer; users can quickly judge the reasonableness of the conclusion by reviewing these few pieces of evidence. Although evidence fragments not selected for the minimum explanatory set are also cited in the reasoning process, their impact on the final conclusion is relatively low, serving as auxiliary references. If the total number of cited evidence fragments is less than or equal to N, then all cited evidence fragments are included in the minimum explanatory set.

[0079] Based on the publicly available solutions described above, by automatically aggregating the dependency weights of each piece of evidence across multiple reasoning steps after constructing the reverse tracing chain to obtain global importance, and then selecting core evidence according to importance to form a minimal set of interpretations and generate an interpretation summary, the most crucial evidence can be extracted from a large amount of cited evidence. This solution can distinguish between key evidence and auxiliary references, and presents the most crucial, limited amount of evidence to the user in a concise interpretation summary format, significantly improving the readability and audit efficiency of AI-generated conclusions. This enables auditors in high-credibility business scenarios to verify the basis of generated conclusions in the shortest possible time.

[0080] In one or more embodiments of this disclosure, such as Figure 8 This is a flowchart illustrating the dependency weight determination method provided in an embodiment of this disclosure. Figure 8As shown, the method for determining dependency weights includes: Step S801: Reading the internal state data of the language model during the inference generation step through the internal state reading interface of the language model. Step S802: Determining the dependency weights based on the attention intensity of each evidence fragment in the internal state data. Alternatively, Step S803: Parsing the citation markers in the language model output, and combining the confidence level, citation frequency, and text matching degree of the language model output to determine the dependency weights.

[0081] In practical applications, in the first dependency weight determination method, the system reads the internal state data of the language model during the inference generation step through the language model's internal state reading interface, and determines the dependency weights based on the attention intensity of each evidence fragment in the internal state data. The internal state data includes intermediate numerical information generated by the model during inference computation, such as the attention matrix, gradient information, or hidden states. The attention matrix represents the distribution of attention weights for each position in the input sequence when the model generates the output at the current position; the gradient information reflects the sensitivity of the model's output relative to each part of the input; and the hidden states represent the model's intermediate semantic representation of the input.

[0082] In its implementation, the system acquires the internal state data corresponding to a certain reasoning step in real time through an internal state reading interface during the content generation process of a language model executing a certain reasoning step. Based on the position interval occupied by the evidence fragment in the input sequence, the system extracts the value corresponding to that position interval from the internal state data. Taking the attention matrix as an example, the system obtains the attention weight distribution generated by each layer and attention head of the language model, and aggregates the attention weights corresponding to the position interval of a certain evidence fragment in the input sequence. Aggregation processing may include averaging the attention weights of multiple layers and multiple heads, or specifically extracting the attention weights at the corresponding positions when generating key content for the reasoning step. The comprehensive value obtained after aggregation reflects the intensity of attention received by the evidence fragment in that reasoning step. The system directly uses the intensity of attention as the dependency weight of the evidence fragment for that reasoning step, or normalizes the intensity of attention to obtain the dependency weight. Normalization makes the dependency weights between different reasoning steps comparable; for example, the sum of the intensity of attention of all cited evidence fragments is normalized to one, and the dependency weight of each evidence fragment is the proportion of its intensity of attention to the total intensity of attention. The dependency weights determined in the above manner reflect the objective degree of dependence of the language model on each piece of evidence when performing this reasoning step.

[0083] In the second method of determining dependency weights, the system parses the citation markers in the language model output and determines the dependency weights by combining the confidence level, citation frequency, and text matching degree of the language model output. This method is suitable for scenarios where the language model outputs citation markers in a preset format when generating content for the inference steps. Citation markers refer to the specific formatted annotations used by the language model in the generated text to indicate the cited evidence identifiers. Confidence level refers to the certainty score of the generated content with citation markers, provided by the language model's internal calculation mechanism. Citation frequency refers to the number of times the same evidence identifier is cited in the generated content of the same inference step. Text matching degree refers to the semantic similarity between the citation context generated by the language model and the actual content of the cited evidence fragment.

[0084] In its implementation, the system first parses the text of the reasoning steps generated by the language model, identifying and extracting citation markers to obtain the cited evidence identifiers. Simultaneously, the system obtains the confidence score of the language model when generating content containing each citation marker. The system then calculates the citation frequency of the same evidence identifier within the reasoning step content. Furthermore, the system calculates the text matching degree between the contextual text surrounding the citation marker in the generated content and the actual content of the evidence fragment, verifying whether the evidence fragment pointed to by the citation marker is consistent with the information actually used by the model. Text matching degree can be calculated using methods such as semantic vector similarity or keyword matching. The system then combines the three factors—confidence, citation frequency, and text matching degree—to determine the dependency weight of the evidence fragment on the reasoning step. Strategies for comprehensive calculation include: weighted summation of the values ​​of the three factors; or using confidence as the base value and multiplicatively weighting with citation frequency and text matching degree as adjustment coefficients.

[0085] Both dependency weighting methods quantify the influence of each piece of evidence on each reasoning step using numerical values. The system associates and stores the determined dependency weights with the corresponding reasoning step identifiers and evidence identifiers, serving as the basis for global importance aggregation calculations. The two methods can be used individually or deployed simultaneously within the system, with the appropriate weighting method selected based on the actual capabilities of the language model.

[0086] Based on the aforementioned publicly available solutions, by providing two methods for determining dependency weights—one based on the intensity of attention to internal state data and the other based on citation tag parsing—the calculation of dependency weights can be adapted to different types and levels of openness of language models. For language models with open internal state reading interfaces, their fine-grained internal attention values ​​can be directly used to objectively reflect the model's dependence on each piece of evidence. For language models that do not open their internal state but can follow output format instructions, dependency weights can be determined by parsing citation tags in their output and combining confidence, citation frequency, and text matching degree. Both methods can provide quantitative basic data for subsequent evidence contribution analysis and minimal explanatory set extraction, making the extraction of the minimal explanatory set and the generation of explanatory summaries have quantifiable basis.

[0087] In one or more embodiments of this disclosure, the evidence fragments are sorted according to their global importance, and the top N evidence fragments with the highest importance are selected to form the minimum interpretation set. This includes: calculating the sum of the global importance of multiple evidence fragments to obtain the total importance; sorting the multiple evidence fragments in descending order of global importance; accumulating the global importance of multiple evidence fragments one by one to obtain the current accumulated value; calculating the ratio of the current accumulated value to the total importance; stopping the accumulation when the ratio reaches a preset ratio threshold; and determining the set of evidence fragments that have participated in the accumulation as the minimum interpretation set.

[0088] Obtain the global importance of the evidence fragments corresponding to all leaf nodes in the reverse tracing chain of this generation task. Sum all global importance values ​​to obtain the total importance. The total importance serves as the benchmark value for subsequently judging the coverage of the minimum interpretation set. If only one evidence fragment is cited in this generation task, the total importance is the global importance of that evidence fragment.

[0089] In the step of sorting multiple evidence fragments according to their global importance from highest to lowest, the system arranges all cited evidence fragments in descending order of their global importance, with the evidence fragment with the highest global importance placed first and the evidence fragment with the lowest global importance placed last. If multiple evidence fragments have the same global importance, their order can be determined according to the lexicographical order of the evidence identifiers or other rules.

[0090] In the process of accumulating the global importance of multiple evidence fragments to obtain the current accumulated value, the system starts from the first position of the sorted evidence fragment sequence and adds the global importance of each evidence fragment to the current accumulated value. During the first accumulation, the current accumulated value is the global importance of the first evidence fragment in the sequence; during the second accumulation, the current accumulated value is the sum of the global importance of the first and second evidence fragments; and so on, with the current accumulated value increasing after each accumulation.

[0091] After each accumulation, the ratio of the current accumulated value to the total importance is calculated. This ratio reflects the proportion of the evidence fragments that have been accumulated that cover the global importance of the final answer.

[0092] After each ratio calculation, the ratio is compared with a preset ratio threshold. If the ratio is less than the preset ratio threshold, it indicates that the sum of the global importance of the evidence fragments already included in the accumulation has not yet reached a level sufficient to fully explain the final answer. The system continues to accumulate the global importance of the next evidence fragment in the sequence and calculates the ratio again for comparison. If the ratio reaches or exceeds the preset ratio threshold, it indicates that the evidence fragments already included in the accumulation are sufficient to explain the source of the final answer to a preset degree of sufficiency. The system stops the accumulation operation and no longer selects the remaining evidence fragments in the sequence. If the ratio still does not reach the preset ratio threshold after all evidence fragments have been included in the accumulation, the system includes all evidence fragments in the minimum explanation set, indicating that the source of the final answer is relatively dispersed and requires more evidence to fully explain it.

[0093] The set of all evidence fragments that have been accumulated when accumulation stops is determined as the minimum explanatory set for this generation task. The evidence fragments in this minimum explanatory set are the core evidence that has the greatest impact on the final answer; their sum of global importance covers a proportion above a preset threshold of total importance, enabling a sufficient explanation of the main sources of the final answer with a relatively small amount of evidence. After determining the minimum explanatory set, the system can further generate an explanatory summary based on this minimum explanatory set.

[0094] Based on the aforementioned publicly available scheme, the minimum explanatory set size is dynamically and adaptively determined by calculating the total importance, sorting by global importance, and accumulating the sums until the accumulation ratio reaches a preset threshold. Compared to the method of fixedly selecting the first N evidence fragments, this scheme can automatically adjust the size of the minimum explanatory set according to the actual distribution of evidence citations in different generation tasks. When evidence citations are relatively concentrated and a few core pieces of evidence are sufficient to explain the conclusion, the minimum explanatory set size automatically shrinks; when evidence citations are relatively dispersed and more evidence is needed to fully explain the conclusion, the minimum explanatory set size automatically expands. This dynamic determination mechanism ensures that the minimum explanatory set always maintains the most concise size while ensuring sufficient explanation, avoiding the problems of missing core evidence or introducing redundant evidence that may be caused by a fixed N value.

[0095] In one or more embodiments of this disclosure, when constructing the reverse tracing chain, the method further includes: obtaining environmental context information when generating the final answer; wherein the environmental context information includes at least one of the following: version information of the language model used to generate the final answer, version information of the knowledge base used to obtain the evidence fragments, and the timestamp of generating the final answer; and storing the environmental context information as a version record in association with the output identifier.

[0096] During the construction of the reverse tracing chain, system environment snapshot information for this generation task is collected synchronously. The system can obtain the language model version information, including the model identifier and version number of the language model currently invoked to perform the inference step. This information can be read from the language model's service interface metadata or obtained from the system configuration file. The system can obtain the knowledge base version information, including the current release version number or content snapshot identifier of the knowledge base used to retrieve evidence fragments. This information can be obtained from the knowledge base management system's version interface. The system can obtain timestamp information by reading the system clock; the timestamp marks the moment the final answer is generated. The system can obtain at least one of the above environmental context information as content for subsequent associated storage. In specific implementations, if the language model service is an external service called through an application programming interface (API), the system can obtain model version information through the metadata endpoint provided by that API; if the knowledge base adopts a versioned management mechanism, the system can query the current version identifier of the knowledge base; the timestamp can be obtained by calling the system logs.

[0097] The acquired environmental context information is combined into a version record. This version record is then associated with the output identifier assigned to the final answer and persistently stored. The associated storage can be implemented in several ways: the version record is written as an additional field into the root node data structure of the reverse tracing chain with the output identifier as the root node; an independent mapping table from the output identifier to the version record is established, allowing the corresponding version record to be retrieved via the output identifier; or the output identifier and version record are stored together in the same row of a relational database. After the association between the version record and the output identifier is established, when a user queries the final answer and its reverse tracing chain of a historical generation task via the output identifier, the environmental context information of the current generation task can be obtained simultaneously, thus understanding the system environment state on which the final answer was generated.

[0098] If, when retrieving environment context information, a certain piece of information is temporarily unavailable due to system configuration reasons (e.g., the language model service does not provide an external version information interface), the system can record that information as null or use a substitute identifier, and continue to associate and store the other retrieved environment context information. The absence of information in the version record does not affect the main structure and use of the reverse tracing chain, but a complete version record provides stronger reproducibility and auditability assurance.

[0099] Based on the publicly available solutions described above, by acquiring environmental context information while constructing the reverse tracing chain and storing this information as a version record associated with the output identifier, the output of each generation task is traceable not only to its dependencies on reasoning steps and evidence fragments but also to its system environment state at the time of generation. By binding environmental context information such as language model version, knowledge base version, and timestamps, even after iterative updates to the system model or knowledge base, it is still possible to trace back to the specific model version and knowledge base state used when generating a certain conclusion through historical version records, thereby achieving cross-version reproducibility and auditability of the generated results.

[0100] In one or more embodiments of this disclosure, after associating and storing the version record with the output identifier, the method further includes: traversing the reverse tracing chain, counting the number of step identifiers associated with evidence identifiers in the reverse tracing chain as the number of inference steps supported by evidence; obtaining the total number of step identifiers contained in the reverse tracing chain; and calculating the ratio of the number of inference steps supported by evidence to the total number of step identifiers to obtain the explanation coverage index.

[0101] Using the constructed reverse tracing chain as the traversal object, starting from the root node, the system traverses all intermediate nodes, i.e., it traverses the step identifiers of all reasoning steps. For each intermediate node, the system checks whether it has at least one associated leaf node, i.e., whether the reasoning step establishes a dependency relationship with at least one evidence identifier. If the intermediate node of the reasoning step has at least one leaf node, it means that the reasoning step references at least one piece of evidence, indicating that the conclusion of the reasoning step has a clear source of external evidence, and the system counts this reasoning step in the number of reasoning steps supported by evidence. If the intermediate node of the reasoning step has no leaf nodes, it means that the reasoning step does not reference any piece of evidence, indicating that the conclusion of the reasoning step is derived based on the internal knowledge of the language model and does not have clear external evidence, and the system does not count this reasoning step in the number of reasoning steps supported by evidence. After completing the traversal of all intermediate nodes, the system obtains the statistical result of the number of reasoning steps supported by evidence. If all reasoning steps in the reverse tracing chain have associated evidence identifiers, then the number of reasoning steps supported by evidence equals the total number of step identifiers; if none of the reasoning steps have associated evidence identifiers, then the number of reasoning steps supported by evidence is zero.

[0102] Further, count the total number of intermediate nodes in the reverse tracing chain, which is the total number of step identifiers for all inference steps executed in this generation task. The total number of step identifiers can be obtained from the list of step identifiers recorded during the inference step creation phase, or by directly traversing the intermediate node layer of the reverse tracing chain to count them.

[0103] The explanation coverage metric is obtained by dividing the number of reasoning steps supported by evidence by the total number of step identifiers. The explanation coverage metric is calculated as follows: Explanation Coverage Metric = Number of Reasoning Steps Supported by Evidence / Total Number of Step Identifiers. The system can associate and store the explanation coverage metric with the output identifiers, displaying the metric synchronously when presenting the final answer to the user, or using it as a reference in the audit report to assess the credibility of the generated results.

[0104] Based on the aforementioned publicly available scheme, by traversing the reverse traceback chain to count the number of reasoning steps supported by evidence and comparing this count with the total number of step identifiers to calculate the explanation coverage metric, a quantitative assessment of the degree to which the final answer is supported by evidence fragments is achieved. Using the structured data from the reverse traceback chain as a foundation, the explanation coverage metric provides comparable and quantifiable credibility assessment data for each generation task. The explanation coverage metric not only serves as an intuitive reference for users to judge the reliability of the generated results, but also...

[0105] Based on any of the above embodiments, this disclosure also provides a traceable information chain output device. Figure 9 This is a schematic block diagram of a traceable information chain output device according to one embodiment of the present disclosure. Figure 9 As shown, the traceable information chain output device includes: an acquisition module 901 for acquiring a question input by a user; a determination module 902 for determining multiple pieces of evidence related to the question; an execution module 903 for performing a reasoning process, including at least one reasoning step, based on the question and the evidence pieces to generate a final answer; a recording module 904 for recording the dependency relationship between each reasoning step and the cited evidence piece during the reasoning process; and a construction module 905 for constructing a reverse traceability chain, starting from the final answer, containing the final answer, the reasoning steps, and the cited evidence pieces, based on the dependency relationships.

[0106] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0107] The entity executing the information sending method in the specific embodiments of this disclosure may be an electronic device such as a server (including a local server or a cloud computing platform).

[0108] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the traceable information chain output method of any of the embodiments described above.

[0109] Figure 10 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure.

[0110] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0111] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0112] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0113] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0114] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0115] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0121] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for outputting a traceable information chain, characterized in that, The method includes: The problem of obtaining user input; Identify multiple pieces of evidence relevant to the issue; Based on the question and the evidence fragments, a reasoning process including at least one reasoning step is performed to generate a final answer; During the reasoning process, the dependency relationship between each reasoning step and the cited evidence fragment is recorded; Based on the dependencies, a reverse tracing chain is constructed starting from the final answer, including the final answer, the reasoning steps, and the cited evidence fragments.

2. The traceable information chain output method according to claim 1, characterized in that, The determination of multiple pieces of evidence related to the problem includes: Retrieve multiple candidate contents related to the semantics of the question from the knowledge base; The candidate content is segmented into at least one evidence fragment; and, Assign a unique evidence identifier to each of the aforementioned evidence fragments.

3. The traceable information chain output method according to claim 2, characterized in that, The reasoning process based on the question and the evidence fragment, including at least one reasoning step, includes: The problem is subjected to intent recognition and task decomposition to generate at least one sub-problem; Generate corresponding reasoning steps for each subproblem; and, Assign a step identifier to each of the aforementioned reasoning steps.

4. The traceable information chain output method according to claim 3, characterized in that, The record of the dependency relationship between each reasoning step and the cited evidence fragment includes: Output the evidence identifier corresponding to the cited evidence fragment, and record the dependency weight of the cited evidence fragment on the reasoning step; Establish the dependency relationship between the step identifier and the evidence identifier of the reasoning step.

5. The traceable information chain output method according to claim 4, characterized in that, The establishment of the dependency relationship between the step identifier and the evidence identifier in the reasoning step includes: The internal state data of the language model during the generation of this inference step is read through the language model's internal state reading interface. Based on the intensity of attention given to each piece of evidence in the internal state data, the evidence identifier of the cited evidence piece is determined; and, Establish the dependency relationship between the step identifier of the reasoning step and the evidence identifier.

6. The traceable information chain output method according to claim 1, characterized in that, The step of constructing a reverse tracing chain based on the dependency relationship, including the final answer, the reasoning steps, and the cited evidence fragments, includes: Assign an output identifier to the final answer; The reverse tracing chain is generated using a hierarchical mapping structure with the output identifier as the root node, the step identifiers of each reasoning step as intermediate nodes, and the evidence identifiers of the referenced evidence fragments as leaf nodes.

7. The traceable information chain output method according to claim 4, characterized in that, After constructing the reverse tracing chain, the following is also included: By aggregating the dependency weights of the same evidence identifier in the at least one reasoning step, the global importance of the evidence fragment to the final answer is obtained; After sorting the evidence fragments according to the global importance, the global importance of each evidence fragment is accumulated. When the proportion of the accumulated value to the total global importance of all evidence fragments exceeds the proportion threshold, the evidence fragments that have participated in the accumulation will be constructed into a minimum interpretation set; An explanation summary is generated based on the minimum explanation set.

8. The traceable information chain output method according to claim 7, characterized in that, The methods for determining the dependency weights include: The internal state data of the language model during the generation of the inference step is read through the internal state reading interface of the language model. The dependency weight is determined based on the intensity of attention paid to each of the evidence fragments in the internal state data; or, The citation markers in the language model output are parsed, and the dependency weights are determined by combining the confidence level, citation frequency, and text matching degree of the language model output.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as, A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images