Multi-modal inference method and inference system based on error attribution

Abstract

The invention belongs to the technical field of thinking chain reasoning, and particularly relates to a multi-modal reasoning method and system based on error attribution. The reasoning method comprises the following steps: on the basis of a current modal fusion weight, performing modal fusion on each piece of initial information in an initial information set, and then generating a thinking chain; after a reasoning dependency graph is constructed based on the thinking chain, check points are selected in the reasoning dependency graph; based on consistency, factuality and logicality, performing error possibility scoring on each check point, and if the error possibility scores of all check points in the current thinking chain are below a set threshold, outputting the current thinking chain; otherwise, marking the check points of which the error possibility scores exceed a set threshold value as error nodes; calculating relative contribution strength of different modes to error nodes; and on the basis of the relative contribution strength, updating the modal fusion weight, and regenerating the thinking chain. According to the invention, the accuracy of the reasoning result and the stability of the accuracy can be improved.

Description

Technical Field

[0001] This invention belongs to the field of thought chain reasoning technology, and particularly relates to a multimodal reasoning method and reasoning system based on error attribution. Background Technology

[0002] Thought chain reasoning (CoT) technology has made significant progress in the field of artificial intelligence in recent years, evolving from early single-step reasoning to complex step-by-step reasoning frameworks based on large language models (LLMs). Modern thought chain technology simulates the human problem-solving process by decomposing complex problems into a series of reasoning steps, effectively improving the model's performance on tasks such as mathematical reasoning, logical judgment, and common-sense reasoning.

[0003] The degree of involvement of different modal information, which forms the basis of reasoning, directly affects the accuracy of the reasoning result. For example, an inappropriate degree of involvement of different modal information can directly lead to an intrinsic error in a certain reasoning step (an intrinsic error refers to an error whose cause is unrelated to the correctness of the preceding steps as input, but only related to the calculation of the reasoning step itself). Furthermore, a reasoning step with an intrinsic error will accumulate errors in subsequent reasoning steps. For instance, in a mathematical problem-solving scenario, a small calculation error (such as an intrinsic error in a reasoning step due to an inappropriate degree of involvement of modal information) can cause the entire problem-solving process to deviate from the correct path; and as the number of reasoning steps increases, errors in the preceding reasoning steps will accumulate, and the interaction of these factors significantly reduces the accuracy of the reasoning result.

[0004] Even with the introduction of verification mechanisms, only the correctness of the reasoning result can be verified. The intermediate reasoning steps that lead to errors remain unknown, which naturally reduces the efficiency of correcting the reasoning result, resulting in unstable accuracy and low efficiency in correcting reasoning errors. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal reasoning method based on error attribution, which can improve the accuracy and stability of the reasoning results.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal reasoning method based on misattribution includes the following steps: S1, based on the current modal fusion weights, modal fusion is performed on each piece of initial information in the initial information set to generate a thought chain; S2, after constructing the inference dependency graph based on the thought chain, select checkpoints in the inference dependency graph; S3: Based on consistency, factuality, and logic, score the error probability of each checkpoint. If the error probability scores of all checkpoints in the current thought chain are below the set threshold, output the current thought chain; otherwise, mark the checkpoints with error probability scores exceeding the set threshold as error nodes and execute S4. S4, calculate the relative contribution intensity of different modes to the faulty node; S5, based on the relative contribution strength, after updating the modal fusion weights, return to S1 and use the updated modal fusion weights to generate a new thought chain.

[0007] Preferably, step S1 further includes the following sub-steps: S11, Extract the feature matrix of each piece of initial information in the initial information set; S12, map all feature matrices to the same dimensional space for dimensional alignment; S13, based on the bidirectional cross-attention mechanism and modality fusion weight, the feature matrix after dimension alignment is modally fused to obtain the modality fusion matrix; S14, based on the modality fusion matrix, uses an inference decoder to generate a thought chain containing a sequence of inference steps.

[0008] Preferably, step S11 further includes the following: extracting initial information of the text type. semantic feature matrix Extract initial information of image type Image feature matrix ; S12 also includes the following: image feature matrix Mapped into the visual feature matrix in the text feature space ,in = ; It is a learnable linear projection weight matrix; To extract semantic feature matrix The model bias vector; S13 also includes the following: fusion weight vector based on the current visual modality Weight vector fused with text modality A bidirectional cross-attention mechanism is used to generate a modality fusion matrix. : ; ; ; ; ; in, This represents the calculation result of the scaled dot product attention mechanism; Indicates use The function performs the calculation; , and These represent the text query matrix, text key matrix, and text value matrix, respectively. , and These represent the visual query matrix, visual key matrix, and visual value matrix, respectively. , and These are the first projection matrix, the second projection matrix, and the third projection matrix, respectively. This is a learnable projection matrix that, after training, has the optimal query mapping relationship. This is a matrix whose learnable projection matrix has the optimal key mapping relationship after training. It is a matrix that has the optimal value mapping relationship after the learnable projection matrix is ​​trained; express Transpose of; express Transpose of; The dimension representing the attention head, ,in Represents the extraction of semantic feature matrix The feature dimension of the model is h, which represents the number of attention heads.

[0009] Preferably, S14 also includes the following: based on the modality fusion matrix Using a pre-trained multimodal large model as an inference decoder, a thought chain containing a sequence of inference steps is generated: ; ; in, This represents the i-th reasoning step in the current thought chain, where 1 ≤ i ≤ n and i and n are both positive integers. This indicates a structured suggestion template; This represents the sequence of preceding inference steps that have already been generated; LLM ( This indicates the use of an LLM multimodal large model, based on the modality fusion matrix. and the sequence of preceding reasoning steps To generate the current inference step When i=1, LLM multimodal large model Content( () indicates the reasoning steps The reasoning content; CL( () indicates the reasoning steps Confidence score; DL ( () indicates the reasoning steps A list of logical dependencies.

[0010] Preferably, step S2 further includes the following sub-steps: S21, Constructing a reasoning dependency graph based on the thought chain. : Where N represents the set of nodes, and each node corresponds to a reasoning step in the current thought chain; This represents the set of edges, which are used to connect nodes in the logical dependency list that represent adjacent reasoning steps. S22, inference dependency graph Nodes that meet one or more of the following conditions are selected as checkpoints: Condition 1, confidence score is lower than the first score threshold; Condition 2, out-degree is greater than the second score threshold; Condition 3, in-degree is greater than the third score threshold.

[0011] Preferably, step S3 further includes the following sub-steps: S31, the average semantic consistency between the reasoning content corresponding to the current checkpoint and the reasoning content in the preceding reasoning step sequence on which it depends is recorded as the consistency score of the current checkpoint; the degree of conformity between the reasoning content corresponding to the current checkpoint and the facts is recorded as the factual score of the current checkpoint; the logical rationality between the reasoning content corresponding to the current checkpoint and the reasoning content corresponding to all nodes in the current thought chain is recorded as the logical score of the current checkpoint. S32, record the current checkpoint as... checkpoint The corresponding reasoning steps are as follows: Then calculate the current checkpoint. Error probability rating : ; in, , and These are the first weight parameter, the second weight parameter, and the third weight parameter, respectively. and ;; , and These are the current checkpoints. Consistency score, factual score, and logical score; When the probability of error is rated When the score exceeds the fourth threshold, the corresponding checkpoint is marked as an error node; after marking all error nodes in the current thought chain, execute S4; if there are no error nodes in the current thought chain, output the current thought chain.

[0012] Preferably, step S4 further includes the following sub-steps: S41, Computational Visual Modality Fusion Matrix Text modality fusion matrix : ; ; ; ; in, This represents the calculation result of the scaled dot product attention mechanism; Indicates use The function performs the calculation; , and These represent the text query matrix, text key matrix, and text value matrix, respectively. , and These represent the visual query matrix, visual key matrix, and visual value matrix, respectively. express Transpose of; express Transpose of; The dimension representing the attention head, ,in Represents the extraction of semantic feature matrix The feature dimension of the model, h represents the number of attention heads; S42, based on the visual modality fusion matrix Text modality fusion matrix Using a pre-trained multimodal large model as an inference decoder, the corresponding visual thought chain Chain´ and text thought chain Chain´´ are generated. S43, select the earliest erroneous node in the current thought chain, and denote this erroneous node as... ; Retrieve the corresponding nodes in the visual thinking chain Chain´ and the text thinking chain Chain´´ respectively. Error probability rating and ´´; S44, calculate the visual modality and text modality for the erroneous nodes in the current thought chain, respectively. absolute contribution intensity and : ; ; S45, calculate the visual modality and text modality for the erroneous nodes in the current thought chain, respectively. relative contribution intensity and : ; .

[0013] Preferably, step S5 further includes the following sub-steps: S51, Error Probability Scoring Calculate the weight adjustment coefficient β: Where λ is the sensitivity parameter; This is the upper limit of the weight adjustment coefficient; min( , () represents the function that takes the minimum value; S52, Update the modal fusion weights based on the current weight adjustment coefficient β: ; ; in, and These represent the updated text modality fusion weight vector and the visual modality fusion weight vector, respectively.

[0014] Preferably, replace S52 with S52': S52´, based on the current weight adjustment coefficient β, update the modality fusion weights: ; ; in, and These represent the updated text modality fusion weight vector and the visual modality fusion weight vector, respectively. This represents the 2-norm.

[0015] This invention also provides a reasoning system, comprising: a thought chain generation module, a checkpoint selection module, a scoring module, a calculation module, and an update module. The thought chain generation module generates a thought chain by performing modal fusion on each initial piece of information in the initial information set based on the current modal fusion weights, and then sends the generated thought chain to the checkpoint selection module. The checkpoint selection module constructs a reasoning dependency graph based on the thought chain, selects checkpoints in the graph, and then sends the dependency graph and checkpoints to the scoring module. The scoring module scores the error probability of each checkpoint. If the error probability scores of all checkpoints in the current thought chain are below a set threshold, the scoring module outputs the current thought chain; otherwise, the scoring module marks checkpoints with error probability scores exceeding the set threshold as error nodes and sends them to the calculation module. The calculation module calculates the relative contribution strength of different modalities to the error nodes and sends it to the update module. The update module updates the modal fusion weights based on the relative contribution strengths and then sends the updated weights to the thought chain generation module. Each module is programmed or configured to execute the steps of the multimodal reasoning method based on error attribution as described above.

[0016] The beneficial effects of this invention are as follows: (1) The reasoning method of the present invention first uses initial modal fusion weights to obtain an initial thought chain; then it performs error checking on this thought chain. If there are no error checking nodes in the current thought chain, the current thought chain is directly output; otherwise, the modal fusion weights are adjusted using the current thought chain, and a new thought chain is generated with the new modal fusion weights. That is, the reasoning method of the present invention can obtain an accurate thought chain.

[0017] (2) This invention does not use the obtained thought chain as the final output. Instead, it performs error checking on the obtained thought chain (i.e., scores the probability of errors at the checkpoints in the thought chain). Only thought chains without error nodes are output, which ensures that the thought chain output by this invention has a high accuracy rate. When error checking finds thought chains with error nodes, it calculates the relative contribution strength of different modalities to the error nodes in the thought chain based on the current thought chain, adjusts the modality fusion weights, and then uses the new thought chain generated by the new modality fusion weights to further reduce the number of error nodes and the degree of error. In other words, this invention can automatically and step by step correct the erroneous reasoning steps in the initial thought chain to the correct path through a closed loop. That is, the reasoning method of this invention can not only improve the accuracy of the reasoning results, but also maintain the stability of accurate reasoning results.

[0018] (3) This invention uses the presence or absence of erroneous nodes in the thought chain as direct feedback; and based on the relative contribution strength of erroneous nodes in the thought chain, the entire process can be completed without the intervention of technical personnel, thereby improving the accuracy of the final reasoning result while significantly saving manpower costs.

[0019] (4) In the reasoning method of the present invention, when calculating the relative contribution strength of different modalities to the error node, the "error node" we select is the error node that is at the beginning of the current thought chain. The reasons are as follows: ① While ensuring that the number of faulty nodes in the new thinking chain is reduced and the degree of error is mitigated, we can reduce the amount of computation when adjusting the modality fusion weights (there is no need to calculate the relative contribution strength of different modalities to all faulty nodes in the current thinking chain).

[0020] ② The causes of erroneous nodes in the thought chain are mutually interfering. Some erroneous nodes are not only due to the accumulation of errors from preceding reasoning steps, but also due to inappropriate modality fusion weights. However, in a thought chain, the first erroneous node usually appears because of inappropriate modality fusion weights. Therefore, this invention makes the error correction process more scientific by selecting appropriate erroneous nodes to adjust the modality fusion weights. This scientific nature is also reflected in the ability to directly identify erroneous nodes in the current thought chain. The accumulated errors are stripped away from the new thought chain, ensuring that the cause of the first error node in the newly generated thought chain is still an inappropriate modal fusion weight.

[0021] Therefore, the reasoning method of this invention, while conforming to the understanding that erroneous nodes in the reasoning chain cannot be completely corrected at once due to mutual interference, can obtain a highly accurate thought chain through several corrections, thus improving the efficiency of reasoning error correction. This invention corrects errors in the thought chain based on error attribution; where "error attribution" refers to the selected erroneous nodes used to adjust the modality fusion weights, and each generation of a new thought chain involves error correction.

[0022] (5) For the initial thought chain, as long as the new modality fusion weight value falls within a relatively wide and reasonable range, the error nodes in the new thought chain can be reduced, and the new modality fusion weight we calculated is within this reasonable range. However, in the subsequent adjustment of the modality fusion weight, this reasonable range continues to shrink, so the modality fusion weight calculated by this invention is more refined than the previous modality fusion weight.

[0023] (6) In the reasoning method of the present invention, not all nodes are directly used as checkpoints, but checkpoints are selected based on three conditions. This not only reduces the amount of calculation of the error probability score of checkpoints, but also improves the efficiency of obtaining the thought chain while ensuring accuracy.

[0024] (7) The present invention scores the probability of errors at checkpoints based on consistency scoring, factual scoring and logical scoring, which is more reasonable and has a lower misjudgment rate.

[0025] (8) The reasoning method of the present invention is applicable to efficient and accurate reasoning of initial information sets in various fields and has high versatility. Attached Figure Description

[0026] Figure 1 This is an overall flowchart of a multimodal reasoning method based on error attribution according to the present invention; Figure 2 This is a schematic diagram of a thought chain structure. Detailed Implementation

[0027] To make the technical solution of the present invention clearer and more explicit, the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the present invention without creative effort all fall within the protection scope of the present invention.

[0028] The reasoning in this invention is based not only on the question itself but also on information related to the question; both the related information and the question itself can be multimodal. For ease of description, a question itself and its related information are considered as different initial pieces of information, forming a set of initial information. This invention is based on this initial set of information for reasoning. For example, someone experiencing discomfort asks in writing, "What disease might I have?" (first initial information). Related information includes: the person's written description of their symptoms (second initial information) and the person's lung CT scan image (third initial information). These three pieces of initial information constitute a set of initial information.

[0029] It should be emphasized that in this invention, different modalities refer only to visual and textual modes.

[0030] like Figure 1 The diagram shows a flowchart of a multimodal reasoning method based on error attribution, which includes the following steps: S1, based on the current modal fusion weights, performs modal fusion on each piece of initial information in the initial information set to generate a thought chain.

[0031] S2, after constructing the inference dependency graph based on the thought chain, select checkpoints in the inference dependency graph.

[0032] S3: Based on consistency, factuality, and logic, score the error probability of each checkpoint. If the error probability scores of all checkpoints in the current thought chain are below the set threshold, output the current thought chain; otherwise, mark the checkpoints with error probability scores exceeding the set threshold as error nodes and execute S4.

[0033] S4, calculate the relative contribution strength of different modes to the faulty node.

[0034] S5, based on the relative contribution strength, after updating the modal fusion weights, return to S1 and use the updated modal fusion weights to generate a new thought chain.

[0035] Optionally, if, for the same initial information set, there are still erroneous nodes in the resulting thought chain after updating the modal fusion weights z times, then stop updating the modal fusion weights and simultaneously report the error to the technical staff.

[0036] When the thought chain is output, the reasoning is completed.

[0037] S1 includes the following sub-steps: S11, Extract the feature matrix of each piece of initial information in the initial information set; S12, map all feature matrices to the same dimensional space for dimensional alignment; S13, based on the bidirectional cross-attention mechanism and modality fusion weight, the feature matrix after dimension alignment is modally fused to obtain the modality fusion matrix; S14, based on the modality fusion matrix, uses an inference decoder to generate a thought chain containing a sequence of inference steps.

[0038] Optionally, replace S13 with S13´: S13´, based on the bidirectional cross-attention mechanism and modality fusion weights, performs modality fusion on the dimension-aligned feature matrix and then performs self-attention enhancement, and then obtains the modality fusion matrix.

[0039] S11 also includes the following: using a pre-trained BioBERT model to extract initial information about the text type. semantic feature matrix The pre-trained Vision Transformer model is used to extract initial information about the image type. Image feature matrix .

[0040] S12 also includes the following: image feature matrix Mapped into the visual feature matrix in the text feature space ,in = ; It is a learnable linear projection weight matrix; This represents the bias vector of the BioBERT model. The linear projection weight matrix. The optimal cross-modal semantic mapping relationship is automatically learned through end-to-end training.

[0041] Because the text feature matrix and the visual feature matrix may have different dimensions, to facilitate subsequent processing, we map all feature matrices to the dimensional space of the text feature matrix, thus aligning their dimensions. This results in the visual feature matrix... and semantic feature matrix The dimensions are aligned.

[0042] In this embodiment, It can contain medical text information such as medical records and symptom descriptions. The BioBERT model has a feature dimension of 768. It can be used for medical imaging data, such as CT, MRI, X-ray films, etc. The feature dimension of the Vision Transformer model is 768 or 1024.

[0043] S13 also includes the following: fusion weight vector based on the current visual modality Weight vector fused with text modality A bidirectional cross-attention mechanism is used to generate a modality fusion matrix. : ; ; ; ; ; in, This represents the calculation result of the scaled dot product attention mechanism; Indicates use The function performs the calculation; , and These represent the text query matrix, text key matrix, and text value matrix, respectively. , and These represent the visual query matrix, visual key matrix, and visual value matrix, respectively. , and These are the first projection matrix, the second projection matrix, and the third projection matrix, respectively. This is a learnable projection matrix that, after training, has the optimal query mapping relationship. This is a matrix whose learnable projection matrix has the optimal key mapping relationship after training. This is a matrix whose learnable projection matrix has an optimal value mapping relationship after training. , and Both are parameter matrices; express Transpose of; express Transpose of; The dimension representing the attention head, ,in represents the feature dimension of the pre-trained BioBERT model, and h represents the number of attention heads, usually h=12.

[0044] exist During the calculation process, the softmax function is used to ensure the normalization of attention weights. The term is used for scaling to avoid the gradient vanishing problem.

[0045] S13' also includes the following: based on the current visual modality fusion weight vector Weight vector fused with text modality The first fusion matrix is ​​generated using a bidirectional cross-attention mechanism. Then, a multi-head self-attention mechanism is used to process the first fusion matrix. The modality fusion matrix is ​​obtained after performing self-attention enhancement. : ; ; ; ; ; ; in, This represents the calculation result of the scaled dot product attention mechanism; Indicates use The function performs the calculation; , and These represent the text query matrix, text key matrix, and text value matrix, respectively. , and These represent the visual query matrix, visual key matrix, and visual value matrix, respectively. , and These are the first projection matrix, the second projection matrix, and the third projection matrix, respectively. This is a learnable projection matrix that, after training, has the optimal query mapping relationship. This is a matrix whose learnable projection matrix has the optimal key mapping relationship after training. This is a matrix whose learnable projection matrix has an optimal value mapping relationship after training. , and Both are parameter matrices; express Transpose of; express Transpose of; The dimension representing the attention head, ,in The feature dimension of the pre-trained BioBERT model is represented by h, which represents the number of attention heads, usually h=12; Indicates use Multi-head self-attention mechanisms enhance self-attention.

[0046] If this is the first time reasoning is performed on the current initial information set, then the current visual modality fusion weight vector Weight vector fused with text modality All of these are pre-set initial modal fusion weight vectors.

[0047] S13´ compared to S13 in obtaining the modality fusion matrix Previously, an additional "perform self-attention enhancement" operation was added, which can further establish the first fusion matrix. Internal global dependencies improve the final modality fusion matrix. The completeness and coherence of multimodal representations provide a richer and more consistent semantic foundation for subsequent reasoning generation.

[0048] S14 also includes the following: based on the modality fusion matrix Using a pre-trained multimodal large model as an inference decoder, a thought chain containing a sequence of inference steps is generated: ; ; in, This represents the i-th reasoning step in the current thought chain, where 1 ≤ i ≤ n and i and n are both positive integers. This represents a structured prompt template, set by technical personnel, used to guide the model to directly output reasoning steps containing complete information; This represents the sequence of preceding inference steps that have already been generated; LLM ( This indicates the use of an LLM multimodal large model, based on the modality fusion matrix. and the sequence of preceding reasoning steps To generate the current inference step When i=1, LLM multimodal large model .

[0049] The pre-defined structured prompt template, set by technical personnel, contains inference generation instructions, formatting requirements, and quality assessment criteria. The structured prompt template first provides the current modality fusion matrix to the LLM multimodal large model. Using existing inference steps as context, and explicitly specifying the output format as a standardized JSON structure, the LLM multimodal large model is required to simultaneously evaluate the confidence of this step and identify its logical dependencies with preceding steps when generating inference content. Therefore, the inference steps in this invention... For the structured representation format {i, Content( ),CL( ),DL( )}, where i is the reasoning step identifier, used to uniquely identify the sequence symbol of the current reasoning step in the current thought chain; Content( ) represents the reasoning content of the current reasoning step, which is a natural language description; CL( ) represents the confidence score of the current inference step, reflecting the degree of confidence of the LLM multimodal large model in the accuracy of the current inference step; DL( This represents a list of logical dependencies for the current reasoning step, indicating the sequence of preceding reasoning steps that the current reasoning step depends on.

[0050] It should be noted here that, as Figure 2 The diagram shown illustrates a thought chain structure: where i = 6, and y represents the reasoning steps. Subsequently, three different reasoning steps were obtained simultaneously. , and Then, the reasoning steps The subsequent reasoning steps That is, for the current reasoning step. In other words, the sequence of preceding reasoning steps that has already been generated is: However, DL ( )=[ ].

[0051] Using an inference decoder based on a modality fusion matrix to generate thought chains is an existing technique, which will not be elaborated here.

[0052] S2 also includes the following sub-steps: S21, Constructing a reasoning dependency graph based on the thought chain. : Where N represents the set of nodes, and each node corresponds to a reasoning step in the current thought chain; This represents the set of edges, which are used to connect nodes in the logical dependency list that represent adjacent reasoning steps.

[0053] S22, inference dependency graph Nodes that meet one or more of the following conditions are selected as checkpoints: Condition 1, confidence score is lower than the first score threshold; Condition 2, out-degree is greater than the second score threshold; Condition 3, in-degree is greater than the third score threshold.

[0054] In this embodiment, the first scoring threshold is 0.7; the second scoring threshold is 2; and the third scoring threshold is 2.

[0055] In this invention, the nodes selected under condition one are potential problem nodes; the nodes selected under condition two are decision branch points, and the reasoning quality of decision branch points directly affects multiple subsequent reasoning paths; the nodes selected under condition three are information convergence nodes, and information convergence nodes need to integrate multiple preceding reasoning results, which can easily lead to information conflicts.

[0056] Inference Dependency Graph Some nodes may satisfy two or three conditions simultaneously.

[0057] S3 also includes the following sub-steps: S31, the average semantic consistency between the reasoning content corresponding to the current checkpoint and the reasoning content in the preceding reasoning step sequence on which it depends is recorded as the consistency score of the current checkpoint; the degree of conformity between the reasoning content corresponding to the current checkpoint and the facts is recorded as the factual score of the current checkpoint; the logical rationality between the reasoning content corresponding to the current checkpoint and the reasoning content corresponding to all nodes in the current thought chain is recorded as the logical score of the current checkpoint.

[0058] S32, record the current checkpoint as... checkpoint The corresponding reasoning steps are as follows: Then calculate the current checkpoint. Error probability rating : ; in, , and These are the first weight parameter, the second weight parameter, and the third weight parameter, respectively. and ;; , and These are the current checkpoints. Consistency score, factual score, and logical score; , and The range of values ​​is ; When the probability of error is rated When the score exceeds the fourth threshold, the corresponding checkpoint is marked as an error node; after marking all error nodes in the current thought chain, execute S4; if there are no error nodes in the current thought chain, output the current thought chain.

[0059] The consistency score of a checkpoint can be calculated using the cosine similarity formula of the semantic vectors corresponding to the reasoning content, and then averaged. A higher consistency score indicates a stronger semantic correlation between the current checkpoint and the preceding reasoning steps in the sequence upon which it depends. The factual score of a checkpoint is determined by a pre-trained neural network model, which scores the degree of conformity between the reasoning content and the facts. For example, if the reasoning content of a checkpoint is "What diseases correspond to a certain symptom?", then the fact is authoritative medical knowledge. A higher factual score indicates a smaller error or knowledge bias between the reasoning content and the facts. The logical score of a checkpoint is obtained by a pre-trained neural network model, based on all nodes in the current thought chain, analyzing the sufficiency of the premises, the logical validity of the reasoning process, and the reasonableness of the conclusion. A higher logical score indicates that the current checkpoint possesses a higher / more correct logical reasoning standard compared to other nodes in the current thought chain.

[0060] In this embodiment, the fourth score threshold is 0.6.

[0061] S4 also includes the following sub-steps: S41, Computational Visual Modality Fusion Matrix Text modality fusion matrix : ; .

[0062] S42, based on the visual modality fusion matrix Text modality fusion matrix A pre-trained multimodal large model is used as the inference decoder to generate the corresponding visual thought chain Chain´ and text thought chain Chain´´.

[0063] S43, select the first faulty node in the current thought chain, and denote this faulty node as... ; Retrieve the corresponding nodes in the visual thinking chain Chain´ and the text thinking chain Chain´´ respectively. Error probability rating and The reasoning step corresponding to the foremost error node is also the earliest reasoning step among all error nodes in the current thought chain.

[0064] S44, calculate the visual modality and text modality for the erroneous nodes in the current thought chain, respectively. absolute contribution intensity and : ; .

[0065] S45, calculate the visual modality and text modality for the erroneous nodes in the current thought chain, respectively. relative contribution intensity and : ; .

[0066] In this embodiment, , , and The range of values ​​is .

[0067] The higher the relative contribution strength value, the greater the contribution of the corresponding mode to the faulty node in the current thought chain. The greater the contribution of a reasoning error. and Whichever is larger, the corresponding mode is the fault node. The main source of errors.

[0068] Optionally, generate an error attribution report for the current thought chain. The error attribution report includes: the reasoning steps (i.e., the erroneous reasoning steps) corresponding to all erroneous nodes, the error probability score corresponding to each erroneous node, the sequence of preceding reasoning steps on which each erroneous reasoning step depends, and the relative contribution strength of various modalities to the first erroneous node in the current thought chain.

[0069] S5 also includes the following sub-steps: S51, Error Probability Scoring Calculate the weight adjustment coefficient β: ; Where λ is the sensitivity parameter; This represents the upper limit of the weight adjustment coefficient; min(·,·) denotes the minimum value function. Upper limit of weight adjustment coefficient It is used to limit the changes in the weight adjustment coefficient to prevent the weight adjustment coefficient from changing too drastically.

[0070] In this embodiment, λ=0.5. =0.3.

[0071] S52, Update the modal fusion weights based on the current weight adjustment coefficient β: ; ; in, and These represent the updated text modality fusion weight vector and the visual modality fusion weight vector, respectively.

[0072] Optionally, replace S52 with S52': S52´, based on the current weight adjustment coefficient β, update the modality fusion weights: ; ; in, and These represent the updated text modality fusion weight vector and the visual modality fusion weight vector, respectively. This represents the 2-norm.

[0073] Compared to S52, S52´ has undergone L2 normalization, which further ensures the stability of the fusion weights of each modality during the update process.

[0074] According to technical personnel, when reasoning with the same initial set of information, the reasoning results obtained by the reasoning method of this invention have a much higher accuracy rate than the existing technology, and the stability of the accuracy rate is also higher.

[0075] Technicians also verified that, for the same initial set of information and before the final output thought chain, the later thought chain, obtained sequentially before and after the final output thought chain, showed a decrease in both the number of error nodes and the error probability score of the corresponding error nodes.

[0076] The specific experimental content and data are as follows: ① Experimental environment and dataset configuration: The hardware environment consisted of 8 NVIDIA A100 GPUs (80GB VRAM), an Intel Xeon Gold 6330 processor, and 512GB of RAM; the software environment used Python 3.10, PyTorch 2.2, and CUDA 12.4. The model configuration used BERT-base as the text encoder, ViT-Base-16 as the visual encoder, and Qwen2.5-VL as the multimodal inference generator.

[0077] The dataset selection covers a variety of types to verify the generality of the method. Three datasets were selected: VQA v2, GQA and OK-VQA. (1) VQA v2.0 dataset contains about 1 million question-answer pairs, with 10 human answers for each question. It covers open-ended visual question answering in daily scenarios and mainly examines basic visual understanding and common sense reasoning. (2) GQA dataset contains about 22 million question-answer pairs. It emphasizes combinatorial reasoning and scene graph understanding ability and requires multi-step logical reasoning chains. (3) OK-VQA dataset contains 14,000 visual question answering samples that require external knowledge and evaluates knowledge-enhanced reasoning ability.

[0078] ② Comparison with existing technologies: (1) Existing technology 1: Standard Chain-of-Thought (CoT), a basic chain of thought prompting method that provides the most basic performance benchmark; (2) Existing technology 2: Multimodal-CoT, a two-stage multimodal CoT framework for generating reasoning basis and question answering, representing the current mainstream method of multimodal chain of thought; (3) Existing technology 3: Self-Consistency, a method to improve consistency by generating multiple reasoning paths; (4) Existing technology 4: Verify-and-Edit, a method for verifying enhanced chain of thought; (5) Existing technology 5: KAT (Knowledge Augmented Transformer), which retrieves relevant knowledge from an external knowledge base and integrates it into the reasoning process, and is specifically designed for knowledge-enhanced visual question answering tasks.

[0079] ③ Evaluation indicator system: A multi-dimensional evaluation index system is adopted to comprehensively measure the performance of the method.

[0080] For key performance metrics, VQA v2.0 uses the official VQA accuracy rate; GQA uses the exact match accuracy rate; and OK-VQA uses the official VQA accuracy rate.

[0081] For reasoning quality metrics, to quantify the semantic consistency and structural integrity of the reasoning chain, BERT Score-F1, ROUGE-L, and average reasoning steps are reported: (1) BERT Score F1 value, which assesses the semantic consistency of the reasoning chain with the high-quality reference template; (2) ROUGE-L score, which assesses the integrity and coherence of the reasoning steps; (3) Average reasoning steps, which calculates the average length of the reasoning chain and reflects the reasoning complexity.

[0082] For efficiency metrics, report the following metrics: (1) average inference time, the time cost of completing a single inference; (2) relative computational cost, the proportion of computational overhead relative to the standard CoT.

[0083] ⑤ Overall performance comparison test results

[0084] As shown in Table 1, the comprehensive evaluation results on the three datasets show that the method of the present invention achieves a stable performance improvement: the accuracy of the inference results obtained by the inference method of the present invention far exceeds that of the prior art.

[0085] Table 1 Overall Performance Comparison Results

[0086] Experimental results show that our method achieves a weighted average accuracy of 65.1%, the best performance on all three datasets. It achieves 73.6% on VQA v2.0 and 69.2% on GQA, demonstrating its advantage in combinatorial reasoning tasks. On OK-VQA, it achieves 52.4%, surpassing dedicated knowledge augmentation methods. In terms of efficiency, the inference time is 5.2 seconds, with a computational cost of 1.6 times, achieving a good balance between accuracy and efficiency.

[0087] ⑥ Reasoning quality assessment results

[0088] To verify the advantages of this method in generating high-quality reasoning processes, reasoning chain quality analysis was performed on 1000 samples in the validation set. The semantic consistency, completeness, and logical coherence of the reasoning chains were evaluated using multi-dimensional indicators to verify the effectiveness of the error attribution mechanism in improving reasoning quality. The reasoning quality evaluation results are shown in Table 2.

[0089] Table 2. Results of Inference Chain Quality Assessment

[0090] The inference quality assessment results show that our method achieves improved performance across all evaluation dimensions. The BERT score reaches 0.812, an improvement of 0.091 compared to the standard CoT, indicating that the generated inference chains are more accurate and coherent at the semantic level. The ROUGE-L score is 0.673, an average improvement of approximately 0.05 compared to baseline methods, reflecting improved completeness of inference steps and information coverage. The average number of inference steps is 4.1, a moderate increase reflecting a more detailed and complete inference process, contributing to improved interpretability. Overall, our method improves the quality and credibility of multimodal inference through error attribution and adaptive correction mechanisms.

[0091] Comprehensive experimental results demonstrate that the multimodal thinking chain method based on error attribution improves accuracy, reasoning quality, and computational efficiency. Through systematic comparative experiments and reasoning quality assessments, the methodological innovation of improving multimodal reasoning from an error analysis perspective proves its significant value.

[0092] The reasoning method of this invention first uses initial modal fusion weights to obtain an initial thought chain; then, it performs error checking on this thought chain. If no error-checking node is found in the current thought chain, the current thought chain is directly output; otherwise, the modal fusion weights are adjusted using the current thought chain, and a new thought chain is generated using the new modal fusion weights. In other words, the reasoning method of this invention can obtain an accurate thought chain.

[0093] This invention does not output the thought chain as the final result immediately. Instead, it performs error checking on the obtained thought chain (i.e., scoring the probability of errors at checkpoints in the thought chain). Only thought chains without error nodes are output, ensuring high accuracy. When error checking reveals thought chains with error nodes, the modality fusion weights are adjusted based on the relative contribution strength of different modalities to the error nodes. A new thought chain generated using the new modality fusion weights is then used, further reducing the number of error nodes and the degree of error. In other words, this invention can automatically and step-by-step correct erroneous reasoning steps in the initial thought chain to the correct path through a closed-loop process. Therefore, the reasoning method of this invention not only improves the accuracy of the reasoning results but also maintains the stability of accurate reasoning results.

[0094] This invention uses the presence of erroneous nodes in the thought chain as direct feedback; and based on the relative contribution strength of erroneous nodes in the thought chain, the entire process can be completed without the intervention of technical personnel, thereby enabling targeted adjustment of the modal fusion weights used for different initial information sets, improving the accuracy of the final reasoning result while significantly saving labor costs.

[0095] Furthermore, as mentioned in the background section, inappropriate modality fusion weights can directly lead to an intrinsic error in a certain inference step. This intrinsic error, in turn, causes subsequent inference steps to be identified as erroneous nodes due to error accumulation. Therefore, in the inference method of this invention, when calculating the relative contribution strength of different modalities to erroneous nodes, the "erroneous node" we select is the first erroneous node in the current thought chain. The reasons are as follows: ① While ensuring that the number of faulty nodes in the new thinking chain is reduced and the degree of error is mitigated, we can reduce the amount of computation when adjusting the modality fusion weights (there is no need to calculate the relative contribution strength of different modalities to all faulty nodes in the current thinking chain).

[0096] ② The causes of erroneous nodes in the thought chain are mutually interfering. Some erroneous nodes are not only due to the accumulation of errors from preceding reasoning steps, but also due to inappropriate modality fusion weights. However, in a thought chain, the first erroneous node usually appears because of inappropriate modality fusion weights. Therefore, this invention makes the error correction process more scientific by selecting appropriate erroneous nodes to adjust the modality fusion weights. This scientific nature is also reflected in the ability to directly identify erroneous nodes in the current thought chain. The accumulated errors are stripped away from the new thought chain, ensuring that the cause of the first error node in the newly generated thought chain is still an inappropriate modal fusion weight.

[0097] Therefore, the reasoning method of this invention, while conforming to the understanding that erroneous nodes in the reasoning chain cannot be completely corrected at once due to mutual interference, can obtain a highly accurate thought chain through several corrections, thus improving the efficiency of reasoning error correction. This invention corrects errors in the thought chain based on error attribution; where "error attribution" refers to the selected erroneous nodes used to adjust the modality fusion weights, and each generation of a new thought chain involves error correction.

[0098] If the initial thought chain contains erroneous nodes, then the initial thought chain contains the most erroneous nodes. This can be seen as follows: for the initial thought chain, as long as the new modality fusion weight value falls within a relatively wide and reasonable range, the number of erroneous nodes in the new thought chain can be reduced. The new modality fusion weight we calculated falls within this reasonable range. However, in the subsequent adjustment of the modality fusion weight, this reasonable range continuously narrows. Therefore, the modality fusion weight calculated in this invention is more refined compared to the previous modality fusion weight.

[0099] In the reasoning method of this invention, the modality fusion matrix Used to efficiently fuse information from different modalities.

[0100] In the reasoning method of this invention, not all nodes are directly used as checkpoints. Instead, checkpoints are selected based on three conditions. This not only reduces the computational workload of calculating the error probability score of checkpoints, but also improves the efficiency of obtaining the subsequent thought chain while ensuring accuracy.

[0101] The present invention scores the probability of checkpoint errors based on consistency scoring, factual scoring, and logical scoring, which is more reasonable and has a lower misjudgment rate.

[0102] The reasoning method of this invention is applicable to efficient and accurate reasoning on initial information sets in various fields, and has high versatility.

[0103] The present invention also provides a reasoning system, comprising: The module includes a thought chain generation module, a checkpoint selection module, a scoring module, a calculation module, and an update module. The thought chain generation module is used to perform modal fusion on each piece of initial information in the initial information set based on the current modal fusion weight, generate a thought chain, and then send it to the checkpoint selection module. After the checkpoint selection module constructs a reasoning dependency graph based on the thought chain, it selects checkpoints in the reasoning dependency graph and then sends the dependency graph and checkpoints to the scoring module. The scoring module scores the error probability of each checkpoint. If the error probability scores of all checkpoints in the current thought chain are below the set threshold, the scoring module outputs the current thought chain; otherwise, the scoring module marks the checkpoints with error probability scores exceeding the set threshold as error nodes and sends them to the calculation module. The calculation module is used to calculate the relative contribution strength of different modes to the faulty node and send it to the update module; The update module updates the modal fusion weights based on the relative contribution strength and then sends them to the thought chain generation module.

[0104] Each module is programmed or configured to perform the steps of a multimodal reasoning method based on error attribution, as described above.

[0105] The technologies, shapes, and structures not described in detail in this invention are all well-known technologies. It should also be noted that the above are merely preferred embodiments of this invention and are not intended to limit the scope of the invention. The components or steps in the embodiments of this invention can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered equivalent solutions to this application and should all fall within the protection scope of this invention.

Claims

1. A multimodal reasoning method based on error attribution, characterized in that, Includes the following steps: S1, based on the current modal fusion weights, modal fusion is performed on each piece of initial information in the initial information set to generate a thought chain; S2, after constructing the inference dependency graph based on the thought chain, select checkpoints in the inference dependency graph; S3, based on consistency, factuality and logic, scores the error probability of each checkpoint. If the error probability scores of all checkpoints in the current thought chain are below the set threshold, then output the current thought chain. Otherwise, mark the checkpoints whose error probability scores exceed the set threshold as error nodes and execute S4; S4, calculate the relative contribution intensity of different modes to the faulty node; S5, based on the relative contribution strength, after updating the modal fusion weights, return to S1 and use the updated modal fusion weights to generate a new thought chain.

2. The multimodal reasoning method based on error attribution according to claim 1, characterized in that, S1 also includes the following sub-steps: S11, Extract the feature matrix of each piece of initial information in the initial information set; S12, map all feature matrices to the same dimensional space for dimensional alignment; S13, based on the bidirectional cross-attention mechanism and modality fusion weight, the feature matrix after dimension alignment is modally fused to obtain the modality fusion matrix; S14, based on the modality fusion matrix, uses an inference decoder to generate a thought chain containing a sequence of inference steps.

3. The multimodal reasoning method based on error attribution according to claim 2, characterized in that: S11 also includes the following: extracting initial information for the text type. semantic feature matrix Extract initial information of image type Image feature matrix ; S12 also includes the following: image feature matrix Mapped into the visual feature matrix in the text feature space ,in = ; It is a learnable linear projection weight matrix; To extract semantic feature matrix The model bias vector; S13 also includes the following: Based on the current visual modality fusion weight vector Weight vector fused with text modality A bidirectional cross-attention mechanism is used to generate a modality fusion matrix. : ； ； ； ； ； in, This represents the calculation result of the scaled dot product attention mechanism; Indicates use The function performs the calculation; , and These represent the text query matrix, text key matrix, and text value matrix, respectively. , and These represent the visual query matrix, visual key matrix, and visual value matrix, respectively. , and These are the first projection matrix, the second projection matrix, and the third projection matrix, respectively. This is a learnable projection matrix that, after training, has the optimal query mapping relationship. This is a matrix whose learnable projection matrix has the optimal key mapping relationship after training. It is a matrix that has the optimal value mapping relationship after the learnable projection matrix is ​​trained; express Transpose of; express Transpose of; The dimension representing the attention head, ,in Represents the extraction of semantic feature matrix The feature dimension of the model is h, which represents the number of attention heads.

4. The multimodal reasoning method based on error attribution according to claim 3, characterized in that, S14 also includes the following: Based on modality fusion matrix Using a pre-trained multimodal large model as an inference decoder, a thought chain containing a sequence of inference steps is generated: ； ； in, This represents the i-th reasoning step in the current thought chain, where 1 ≤ i ≤ n and i and n are both positive integers. This indicates a structured suggestion template; This represents the sequence of preceding inference steps that have already been generated; LLM ( This indicates the use of an LLM multimodal large model, based on the modality fusion matrix. and the sequence of preceding reasoning steps To generate the current inference step When i=1, LLM multimodal large model Content( () indicates the reasoning steps The reasoning content; CL( () indicates the reasoning steps Confidence score; DL ( () indicates the reasoning steps A list of logical dependencies.

5. A multimodal reasoning method based on error attribution according to claim 2, characterized in that, S2 also includes the following sub-steps: S21, Constructing a reasoning dependency graph based on the thought chain. : Where N represents the set of nodes, and each node corresponds to a reasoning step in the current thought chain; This represents the set of edges, which are used to connect nodes in the logical dependency list that represent adjacent reasoning steps. S22, inference dependency graph Nodes that meet one or more of the following conditions are selected as checkpoints: Condition 1, confidence score is lower than the first score threshold; Condition 2, out-degree is greater than the second score threshold; Condition 3, in-degree is greater than the third score threshold.

6. The multimodal reasoning method based on error attribution according to claim 5, characterized in that, S3 also includes the following sub-steps: S31, the average semantic consistency between the reasoning content corresponding to the current checkpoint and the reasoning content in the preceding reasoning step sequence on which it depends is recorded as the consistency score of the current checkpoint; the degree of conformity between the reasoning content corresponding to the current checkpoint and the facts is recorded as the factual score of the current checkpoint; the logical rationality between the reasoning content corresponding to the current checkpoint and the reasoning content corresponding to all nodes in the current thought chain is recorded as the logical score of the current checkpoint. S32, record the current checkpoint as... checkpoint The corresponding reasoning steps are as follows: Then calculate the current checkpoint. Error probability rating : ； in, , and These are the first weight parameter, the second weight parameter, and the third weight parameter, respectively. and ; , and These are the current checkpoints. Consistency score, factual score, and logical score; When the probability of error is rated When the score exceeds the fourth threshold, the corresponding checkpoint is marked as an error node; after marking all error nodes in the current thought chain, execute S4; if there are no error nodes in the current thought chain, output the current thought chain.

7. A multimodal reasoning method based on error attribution according to claim 6, characterized in that, S4 also includes the following sub-steps: S41, Computational Visual Modality Fusion Matrix Text modality fusion matrix : ； ； ； ； in, This represents the calculation result of the scaled dot product attention mechanism; Indicates use The function performs the calculation; , and These represent the text query matrix, text key matrix, and text value matrix, respectively. , and These represent the visual query matrix, visual key matrix, and visual value matrix, respectively. express Transpose of; express Transpose of; The dimension representing the attention head, ,in Represents the extraction of semantic feature matrix The feature dimension of the model, h represents the number of attention heads; S42, based on the visual modality fusion matrix Text modality fusion matrix Using a pre-trained multimodal large model as an inference decoder, the corresponding visual thought chain Chain´ and text thought chain Chain´´ are generated. S43, select the earliest erroneous node in the current thought chain, and denote this erroneous node as... ; Retrieve the corresponding nodes in the visual thinking chain Chain´ and the text thinking chain Chain´´ respectively. Error probability rating and ´´; S44, calculate the visual modality and text modality for the erroneous nodes in the current thought chain, respectively. absolute contribution intensity and : ； ； S45, calculate the visual modality and text modality for the erroneous nodes in the current thought chain, respectively. relative contribution intensity and : ； 。 8. A multimodal reasoning method based on error attribution according to claim 7, characterized in that, S5 also includes the following sub-steps: S51, Error Probability Scoring Calculate the weight adjustment coefficient β: Where λ is the sensitivity parameter; This is the upper limit of the weight adjustment coefficient; min( , () represents the function that takes the minimum value; S52, Update the modal fusion weights based on the current weight adjustment coefficient β: ； ； in, and These represent the updated text modality fusion weight vector and the visual modality fusion weight vector, respectively.

9. A multimodal reasoning method based on error attribution according to claim 8, characterized in that, Replace S52 with S52': S52´, based on the current weight adjustment coefficient β, update the modality fusion weights: ； ； in, and These represent the updated text modality fusion weight vector and the visual modality fusion weight vector, respectively. This represents the 2-norm.

10. A reasoning system, characterized in that, include: The module includes a thought chain generation module, a checkpoint selection module, a scoring module, a calculation module, and an update module. The thought chain generation module is used to perform modal fusion on each piece of initial information in the initial information set based on the current modal fusion weight, generate a thought chain, and then send it to the checkpoint selection module. After constructing a reasoning dependency graph based on the thought chain, the checkpoint selection module selects checkpoints in the graph and sends the graph and checkpoints to the scoring module. The scoring module scores the error probability of each checkpoint. If the error probability scores of all checkpoints in the current thought chain are below a set threshold, the scoring module outputs the current thought chain. Otherwise, the scoring module marks checkpoints with error probability scores exceeding the set threshold as erroneous nodes and sends them to the calculation module. The calculation module calculates the relative contribution strength of different modalities to erroneous nodes and sends it to the update module. The update module updates the modality fusion weights based on the relative contribution strength and then sends them to the thought chain generation module; Each module is programmed or configured to perform the steps of a multimodal reasoning method based on error attribution as described in any one of claims 1-9.

Images