A multi-modal reasoning method and reasoning system based on error attribution

Abstract

The present application belongs to the technical field of thought chain reasoning, and particularly relates to a multi-modal reasoning method and reasoning system based on error attribution. The reasoning method comprises: based on the current modal fusion weight, performing modal fusion on each initial information in the initial information set to generate a thought chain; after constructing a reasoning dependency graph based on the thought chain, selecting a checkpoint in the reasoning dependency graph; based on consistency, factuality and logicality, scoring the error possibility of each checkpoint, if the error possibility scores of all checkpoints in the current thought chain are below a set threshold, the current thought chain is output; otherwise, the checkpoint with an error possibility score exceeding the set threshold is marked as an error node; the relative contribution intensity of different modalities to the error node is calculated; based on the relative contribution intensity, the modal fusion weight is updated, and the thought chain is regenerated. The present application can improve the accuracy and stability of the reasoning result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of thought chain reasoning, and particularly relates to a multi-modal reasoning method and a reasoning system based on error attribution. BACKGROUND

[0002] In recent years, the CoT (Chain-of-Thought) technology has made significant progress in the field of artificial intelligence, evolving from early single-step reasoning to complex step-by-step reasoning frameworks based on large language models (LLM). Modern CoT technology simulates the human problem-solving process by breaking down complex problems into a series of reasoning steps, effectively improving the model's performance on tasks such as mathematical reasoning, logical judgment, and common sense reasoning.

[0003] The degree of participation of different modal information as the basis for reasoning directly affects the accuracy of the reasoning result. For example, if the degree of participation of different modal information is not appropriate, it will directly lead to self-error in a reasoning step (self-error refers to the cause of the error being unrelated to the correctness of the previous step as input, and only related to the calculation of the reasoning step itself). The reasoning step with self-error will also accumulate errors in subsequent reasoning steps. For example, in the context of solving mathematical problems, a small calculation error (such as inappropriate degree of participation of modal information leading to self-error in a reasoning step) will cause the entire problem-solving process to deviate from the correct path; and as the number of reasoning steps increases, the errors in the previous reasoning steps will also accumulate, and the interaction of these reasons will significantly reduce the accuracy of the reasoning result.

[0004] Even if a verification mechanism is introduced, it can only verify the correctness of the reasoning result, and the intermediate reasoning steps that produce errors cannot be known, which naturally reduces the efficiency of correcting the reasoning result, leading to unstable accuracy of the reasoning result and low efficiency of correcting reasoning errors. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a multi-modal reasoning method based on error attribution, which can improve the accuracy and stability of the reasoning result.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A multi-modal reasoning method based on error attribution, comprising the following steps:

[0008] S1, based on the current modal fusion weight, performing modal fusion on each initial information in the initial information set to generate a thought chain;

[0009] S2, constructing a reasoning dependency graph based on the thought chain, and selecting a checkpoint in the reasoning dependency graph;

[0010] S3, based on consistency, factuality and logicality, scoring the error possibility of each checkpoint, if the error possibility scores of all checkpoints in the current thought chain are below the set threshold, output the current thought chain; otherwise, mark the checkpoint whose error possibility score exceeds the set threshold as an error node and perform S4;

[0011] S4, calculating the relative contribution intensity of different modalities to the error node;

[0012] S5, based on the relative contribution intensity, updating the modal fusion weight, and then returning to S1 to generate a new thought chain using the updated modal fusion weight.

[0013] Preferably, in S1, the following sub-steps are further included:

[0014] S11, extracting the feature matrix of each initial information in the initial information set;

[0015] S12, mapping all feature matrices to the same dimensional space for dimensional alignment;

[0016] S13, based on the bidirectional cross-attention mechanism and the modal fusion weight, performing modal fusion on the dimensionally aligned feature matrix to obtain a modal fusion matrix;

[0017] S14, based on the modal fusion matrix, using a reasoning decoder to generate a thought chain containing a sequence of reasoning steps.

[0018] Preferably, in S11, the following is further included: extracting the semantic feature matrix of the text type initial information ; extracting the image feature matrix of the image type initial information ; ;

[0019] In S12, the following is further included: mapping the image feature matrix into a visual feature matrix in the text feature space , where = ; is a learnable linear projection weight matrix; is a model bias vector for extracting the semantic feature matrix ;

[0020] In S13, the following is further included: based on the current visual modal fusion weight vector and the text modal fusion weight vector , using the bidirectional cross-attention mechanism to generate a modal fusion matrix :

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] wherein, denotes the scaled dot-product attention mechanism calculation result; denotes the calculation using function; , and denote the text query matrix, the text key matrix and the text value matrix, respectively; , and denote the visual query matrix, the visual key matrix and the visual value matrix, respectively; , and are the first projection matrix, the second projection matrix and the third projection matrix, respectively; is the matrix of the learnable projection matrix having the optimal query mapping relationship after training and learning, is the matrix of the learnable projection matrix having the optimal key mapping relationship after training and learning, is the matrix of the learnable projection matrix having the optimal value mapping relationship after training and learning; denotes the transpose of ; denotes the transpose of ; denotes the dimension of the attention head, wherein denotes the feature dimension of the model extracting the semantic feature matrix , and h denotes the number of attention heads.

[0027] Preferably, in S14, the following content is further included: based on the modal fusion matrix , using a pre-trained multi-modal large model as a reasoning decoder, a thought chain Chain containing a sequence of reasoning steps is generated:

[0028] ;

[0029] ;

[0030] wherein, denotes the i-th reasoning step in the current thought chain Chain, 1≤i≤n and i and n are both positive integers; representing a structured prompt template; representing a current generated sequence of antecedent reasoning steps; representing generating a current reasoning step using a LLM multimodal large model based on a modal fusion matrix and a sequence of antecedent reasoning steps ; when i = 1, the LLM multimodal large model ; Content( ) represents the reasoning content of the reasoning step ; CL( ) represents the confidence score of the reasoning step ; DL( ) represents the list of logical dependencies of the reasoning step .

[0031] Preferably, the following sub-steps are further included in S2:

[0032] S21, constructing a reasoning dependency graph based on the thought chain : ; wherein N represents a set of nodes, each node corresponding to a reasoning step in the current thought chain; represents a set of edges, edges are used to connect nodes corresponding to adjacent reasoning steps in the list of logical dependencies;

[0033] S22, selecting nodes in the reasoning dependency graph that meet one or more conditions as checkpoints: condition one, the confidence score is lower than a first score threshold; condition two, the out-degree is greater than a second score threshold; condition three, the in-degree is greater than a third score threshold.

[0034] Preferably, the following sub-steps are further included in S3:

[0035] S31, recording the average value of semantic consistency between the reasoning content corresponding to the current checkpoint and the reasoning content in the sequence of antecedent reasoning steps on which it depends as the consistency score of the current checkpoint; recording the degree of compliance of the reasoning content corresponding to the current checkpoint with the fact as the factuality score of the current checkpoint; recording the logical rationality between the reasoning content corresponding to the current checkpoint and the reasoning content corresponding to all nodes in the current thought chain Chain as the logicality score of the current checkpoint;

[0036] S32, recording the current checkpoint as , the reasoning step corresponding to the checkpoint is , then calculating the error possibility score of the current checkpoint :

[0037] ​ ;

[0038] wherein, , and are a first weight parameter, a second weight parameter and a third weight parameter, respectively, and ; , and are a consistency score, a factuality score and a logicality score of the current checkpoint, respectively;

[0039] When the error likelihood score exceeds a fourth score threshold, the corresponding checkpoint is marked as an error node; after marking all error nodes in the current thinking chain Chain, S4 is executed; if there is no error node in the current thinking chain, the current thinking chain Chain is output.

[0040] Preferably, S4 further comprises the following sub-steps:

[0041] S41, calculating a visual modality fusion matrix and a text modality fusion matrix :

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] wherein, denotes the scaled dot-product attention mechanism calculation result; denotes the calculation using function; , and denote the text query matrix, the text key matrix and the text value matrix, respectively; , and denote the visual query matrix, the visual key matrix and the visual value matrix, respectively; denotes the transpose of ; denotes the transpose of ; denotes the dimension of the attention head, wherein denotes the extracted semantic feature matrix the feature dimension of the model, h represents the number of attention heads;

[0047] S42, respectively based on the visual modality fusion matrix and the text modality fusion matrix , using a pre-trained multi-modal large model as a reasoning decoder, generate the corresponding visual thought chain Chain´ and the text thought chain Chain´´;

[0048] S43, select the first error node of the reasoning step in the current thought chain Chain, mark the error node as ; respectively obtain the error possibility score ´ and ´´ of the corresponding node in the visual thought chain Chain´ and the text thought chain Chain´´;

[0049] S44, respectively calculate the absolute contribution intensity of the visual modality and the text modality to the error node in the current thought chain Chain and :

[0050] ; ;

[0051] S45, respectively calculate the relative contribution intensity of the visual modality and the text modality to the error node in the current thought chain Chain and :

[0052] ; .

[0053] Preferably, in S5, the following sub-steps are further included:

[0054] S51, based on the error possibility score , calculate the weight adjustment coefficient β:

[0055] ; wherein, λ is a sensitivity parameter; is the upper limit of the weight adjustment coefficient; min( , ) represents the minimum value function;

[0056] S52, based on the current weight adjustment coefficient β, update the modality fusion weight:

[0057] ; ;

[0058] wherein, and respectively represent the updated text modality fusion weight vector and the updated visual modality fusion weight vector.

[0059] Preferably, S52 is replaced by S52´:

[0060] S52´, based on the current weight adjustment coefficient β, updates the modality fusion weight:

[0061] ; ;

[0062] wherein, and respectively represent the updated text modality fusion weight vector and the updated visual modality fusion weight vector; represents the 2-norm.

[0063] The present application also provides a system, a reasoning system, comprising: a thought chain generation module, a checkpoint selection module, a scoring module, a calculation module, an updating module; the thought chain generation module is used to generate a thought chain after modality fusion of each initial information in an initial information set based on a current modality fusion weight, and then send the thought chain to the checkpoint selection module; the checkpoint selection module constructs a reasoning dependency graph based on the thought chain, selects a checkpoint in the reasoning dependency graph, and then sends the dependency graph and the checkpoint to the scoring module; the scoring module scores the error possibility of each checkpoint, and if the error possibility scores of all the checkpoints in the current thought chain are all below a set threshold, the scoring module outputs the current thought chain; otherwise, the scoring module marks the checkpoint whose error possibility score exceeds the set threshold as an error node, and then sends the error node to the calculation module; the calculation module is used to calculate the relative contribution intensity of different modalities to the error node and send the relative contribution intensity to the updating module; the updating module updates the modality fusion weight based on the relative contribution intensity, and then sends the updated modality fusion weight to the thought chain generation module; each module is programmed or configured to perform the steps of the error attribution based multi-modal reasoning method as described above.

[0064] The present application has the following advantages:

[0065] (1) The reasoning method of the present application first uses an initial modality fusion weight to obtain an initial thought chain; then performs error checking on the thought chain, and if there is no error checking node in the current thought chain, the current thought chain is directly output; otherwise, the modality fusion weight is adjusted using the current thought chain, and a new thought chain is generated using the new modality fusion weight. That is, the reasoning method of the present application can obtain an accurate thought chain.

[0066] (2) The present application is not a thought chain as the final output, but the error check (i.e. the error possibility score of the checkpoint in the thought chain) of the obtained thought chain, only the thought chain without error node will be output, which ensures that the thought chain output by the present application has high accuracy. In the error check, the thought chain with error node is found, the relative contribution intensity of different modalities to the error node in the thought chain is calculated based on the current thought chain, the modal fusion weight is adjusted, and the new thought chain generated by the new modal fusion weight is used to further reduce the error node and the error degree, that is, the present application can automatically and step by step through the closed loop cycle to place the error reasoning step in the initial thought chain to the correct path. That is, the reasoning method of the present application can not only improve the accuracy of the reasoning result, but also maintain the stability of the accurate reasoning result.

[0067] (3) The present application uses the existence of error nodes in the thought chain as direct feedback, and according to the relative contribution intensity of the error nodes in the thought chain, the modal fusion weight used for different initial information sets can be adjusted without the intervention of technical personnel, which not only improves the accuracy of the final reasoning result, but also greatly saves the labor cost.

[0068] (4) In the reasoning method of the present application, in the calculation of the relative contribution intensity of different modalities to the error node, the error node we select is the most front error node in the current thought chain Chain , the reasons are as follows:

[0069] ① While ensuring the reduction of the number of error nodes and the reduction of the error degree of the new thought chain, the calculation amount of adjusting the modal fusion weight can be reduced (without calculating the relative contribution intensity of different modalities to all error nodes in the current thought chain Chain).

[0070] ② The reason for the existence of error nodes in the thought chain is mutual interference, some error nodes are not only caused by error accumulation of previous reasoning steps, but also caused by inappropriate modal fusion weight. However, in a thought chain, the most front error node is usually caused by inappropriate modal fusion weight, so the present application adjusts the modal fusion weight by selecting the appropriate error node to make the correction process of reasoning error more scientific, and this science also reflects that the error accumulation of the error node in the current thought chain can be removed from the new thought chain, ensuring that the most front error node in the newly generated thought chain is still caused by inappropriate modal fusion weight.

[0071] Therefore, the reasoning method of the present application can obtain a thinking chain with high accuracy through several times of correction while conforming to the cognition that error nodes in the reasoning chain cannot be corrected all at once due to mutual interference, thereby improving the correction efficiency of reasoning errors. The present application performs error correction of the thinking chain based on error attribution. The "error attribution" refers to the error node selected to adjust the modal fusion weight. Each time a new thinking chain is generated, error correction is performed.

[0072] (5) For the initial thinking chain, as long as the new modal fusion weight value falls within a relatively wide reasonable range, the error nodes in the new thinking chain can be reduced, and the new modal fusion weight calculated by the present application is within the reasonable range. However, during the subsequent adjustment of the modal fusion weight, the reasonable range is continuously reduced, so the modal fusion weight calculated by the present application is more precise than the previous modal fusion weight.

[0073] (6) In the reasoning method of the present application, instead of directly taking all nodes as checkpoints, the checkpoints are selected based on three conditions, which not only reduces the calculation amount of the error possibility score of the checkpoints, but also improves the efficiency of obtaining the thinking chain while ensuring accuracy.

[0074] (7) The error possibility score of the checkpoints in the present application is obtained based on consistency score, fact score and logic score, which is more reasonable and has a lower misjudgment rate.

[0075] (8) The reasoning method of the present application is suitable for efficient and high-accuracy reasoning of initial information sets in various fields, and has high universality. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 is a whole flowchart of a multi-modal reasoning method based on error attribution of the present application;

[0077] Figure 2 is a structural schematic diagram of a thinking chain. DETAILED DESCRIPTION

[0078] In order to make the technical solutions of the present application clearer and more explicit, the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Those skilled in the art can obtain solutions by equivalent replacement and conventional reasoning of the technical features of the technical solutions of the present application without making creative efforts, which fall within the protection scope of the present application.

[0079] The basis of the reasoning of the present application not only includes the problem itself, but also the information related to the problem; the related information and the problem itself can be multi-modal. For the convenience of description, the problem itself and its related information are different initial information, which constitutes an initial information set, that is, the present application is based on an initial information set for reasoning. For example: a person is ill, and the question "What disease may I have?" is asked through the text (the first initial information), and the information related to the question includes: the person's own symptom description (the second initial information), and the person's lung CT image (the third initial information). The above three initial information constitutes an initial information set.

[0080] It should be emphasized that the different modalities in the present application refer to vision and text only.

[0081] As shown in Figure 1 , it is a flow chart of a multi-modal reasoning method based on error attribution, which includes the following steps:

[0082] S1, based on the current modal fusion weight, the modal fusion of each initial information in the initial information set is carried out, and a thought chain is generated.

[0083] S2, after constructing a reasoning dependency graph based on the thought chain, selecting a checkpoint in the reasoning dependency graph.

[0084] S3, based on consistency, factuality and logicality, scoring the error possibility of each checkpoint, if the error possibility scores of all checkpoints in the current thought chain are below the set threshold, the current thought chain is output; otherwise, the checkpoint whose error possibility score exceeds the set threshold is marked as an error node and S4 is executed.

[0085] S4, calculating the relative contribution intensity of different modalities to the error node.

[0086] S5, based on the relative contribution intensity, updating the modal fusion weight, and returning to S1 to generate a new thought chain using the updated modal fusion weight.

[0087] Optionally, if there are still error nodes in the thought chain obtained after the modal fusion weight is updated z times for the same initial information set, the updating of the modal fusion weight is stopped and an error is reported to the technician at the same time.

[0088] When the thought chain is output, the reasoning is completed.

[0089] In S1, the following sub-steps are included:

[0090] S11, extracting the feature matrix of each initial information in the initial information set;

[0091] S12, map all feature matrices to the same dimensional space for dimensional alignment;

[0092] S13, based on the bidirectional cross-attention mechanism and the modal fusion weight, modal fusion is performed on the dimensionally aligned feature matrix to obtain a modal fusion matrix;

[0093] S14, based on the modal fusion matrix, a reasoning decoder is used to generate a thought chain containing a sequence of reasoning steps.

[0094] Optionally, S13 is replaced by S13': S13', based on the bidirectional cross-attention mechanism and the modal fusion weight, the dimensionally aligned feature matrix is modal fused and then self-attention enhanced to obtain the modal fusion matrix.

[0095] In S11, the following is further included: the pre-trained BioBERT model is used to extract the semantic feature matrix of the text type initial information ; the pre-trained Vision Transformer model is used to extract the image feature matrix of the image type initial information

[0096] In S12, the following is further included: the image feature matrix is mapped into a visual feature matrix in the text feature space , where = ; is a learnable linear projection weight matrix; is a bias vector of the BioBERT model. The linear projection weight matrix learns the optimal cross-modal semantic mapping relationship automatically through end-to-end training.

[0097] Because the text feature matrix and the visual feature matrix may have different dimensions, in order to facilitate subsequent processing, all feature matrices are mapped to the dimensional space where the text feature matrix is located, so that they are dimensionally aligned, that is, the dimensions of the visual feature matrix and the semantic feature matrix are aligned.

[0098] In this embodiment, may contain medical text information such as medical records and symptom descriptions, and the feature dimension of the BioBERT model is 768; may be medical image data such as CT, MRI, and X-ray films, and the feature dimension of the Vision Transformer model is 768 or 1024.

[0099] ​​​In S13, the following is also included: generating a modal fusion matrix based on the current visual modal fusion weight vector and the text modal fusion weight vector using a bidirectional cross-attention mechanism :

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] wherein, denotes the scaled dot-product attention mechanism calculation result; denotes the calculation using function; , and denote the text query matrix, the text key matrix and the text value matrix respectively; , and denote the visual query matrix, the visual key matrix and the visual value matrix respectively; , and are the first projection matrix, the second projection matrix and the third projection matrix respectively; is a matrix with optimal query mapping relationship after training and learning of the learnable projection matrix, is a matrix with optimal key mapping relationship after training and learning of the learnable projection matrix, is a matrix with optimal value mapping relationship after training and learning of the learnable projection matrix, , and are all parameter matrices; denotes the transpose of ; denotes the transpose of ; denotes the dimension of the attention head, wherein denotes the feature dimension of the pre-trained BioBERT model, and h denotes the number of attention heads, usually h=12.

[0106] In the calculation process of , the softmax function is used to ensure the normalization of the attention weight, The item is used for scaling to avoid the problem of gradient disappearance.

[0107] In S13', the following is further included: based on the current visual modal fusion weight vector and the text modal fusion weight vector , a first fusion matrix is generated using a bidirectional cross attention mechanism; and after self-attention enhancement of the first fusion matrix using a multi-head self-attention mechanism, a modal fusion matrix is obtained:

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] wherein, represents a scaled dot-product attention mechanism calculation result; represents calculation using function; , and represent a text query matrix, a text key matrix and a text value matrix, respectively; , and represent a visual query matrix, a visual key matrix and a visual value matrix, respectively; , and are a first projection matrix, a second projection matrix and a third projection matrix, respectively; is a matrix with optimal query mapping relationship of the learnable projection matrix after training and learning, is a matrix with optimal key mapping relationship of the learnable projection matrix after training and learning, is a matrix with optimal value mapping relationship of the learnable projection matrix after training and learning, , and are all parameter matrices; represents a transpose of ; represents a transpose of ; represents a dimension of an attention head, wherein denotes the feature dimension of the pre-trained BioBERT model, h denotes the number of attention heads, usually h = 12; denotes the use of multi-head self-attention mechanism for self-attention enhancement.

[0115] If it is the first time to reason on the current initial information set, the current visual modality fusion weight vector and the text modality fusion weight vector are both pre-set initial modality fusion weight vectors.

[0116] S13' has an additional operation of "self-attention enhancement" than S13 before obtaining the modality fusion matrix , which can further establish the global dependency within the first fusion matrix , and improve the integrity and coherence of the multi-modal representation in the final obtained modality fusion matrix , providing a more abundant and consistent semantic basis for subsequent reasoning generation.

[0117] In S14, the following content is also included: based on the modality fusion matrix , using a pre-trained multi-modal large model as a reasoning decoder, a thought chain Chain containing a sequence of reasoning steps is generated:

[0118] ;

[0119] ;

[0120] wherein, denotes the i-th reasoning step in the current thought chain Chain, 1≤i≤n and i and n are both positive integers; denotes a structured prompt template set by the technical personnel, used to guide the model to directly output a reasoning step containing complete information; denotes the sequence of previously generated reasoning steps; LLM( ) denotes the use of an LLM multi-modal large model to generate the current reasoning step based on the modality fusion matrix and the sequence of previous reasoning steps ; when i = 1, the LLM multi-modal large model .

[0121] The structured prompt template prompt set by the technical personnel contains reasoning generation instructions, format specification requirements and quality evaluation standards. The structured prompt template prompt first provides the current modality fusion matrix and the existing reasoning steps as context, then explicitly specify the output format as a standardized JSON structure, requiring the LLM multimodal large model to evaluate the confidence of this step and identify the logical dependency relationship with the previous step when generating the reasoning content. Therefore, the reasoning step in the present application is structured representation format {i, Content( ), CL( ), DL( )}; where i is the reasoning step identifier, used to uniquely identify the sequence symbol of the current reasoning step in the current thought chain Chain; Content( ) represents the reasoning content of the current reasoning step, which is a natural language description; CL( ) represents the confidence score of the current reasoning step, reflecting the degree of certainty of the LLM multimodal large model about the accuracy of the current reasoning step; DL( ) represents the logical dependency relationship list of the current reasoning step, indicating the sequence of the previous reasoning steps that the current reasoning step depends on.

[0122] It should be noted that, as Figure 2 shown, it is a structural diagram of a thought chain: where i = 6, y that is, after the reasoning step , three different reasoning steps , and are obtained at the same time, and then after the reasoning step , the reasoning step is obtained. That is, for the current reasoning step , the sequence of the previous reasoning steps that have been generated is , but DL( ) = [ ].

[0123] The reasoning decoder is used to generate a thought chain based on a modal fusion matrix, which is the prior art and will not be described here.

[0124] The following sub-steps are also included in S2:

[0125] S21, constructing a reasoning dependency graph based on the thought chain : ; where N represents a set of nodes, each node corresponding to a reasoning step in the current thought chain; represents a set of edges, which are used to connect the nodes corresponding to adjacent reasoning steps in the logical dependency relationship list.

[0126] S22, constructing a reasoning dependency graph The nodes meeting one or more conditions are selected as the checkpoints: condition one, the confidence score is lower than a first score threshold; condition two, the out-degree is greater than a second score threshold; and condition three, the in-degree is greater than a third score threshold.

[0127] In this embodiment, the first score threshold is 0.7, the second score threshold is 2, and the third score threshold is 2.

[0128] In the present application, the nodes screened out by condition one are potential problem nodes; the nodes screened out by condition two are decision branch points, and the reasoning quality of the decision branch points directly affects multiple subsequent reasoning paths; and the nodes screened out by condition three are information aggregation nodes, and the information aggregation nodes need to integrate multiple previous reasoning results and are prone to information conflicts.

[0129] Inference dependency graph Some nodes in the inference dependency graph may meet two or three conditions at the same time.

[0130] The S3 further includes the following sub-steps:

[0131] In S31, the semantic consistency average value between the reasoning content corresponding to the current checkpoint and the reasoning content in the sequence of previous reasoning steps on which the current checkpoint depends is recorded as the consistency score of the current checkpoint; the degree of coincidence between the reasoning content corresponding to the current checkpoint and the fact is recorded as the factuality score of the current checkpoint; and the logical rationality between the reasoning content corresponding to the current checkpoint and the reasoning content corresponding to all nodes in the current thought chain Chain is recorded as the logicality score of the current checkpoint.

[0132] In S32, the current checkpoint is recorded as , the checkpoint corresponds to the reasoning step , the error possibility score of the current checkpoint is calculated as follows:

[0133]

[0134] wherein , and are respectively a first weight parameter, a second weight parameter and a third weight parameter, and ; , and are respectively the consistency score, the factuality score and the logicality score of the current checkpoint ; , and have a value range of ;​​

[0135] When the error possibility score exceeds the fourth score threshold, the corresponding checkpoint is marked as an error node; after marking all error nodes in the current thought chain Chain, S4 is executed; if there is no error node in the current thought chain, the current thought chain Chain is output.

[0136] The consistency score of the checkpoint can be calculated using the cosine similarity formula of the semantic vector corresponding to the reasoning content and then averaged; the higher the consistency score of the checkpoint, the higher the semantic correlation between the current checkpoint and each reasoning step in the sequence of previous reasoning steps it depends on. The factuality score of the checkpoint is obtained by pre-training a neural network model to score the degree of conformity of the reasoning content corresponding to the checkpoint with the facts; for example, the reasoning content corresponding to the checkpoint is "what diseases does a certain symptom correspond to", and the fact is authoritative medical knowledge; the higher the factuality score of the checkpoint, the smaller the error or knowledge bias between the reasoning content corresponding to the current checkpoint and the fact. The logicality score of the checkpoint is obtained by pre-training a neural network model based on all nodes in the current thought chain Chain to analyze the sufficiency of the reasoning premise, the logical validity of the reasoning process, and the rationality of the reasoning conclusion of the current checkpoint; the higher the logicality score of the checkpoint, the higher / more correct the logical reasoning specification of the current checkpoint relative to other nodes in the current thought chain Chain.

[0137] In this embodiment, the fourth score threshold is 0.6.

[0138] The following sub-steps are also included in S4:

[0139] S41, calculate the visual modality fusion matrix and the text modality fusion matrix :

[0140] ;

[0141] .

[0142] S42, based on the visual modality fusion matrix and the text modality fusion matrix respectively, use a pre-trained multi-modal large model as a reasoning decoder to generate corresponding visual thought chain Chain´ and text thought chain Chain´´.

[0143] S43, select the first error node in the current thought chain Chain, denoted as ; obtain the corresponding nodes ​error likelihood score of the error node ´ and ´´; the inference step corresponding to the foremost error node is also the earliest inference step among all error nodes in the current thought chain Chain.

[0144] S44, respectively calculate the absolute contribution intensity of the visual modality and the text modality to the error node in the current thought chain Chain and :

[0145] ; .

[0146] S45, respectively calculate the relative contribution intensity of the visual modality and the text modality to the error node in the current thought chain Chain and :

[0147] ; .

[0148] In this embodiment, , , and , the value range is .

[0149] The higher the relative contribution intensity value is, the greater the contribution of the corresponding modality to the reasoning error of the error node in the current thought chain Chain . and who is greater, then the corresponding modality is the main error source of the error node .

[0150] Optionally, generate an error attribution report of the current thought chain Chain, and the error attribution report includes: the inference step corresponding to all error nodes (i.e. error reasoning step), the error likelihood score corresponding to each error node, the sequence of the preceding inference step on which each error reasoning step depends, and the relative contribution intensity of various modalities to the foremost error node in the current thought chain.

[0151] In S5, the following sub-steps are further included:

[0152] S51, based on the error likelihood score , calculate the weight adjustment coefficient β:

[0153] ;

[0154] wherein λ is a sensitivity parameter;​​ is the upper limit of the weight adjustment coefficient; min(·, ·) represents a minimum function. The upper limit of the weight adjustment coefficient is used to limit the change of the weight adjustment coefficient to prevent the weight adjustment coefficient from changing too drastically.

[0155] In this embodiment, λ = 0.5, = 0.3.

[0156] S52, based on the current weight adjustment coefficient β, update the modal fusion weights:

[0157] ; ;

[0158] wherein, and respectively represent the updated text modal fusion weight vector and the visual modal fusion weight vector.

[0159] Optionally, S52 is replaced by S52':

[0160] S52', based on the current weight adjustment coefficient β, update the modal fusion weights:

[0161] ; ;

[0162] wherein, and respectively represent the updated text modal fusion weight vector and the visual modal fusion weight vector. represents the 2-norm.

[0163] Compared with S52, S52' is subjected to L2 normalization, further ensuring the stability of the modal fusion weights in the updating process.

[0164] After verification by the skilled person, for the same initial information set, the accuracy of the reasoning result obtained by the reasoning method of the present application is much higher than that of the prior art, and the stability of the accuracy is higher.

[0165] The skilled person has also verified that, for the same initial information set, before the final output thought chain, the two thought chains obtained in sequence in the process, the latter obtained thought chain is reduced in terms of both the number of error nodes and the error possibility score of the corresponding error nodes.

[0166] The specific experimental content and experimental data are as follows:

[0167] ① Experimental environment and data set configuration:

[0168] The hardware environment is NVIDIA A100 GPU (80GB memory) x 8, Intel Xeon Gold 6330 processor, 512GB memory; the software environment uses Python 3.10, PyTorch 2.2, CUDA 12.4. The model configuration selects BERT-base as the text encoder, ViT-Base-16 as the visual encoder, and Qwen2.5-VL as the multi-modal reasoning generator.

[0169] The data set selection covers multiple types to verify the universality of the method, and selects VQA v2, GQA and OK-VQA three data sets. (1) VQA v2.0 data set, containing about 1 million question and answer pairs, 10 artificial answers per question, covering open visual question answering in daily scenes, mainly testing basic visual understanding and common sense reasoning; (2) GQA data set, containing about 22 million question and answer pairs, emphasizing the ability of combined reasoning and scene graph understanding, requiring multi-step logical reasoning chain; (3) OK-VQA data set, containing 1.4 million visual question answering samples requiring external knowledge, evaluating knowledge-enhanced reasoning ability.

[0170] ②Comparison with prior art:

[0171] (1) Prior art one: standard Chain-of-Thought (CoT), basic thought chain prompting method, providing the most basic performance benchmark; (2) Prior art two: Multimodal-CoT, two-stage generation of reasoning basis and question and answer multimodal CoT framework, representing the mainstream method of current multi-modal thought chain; (3) Prior art three: Self-Consistency, a method for improving consistency by generating multiple reasoning paths; (4) Prior art four: Verify-and-Edit, verification-enhanced thought chain method; (5) Prior art five: KAT (Knowledge Augmented Transformer), which retrieves relevant knowledge from an external knowledge base and integrates it into the reasoning process, designed specifically for knowledge-enhanced visual question answering tasks.

[0172] ③Evaluation index system:

[0173] A multi-dimensional evaluation index system is adopted to comprehensively measure the performance of the method.

[0174] For the main performance indicators, VQA v2.0 uses the official VQA accuracy; GQA uses the precision matching accuracy; OK-VQA uses the official VQA accuracy.

[0175] For the inference quality indicators, to quantify the semantic consistency and structural integrity of the reasoning chain, the BERT Score-F1, ROUGE-L, and average reasoning step number are reported: (1) BERT Score F1 value, to evaluate the semantic consistency of the reasoning chain with the high-quality reference template; (2) ROUGE-L score, to evaluate the integrity and coherence of the reasoning steps; (3) average reasoning step number, to count the average length of the reasoning chain and reflect the reasoning complexity.

[0176] For the efficiency indicators, the following indicators are reported: (1) average reasoning time, the time cost of completing a single inference; (2) relative computing cost, the proportion of computing overhead relative to the standard CoT.

[0177] (5) Overall performance comparison experiment results

[0178] As shown in Table 1, the comprehensive evaluation results on the three data sets show that the present method achieves stable performance improvement: the accuracy of the reasoning results obtained by the present reasoning method far exceeds that of the prior art.

[0179] Table 1 Overall performance comparison results

[0180]

[0181] The experimental results show that the present method achieves a weighted average accuracy of 65.1%, achieving the best performance on all three data sets. On VQA v2.0, it achieves 73.6%, on GQA, it achieves 69.2%, showing an advantage in combined reasoning tasks, and on OK-VQA, it achieves 52.4%, surpassing specialized knowledge enhancement methods. In terms of efficiency, the reasoning time is 5.2 seconds, and the computing cost is 1.6 times, achieving a good balance between precision and efficiency.

[0182] (6) Inference quality evaluation results

[0183] To verify the advantages of the present method in generating high-quality reasoning processes, the quality of the reasoning chain of 1000 samples in the validation set is analyzed. The semantic consistency, integrity, and logical coherence of the reasoning chain are evaluated through multi-dimensional indicators to verify the improvement effect of the error attribution mechanism on the reasoning quality. The inference quality evaluation results are shown in Table 2.

[0184] Table 2 Inference chain quality evaluation results

[0185]

[0186] The inference quality evaluation result shows that the method improves the effect in all evaluation dimensions. The BERT Score reaches 0.812, which is improved by 0.091 compared with the standard CoT, indicating that the generated reasoning chain is more accurate and coherent in the semantic level. The ROUGE-L score is 0.673, which is improved by about 0.05 compared with the baseline method on average, reflecting the improvement of the completeness and information coverage of the reasoning steps. The average reasoning step number is 4.1 steps, which is moderately increased, indicating that the reasoning process is more detailed and complete, which helps to improve the explainability. Overall, the method improves the quality and credibility of the multi-modal reasoning through error attribution and adaptive correction mechanism.

[0187] The experimental results show that the multi-modal thought chain method based on error attribution improves in the three dimensions of accuracy, reasoning quality and computing efficiency. Through the comparative experiment and the reasoning quality evaluation, it is proved that the methodological innovation of improving multi-modal reasoning from the perspective of error analysis has important value.

[0188] The reasoning method of the application uses the initial modal fusion weight to obtain an initial thought chain first, then performs error checking on the thought chain, if there is no error checking node in the current thought chain, the current thought chain is directly output, otherwise the modal fusion weight is adjusted using the current thought chain, and a new thought chain is generated using the new modal fusion weight. That is, the reasoning method of the application can obtain an accurate thought chain.

[0189] The application is not to output the thought chain as the final output, but to perform error checking on the obtained thought chain (i.e. to score the error possibility of the check point in the thought chain), only the thought chain without error node will be output, which ensures that the thought chain output by the application has high accuracy. The thought chain with error node found in error checking will adjust the modal fusion weight based on the relative contribution intensity of different modalities to the error node in the current thought chain, and then generate a new thought chain using the new modal fusion weight, which further reduces the error node and the error degree, that is, the application can automatically and step by step place the error reasoning step in the initial thought chain to the correct path through the closed loop cycle. That is, the reasoning method of the application not only improves the accuracy of the reasoning result, but also maintains the stability of the accurate reasoning result.

[0190] The application uses whether there is an error node in the thought chain as direct feedback, and according to the relative contribution intensity of the error node in the thought chain, the whole process can complete the targeted adjustment of the modal fusion weight used for different initial information sets without the intervention of technical personnel, which improves the accuracy of the final reasoning result while greatly saving the labor cost.

[0191] Moreover, we also said in the background art that the inappropriate modal fusion weight can directly lead to a self-error in a reasoning step, and the self-error can cause the subsequent reasoning steps to be judged as error nodes due to error accumulation. Therefore, in the reasoning method of the present application, in calculating the relative contribution intensity of different modalities to the error node, we select the most front error node in the current thought chain Chain , for the following reasons:

[0192] ① While ensuring the reduction in the number of error nodes and the reduction in the error degree of the new thought chain, the calculation amount when adjusting the modal fusion weight can be reduced (there is no need to calculate the relative contribution intensity of different modalities to all error nodes in the current thought chain Chain).

[0193] ② The reasons for the existence of error nodes in the thought chain interfere with each other. Some error nodes are not only caused by error accumulation due to previous reasoning steps, but also caused by inappropriate modal fusion weights. However, in a thought chain, the most front error node is usually caused by inappropriate modal fusion weights. Therefore, the present application adjusts the modal fusion weight by selecting a suitable error node to make the correction process of reasoning errors more scientific. This science also reflects that the error accumulation of the error node in the current thought chain can be removed from the new thought chain, ensuring that the most front error node in the newly generated thought chain is still caused by inappropriate modal fusion weights.

[0194] Therefore, the reasoning method of the present application, while conforming to the cognition that error nodes in a reasoning chain cannot be corrected all at once due to mutual interference, can obtain a thought chain with high accuracy through several corrections, thereby improving the correction efficiency of reasoning errors. The present application corrects the error of the thought chain based on error attribution; wherein the "error attribution" refers to the error node selected to adjust the modal fusion weight. Each time a new thought chain is generated, it is in the process of error correction.

[0195] If the initial thought chain contains error nodes, the error nodes in the initial thought chain are the most. It can be considered that for the initial thought chain, as long as the new modal fusion weight value falls within a relatively wide reasonable range, the error nodes in the new thought chain can be reduced, and the new modal fusion weight calculated by us is within this reasonable range. However, during the subsequent adjustment of the modal fusion weight, this reasonable range is continuously reduced, so the modal fusion weight calculated by the present application is more precise than the previous modal fusion weight.

[0196] In the reasoning method of the present application, the modal fusion matrix is used to efficiently fuse information of different modalities.

[0197] In the reasoning method of the present application, instead of directly taking all nodes as checkpoints, the checkpoints are screened based on three conditions, which not only reduces the amount of calculation of the error possibility score of the checkpoints, but also improves the efficiency of obtaining the thought chain while ensuring accuracy.

[0198] The error possibility score of the checkpoints is obtained based on consistency score, factuality score and logicality score, which is more reasonable and has a lower misjudgment rate.

[0199] The reasoning method of the present application is suitable for efficient and high-accuracy reasoning of initial information sets in various fields, and has high universality.

[0200] The present application also provides a reasoning system, comprising:

[0201] a thought chain generation module, a checkpoint selection module, a scoring module, a calculation module and an updating module;

[0202] The thought chain generation module is used to generate a thought chain after modal fusion of each initial information in the initial information set based on the current modal fusion weight, and then send the thought chain to the checkpoint selection module;

[0203] The checkpoint selection module selects a checkpoint in the reasoning dependency graph after constructing the reasoning dependency graph based on the thought chain, and then sends the dependency graph and the checkpoint to the scoring module;

[0204] The scoring module scores the error possibility of each checkpoint, and if the error possibility scores of all checkpoints in the current thought chain are below a set threshold, the scoring module outputs the current thought chain; otherwise, the scoring module marks the checkpoint with an error possibility score exceeding the set threshold as an error node and sends it to the calculation module;

[0205] The calculation module is used to calculate the relative contribution intensity of different modalities to the error node and send it to the updating module;

[0206] The updating module updates the modal fusion weight based on the relative contribution intensity and sends it to the thought chain generation module.

[0207] Each module is programmed or configured to perform the steps of a multi-modal reasoning method based on error attribution as described above.

[0208] The technical, shape and structure parts not described in detail in the present application are well-known technologies. It should be noted that the above is only a preferred embodiment of the present application and does not limit the present application. The components or steps in the embodiments of the present application can be decomposed and / or recombined, which should be regarded as equivalent solutions of the present application and fall within the protection scope of the present application.

Claims

1. A multi-modal reasoning method based on error attribution, characterized in that, The method comprises the following steps: S1, generating a thought chain after modal fusion of each initial information in an initial information set based on a current modal fusion weight; the initial information set comprises text type initial information and image type initial information; S2, selecting a checkpoint in a reasoning dependency graph constructed based on the thought chain; S3, scoring error possibility of each checkpoint based on consistency, factuality and logicality, and outputting the current thought chain if the error possibility scores of all the checkpoints in the current thought chain are below a set threshold; Otherwise, marking the checkpoint with an error possibility score exceeding the set threshold as an error node and performing S4; S4, calculating relative contribution intensity of different modalities to the error node; S5, updating the modal fusion weight based on the relative contribution intensity, and returning to S1 to generate a new thought chain using the updated modal fusion weight.

2. The method of claim 1, wherein, In S1, the following sub-steps are further included: S11, extracting a feature matrix of each initial information in the initial information set; S12, mapping all the feature matrices to the same dimensional space for dimensional alignment; S13, obtaining a modal fusion matrix after modal fusion of the dimensionally aligned feature matrices based on a bidirectional cross-attention mechanism and the modal fusion weight; S14, generating a thought chain containing a reasoning step sequence using a reasoning decoder based on the modal fusion matrix.

3. The multi-modal reasoning method based on error attribution according to claim 2, characterized in that: In S11, the following is also included: extracting initial information of semantic features of the text type ; extracting initial information of image features of the image type ; extracting initial information of semantic features of the text type ; extracting initial information of image features of the image type ; In S12, the following is also included: mapping the image feature matrix into a visual feature matrix in a text feature space , wherein = ; is a learnable linear projection weight matrix; is a model bias vector for extracting a semantic feature matrix . In S13, the following is further included: based on the current visual modality fusion weight vector and the text modality fusion weight vector using a bidirectional cross-attention mechanism to generate a modality fusion matrix : ； ； ； ； ； wherein, denotes the scaled dot-product attention mechanism computation result; denotes the computation using function; , and denote the text query matrix, the text key matrix and the text value matrix, respectively; , and denote the visual query matrix, the visual key matrix and the visual value matrix, respectively; , and are the first projection matrix, the second projection matrix and the third projection matrix, respectively; is the matrix of the learnable projection matrix having the optimal query mapping relationship after training and learning, is the matrix of the learnable projection matrix having the optimal key mapping relationship after training and learning, is the matrix of the learnable projection matrix having the optimal value mapping relationship after training and learning; denotes the transpose of ; denotes the transpose of ; denotes the dimension of the attention head, wherein denotes the feature dimension of the model extracting the semantic feature matrix , and h denotes the number of attention heads.

4. The method of claim 3, wherein, In S14, the following is further included: Based on modal fusion matrix , using a pre-trained multi-modal large model as a reasoning decoder, a thought chain Chain containing a sequence of reasoning steps is generated: ； ； in, This represents the i-th reasoning step in the current thought chain, where 1 ≤ i ≤ n and i and n are both positive integers. This represents a structured suggestion template; This represents the sequence of preceding inference steps that have been generated so far; LLM ( This indicates the use of an LLM multimodal large model, based on the modality fusion matrix. and the sequence of preceding reasoning steps To generate the current inference step When i=1, LLM multimodal large model Content( () indicates the reasoning steps The reasoning content; CL( () indicates the reasoning steps Confidence score; DL ( () indicates the reasoning steps A list of logical dependencies.

5. The method of claim 2, wherein, In S2, the following sub-steps are further included: S21, constructing a reasoning dependency graph based on the thought chain : ; wherein N represents a node set, each node corresponding to a reasoning step in the current thought chain; E represents an edge set, and the edge is used to connect nodes corresponding to adjacent reasoning steps in the list of logical dependency relationships. S22, selecting a reasoning dependency graph The node that meets one or more conditions is selected as a checkpoint: condition one, the confidence score is lower than a first score threshold; condition two, the out-degree is greater than a second score threshold; and condition three, the in-degree is greater than a third score threshold.

6. The method of claim 5, wherein, In S3, the following sub-steps are further included: S31, taking a semantic consistency average value between reasoning content corresponding to the current checkpoint and each reasoning content in a previous reasoning step sequence on which the current checkpoint depends as a consistency score of the current checkpoint; taking a degree of coincidence between the reasoning content corresponding to the current checkpoint and a fact as a factuality score of the current checkpoint; and taking a logical rationality between the reasoning content corresponding to the current checkpoint and reasoning content corresponding to all nodes in the current thought chain Chain as a logicality score of the current checkpoint; S32, record the current checkpoint as , checkpoint The corresponding inference step is Then calculate the error possibility score of the current checkpoint :​ ； in, , and These are the first weight parameter, the second weight parameter, and the third weight parameter, respectively. and ; , and These are the current checkpoints. Consistency score, factual score, and logical score; When the error possibility score When the error possibility score exceeds the fourth score threshold, the corresponding checkpoint is marked as an error node; after marking all error nodes in the current thought chain Chain, S4 is executed; if there is no error node in the current thought chain, the current thought chain Chain is output.

7. The method of claim 6, wherein, In S4, the following sub-steps are further included: S41, calculating a visual modality fusion matrix and a text modality fusion matrix : ； ； ； ； wherein, denotes the scaled dot-product attention mechanism computation result; denotes the computation using function; , and denote the text query matrix, the text key matrix and the text value matrix, respectively; , and denote the visual query matrix, the visual key matrix and the visual value matrix, respectively; denotes the transpose of ; denotes the transpose of ; denotes the dimension of the attention head, wherein denotes the feature dimension of the model extracting the semantic feature matrix , h denotes the number of attention heads; denotes the current visual modality fusion weight vector; denotes the current text modality fusion weight vector; S42, based on the visual modality fusion matrix Text modality fusion matrix Using a pre-trained multimodal large model as an inference decoder, the corresponding visual thought chain Chain´ and text thought chain Chain´´ are generated. S43, select the error node at the most front of the reasoning steps in the current thinking chain Chain, and denote the error node as ; respectively obtain the error possibility scores of the corresponding nodes and ´ and ´´ in the visual thinking chain Chain´ and the text thinking chain Chain´´; S44, respectively calculate the absolute contribution strength of the visual modality and the text modality to the error node in the current thinking chain Chain and :​ ； ； S45, respectively calculate the relative contribution strength of the visual modality and the text modality to the error node in the current thinking chain Chain and :​ ； 。 8. The method of claim 7, wherein, In S5, the following sub-steps are further included: S51, based on the error likelihood score , a weight adjustment coefficient β is calculated: ; where λ is a sensitivity parameter; is a weight adjustment coefficient upper limit; min( , ) denotes a minimum function; S52, updating the modal fusion weight based on a current weight adjustment coefficient β: ； ； wherein, and respectively represent the updated text modality fusion weight vector and the visual modality fusion weight vector.

9. The error attribution based multi-modal reasoning method of claim 8, wherein, Replace S52 with S52´: S52´, updating the modal fusion weight based on a current weight adjustment coefficient β: ； ； wherein, and respectively represent the updated text modality fusion weight vector and the visual modality fusion weight vector; denotes the 2-norm.

10. An inference system, comprising: Comprise: a thought chain generation module, a checkpoint selection module, a scoring module, a calculation module and an updating module; The thought chain generation module is used to generate a thought chain after modal fusion of each initial information in an initial information set based on a current modal fusion weight, and then send the thought chain to the checkpoint selection module; After the checkpoint selection module selects the checkpoints in the reasoning dependency graph based on the thought chain, the dependency graph and the checkpoints are sent to the scoring module; the scoring module scores the error likelihood of each checkpoint, if the error likelihood of all checkpoints in the current thought chain is below the set threshold, the scoring module outputs the current thought chain; otherwise, the scoring module marks the checkpoint whose error likelihood exceeds the set threshold as an error node and sends it to the calculation module; the calculation module is used to calculate the relative contribution intensity of different modalities to the error node and send it to the update module; The update module updates the modal fusion weight based on the relative contribution intensity and sends it to the thought chain generation module; Each module is programmed or configured to perform the steps of the error attribution based multi-modal reasoning method as claimed in any one of claims 1-9.

Images