An iterative contrastive chain-of-thought reasoning method based on MLLMs

Abstract

The present application relates to the technical field of artificial intelligence, and more particularly to an iterative contrastive thinking chain reasoning method based on MLLMs. The method comprises: under a zero-shot setting, obtaining multi-modal data input, independently analyzing problems and images through a decision-making agent module, constructing a reasoning process and proposing sub-questions, and constructing a clear perception decision; answering sub-questions in turn through a reasoning agent module to supplement the decision, and then constructing a collaborative reasoning path; evaluating the agent module to cross-verify between reasoning paths, if an agreement is reached, the answer will be accepted; otherwise, iteration self-optimization is triggered. Under the zero-shot setting, the present application constructs a clear perception decision through visual analysis and sub-questions, reduces hallucinations through cross-verification thinking, iteration self-optimization and collaborative consistency evaluation, enables the model to detect and correct reasoning bias in the reasoning process, and improves the multi-modal reasoning capability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an iterative comparative thought chain reasoning method based on MLLMs. Background Technology

[0002] Multimodal Large Language Models (MLLMs) have demonstrated powerful recognition and analysis capabilities across various visual and language tasks, particularly in visual reasoning. Early methods typically combined LLMs with external visual modules, unifying inputs from different modalities into a single text modality. However, converting visual information to a text modality often resulted in the loss of crucial information. This approach subsequently evolved into the more widely adopted two-stage decision-inference architecture, which separates reasoning generation from answer inference. In this stage, multimodal CoT methods primarily focus on improving the quality of the reasoning process. To address the inherent limitations of LLM-based multimodal CoT methods, which often rely on indirect visual-to-text conversion and lack end-to-end multimodal reasoning, recent multimodal CoT methods leverage the powerful cross-modal understanding and reasoning capabilities of MLLMs to perform direct reasoning within a multimodal context, alleviating long-standing challenges in visual-to-text integration. This includes extracting structured scene graphs from images and designing specialized expert modules to assign visual subtasks.

[0003] However, current multimodal thought chain (CoT) methods still have several limitations. On the one hand, due to contextual blind spots in complex reasoning scenarios, the reasoning process is prone to illusions under the current mainstream decision-reasoning architecture, which may lead to a single critical error destroying the entire reasoning chain. Although some methods attempt to improve reasoning quality by utilizing visual information, these methods generally lack reasoning evaluation mechanisms and iterative optimization processes. On the other hand, some methods still rely primarily on textual cues. When generating reasoning in the decision-making stage, visual information is not fully utilized or is only considered supplementary, resulting in the omission of crucial visual evidence. Furthermore, dynamic reasoning optimization cannot be achieved in zero-sample settings. This invention proposes an iterative comparative thought chain reasoning method based on MLLMs. Summary of the Invention

[0004] Therefore, it is necessary for the present invention to provide an iterative comparative thought chain reasoning method based on MLLMs to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above objectives, an iterative comparative thought chain reasoning method based on MLLMs includes the following steps:

[0006] Step S1: Under zero-sample settings, multimodal data input is obtained through the perception decision unit, and the decision intelligent agent module independently analyzes the problem and image, constructs the reasoning process and proposes sub-problems, and constructs a clear perception decision, that is, formulates parallel decision;

[0007] Step S2: The reasoning agent module answers the sub-questions sequentially to support decision-making, and then constructs a collaborative reasoning path based on the corresponding decisions;

[0008] Step S3: Evaluate the agent module to perform cross-validation between inference paths. If a consensus is reached, the answer will be accepted; otherwise, iterative self-optimization is triggered to form a closed-loop inference process.

[0009] Furthermore, in step S1, the decision-making agent module independently analyzes the problem and image features to formulate parallel decisions, including the following steps:

[0010] Step S11: The decision-making agent module includes multiple decision-making agent units, which perform structured analysis on multimodal data and task problems; each decision-making agent unit can independently analyze problems and images, and generate reasoning arguments by performing context analysis.

[0011] Step S12: The decision-making agent module performs structured analysis on the image. Based on the generated rationale and problem analysis, the decision-making agent module raises a question to obtain the theoretical or conceptual knowledge required to solve the task.

[0012] Step S13: The decision-making agent module proposes specific sub-problems based on reasoning and visual analysis in order to flexibly and comprehensively obtain the visual evidence required for reasoning.

[0013] Furthermore, in step S1, the reasoning process is constructed and sub-problems are proposed, including guiding the model to explicitly analyze visual information through implicit design hints or system flow, proposing questions by explicitly analyzing visual information, and decomposing complex problems into a series of sub-problems for gradual solution; the sub-problems may involve identifying object attributes or analyzing spatial relationships, and the explicit and implicit information of the sub-problems are crucial for solving the task. The visual sub-problems are defined by formula (6).

[0014] ;

[0015] in, Activate the decision-making agent Generate the visual subproblem.

[0016] Furthermore, in step S2, the reasoning agent module includes an answering unit and a comprehensive reasoning unit. The answering unit answers sub-questions sequentially to support perceptual decision-making. The specific steps are as follows:

[0017] Step S21: The reasoning agent module retrieves the problem analysis results and the set of conceptual sub-problems, and the answering unit answers the conceptual sub-problems in sequence, with the answer to the previous sub-problem serving as key supporting evidence for the current sub-problem;

[0018] Step S22: The answering unit answers the set of visual sub-questions, wherein the visual sub-questions and their corresponding answers are independent of each other and do not depend on the answer of the previous sub-question;

[0019] Step S23: The integrated reasoning unit integrates the rationale and all relevant information in the plan, including the answers to conceptual and visual sub-questions. Based on these inputs, it constructs a coherent step-by-step reasoning chain to complete the entire original question-answering process.

[0020] Furthermore, the entire original question-and-answer process in step S23 is formalized as follows:

[0021] ;

[0022] in Indicated by the reasoner The executed inference function, response Including the constructed inference chain and the final answer The complete reasoning path is represented as ,in This indicates different reasoning paths.

[0023] Furthermore, the evaluation agent module in step S3 is a consistency evaluation module, which includes cross-validation thinking and iterative self-optimization. The evaluation agent module independently evaluates each component of the main inference path and its inference chain, and collaborates and interacts between parallel inference paths to further identify potential errors and extract reliable supporting evidence.

[0024] Furthermore, the evaluation agent module also includes integrating sub-questions and answers from the auxiliary reasoning path into the main reasoning path, and prompting the evaluator to perform cross-validation thinking; the cross-validation feature between reasoning paths includes the following steps:

[0025] Step S31: If the two answers are the same in the two reasoning paths, the evaluation agent module performs cross-validation on one of the reasoning paths; otherwise, cross-validation is performed independently on the two paths respectively.

[0026] Step S32: If the results of cross-validation converge to a consistent conclusion, the result is considered reliable and accepted as the final answer; otherwise, the evaluation agent module enters the iterative self-optimization stage for further optimization.

[0027] Furthermore, in the cross-validation process, if the results converge to a consistent conclusion, the result is considered reliable and accepted as the final answer; otherwise, the evaluator enters an iterative self-optimization phase for further optimization. The entire process is defined as follows:

[0028] ;

[0029] in, and This indicates that the reasoning comes from two different paths. and The answer, This indicates the corrected result after single-path cross-validation. and The corrected result was obtained through dual-path cross-validation. This indicates the final accepted answer after the consensus assessment.

[0030] Furthermore, the iterative self-optimization extracts consistency and difference points from the inference path without making an explicit judgment on correctness. The evaluation agent will consider the two inference paths... and As input, comparisons are made, and consistency and difference information between the two reasoning processes are extracted, including reasoning logic and visual evidence, and organized into an iterative guidance set. The iterative self-optimization process is formalized as follows:

[0031] ;

[0032] in, Indicates that the evaluation agent The executed iteration function The extracted consistency and difference information serves as a guide for iterative reasoning optimization.

[0033] Furthermore, the computer-readable storage medium stores a computer program for execution to implement the iterative comparative thought chain reasoning method based on MLLMs as described in any one of claims 1 to 9.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention presents a collaborative consistency multimodal thinking chain (Co-Co-CoT) framework based on a decision-reasoning-evaluation architecture (hereinafter referred to as the Co-Co-CoT framework, also known as an iterative comparative thinking chain reasoning method based on MLLMs). It optimizes the decision-making process and introduces multi-path reasoning and evaluation stages, significantly improving the quality and reliability of multimodal reasoning. Under a zero-sample setting, it enhances the reliability of reasoning through cross-validation thinking, iterative self-optimization, and collaborative consistency evaluation; it effectively mitigates illusions by constructing clear perceptual decisions through visual analysis and sub-problems, enabling the model to detect and correct reasoning biases during the reasoning process, thus improving multimodal reasoning capabilities.

[0036] On the one hand, the CoCo-CoT framework of this invention performs additional analysis, proactively raising questions about visual information to formulate structured perceptual decisions. Through cross-validation thinking and iterative self-optimization, it assesses the consistency and reliability between collaborative reasoning paths, effectively enhancing reasoning capabilities.

[0037] On the other hand, the CoCo-CoT framework of the present invention can alleviate the illusion problem in reasoning under zero-sample settings and achieve dynamic reasoning optimization in multimodal thought chains (CoT). Attached Figure Description

[0038] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0039] Figure 1 This is a schematic diagram of the steps of the iterative comparative thought chain reasoning method based on MLLMs of the present invention;

[0040] Figure 2 This is a complete flowchart of the closed-loop reasoning method of the iterative comparative thinking chain reasoning method based on MLLMs in this invention;

[0041] Figure 3 This is a successful example of the CoCo-CoT application on the ScienceQA dataset.

[0042] The present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] The technical method of this invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0044] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0045] It should be understood that although terms such as "A," "B," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used solely for the purpose of distinguishing different units. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0046] To achieve the above objectives, please refer to Figure 1 and Figure 2 This invention provides an iterative comparative thought chain reasoning method based on MLLMs, which includes the following steps:

[0047] Step S1: Acquire multimodal data input, independently analyze the problem and image through the decision-making agent module, construct the reasoning process and propose sub-problems, and construct a clear perception decision;

[0048] MLLMs integrate visual and textual inputs into a unified representation space to support coherent and comprehensive multimodal reasoning. Specifically, a trainable visual encoder... , by parameters Parameterization is used to encode images, whereas a fixed text encoder... Used to represent input text Both modalities are projected into a common semantic space, in which an autoregressive language model... , by parameters , and prompts Reasoning is conducted under guidance. The response process of MLLM is formalized as follows: (1)

[0049] ;

[0050] By perceiving multimodal inputs and formulating structured decisions, parallel decisions are made through a decision-making agent. These decisions serve as the basis for constructing collaborative reasoning paths, such as... Figure 2 As shown in the blue area, a decision-making agent (or decision-maker) is introduced. This intelligent agent simulates human cognitive mechanisms and involves formulating problem-solving strategies, among which... This provides decision-making cues. By performing contextual analysis, generating reasoning arguments, and asking questions, the agent obtains key conceptual and visual evidence to formulate structured perceptual decisions, thereby improving the quality of decision-making in visual reasoning tasks.

[0051] The detailed steps of the decision generation process are as follows. Problem analysis: The decision-maker first conducts a structured analysis of the task problem, which mainly includes: (1) identifying the type and domain of the problem; (2) explaining each term used in the problem to eliminate potential ambiguities. The purpose is to provide a clear semantic basis for understanding the task context and supporting the generation of reasoning. This process can be described by the following formula (2):

[0052] ;

[0053] in, Representation of decision-making intelligent agents A function used to identify and interpret the key concepts involved in the original problem . Output Including task types and a set of concept-explanation pairs Logical symbols Representing concepts It is semantically derived from the problem.

[0054] Visual analysis: The decision-making agent performs structured analysis on images. The analysis process includes: (1) identifying the type of image; (2) detecting specific features, regions, objects, or elements that may require special attention. This process provides a preliminary understanding of the image and lays the perceptual foundation for subsequent vision-related reasoning steps. The visual analysis process is formalized into formula (3): ;

[0055] in, ) is the visual analysis function of the decision-making agent 𝑫. Output Including image categories and a group of pairs It is used to identify specific features or elements in an image.

[0056] Rationale: This component describes the theoretical basis and reasoning strategies that constitute the core of the overall decision-making process. It also explains the motivations behind the design of conceptual and visual sub-problems, which can be formalized into formula (4):

[0057] ;

[0058] in, Representation of decision-making intelligent agents The rationale generation function.

[0059] Conceptual sub-problems: Based on the generated rationale and problem analysis, the decision agent raises questions to acquire the theoretical or conceptual knowledge needed to solve the task (e.g., "What does photosynthesis require?"). This process can be formalized as formula (5):

[0060] ;

[0061] in, Representation of decision-making intelligent agents The conceptual subproblem generating function.

[0062] Visual sub-problems: Based on reasoning and visual analysis, the decision-making agent proposes specific sub-problems to flexibly and comprehensively obtain the visual evidence required for reasoning.

[0063] These subproblems may involve identifying object properties or analyzing spatial relationships. Explicit and implicit information in the subproblems is crucial for solving the task. Visual subproblems are defined by Equation (6).

[0064] ;

[0065] in, Activate the decision-making agent Generate the visual subproblem.

[0066] Step S2: The reasoning agent module answers the sub-questions sequentially to support decision-making, and then constructs a collaborative reasoning path based on the corresponding decisions;

[0067] like Figure 2 As shown, the purple and red areas represent the answer and comprehensive reasoning parts of the reasoning agent module. Similar to the decision generation phase, a reasoning agent or reasoner is introduced. , represented as ,in This is a prompt used at this stage.

[0068] Answering conceptual sub-questions: The reasoner first retrieves the results of the problem analysis and the set of conceptual sub-questions. Then, it answers these sub-questions sequentially. In answering the sub-questions, the answer to the previous sub-question serves as key supporting evidence for the current sub-question. We formalize this process as formula (7):

[0069] ;

[0070] in, It is the final conceptual sub-problem - the set of answer pairs. Each answer By reasoning intelligent agent Through function Generate in sequence.

[0071] The condition is the current subproblem Problem Analysis and all previously generated answers

[0072] Answering visual sub-questions: The reasoner then begins to answer the set of visual sub-questions. Unlike conceptual sub-questions, visual sub-questions and their corresponding answers are independent of each other because visual sub-questions typically focus on independent perceptual elements and do not depend on the answer to the previous sub-question.

[0073] ;

[0074] in Indicated by the reasoner The visual response function is executed. After obtaining the answers to the conceptual and visual sub-questions, the complete decision is represented as follows:

[0075] Comprehensive Reasoning: At this stage, the reasoning agent integrates the rationale and all relevant information in the plan, including answers to conceptual and visual sub-questions. Based on these inputs, it constructs a coherent, step-by-step reasoning chain to complete the entire original question-answering process. This process is formalized as formula (9):

[0076] ;

[0077] in Indicated by the reasoner The inference function executed. Response. Including the constructed inference chain and the final answer Finally, the complete reasoning path is represented as follows: ,in These represent different reasoning paths, such as A and B.

[0078] This step answers conceptual and visual sub-questions to support perceptual decision-making; and based on the corresponding decisions, it constructs a multi-path collaborative reasoning chain to gradually guide the generation of task solutions.

[0079] Step S3: Evaluate the agent module to perform cross-validation between inference paths. If a consensus is reached, the answer will be accepted; otherwise, trigger iterative self-optimization.

[0080] As shown in the green and yellow areas of Figure 2, an evaluation agent or evaluator is introduced. ,in This is a prompt template designed for the evaluation phase.

[0081] First, the evaluator independently assesses each component of the main reasoning path and its reasoning chain, checking its factual correctness and reasoning rationality. This factual consistency check is formalized as follows: (10)

[0082] ;

[0083] in, Indicates that the evaluation agent The fact-checking function is executed. Output. This refers to a set of evidence that is considered credible.

[0084] Then, the evaluator collaborates between parallel reasoning paths to further identify potential errors and extract reliable supporting evidence. During cross-path interactions, when two reasoning paths exhibit consistency or contradiction on specific visual or factual elements, these become valuable clues for error identification. The evaluator analyzes and infers the reliability of these clues, thereby constructing a set of credible evidence. To this end, we integrate sub-questions and answers from auxiliary reasoning paths into the main reasoning path and prompt the evaluator to perform cross-validation thinking. We first construct a cross-validation context set. The following formula (11):

[0085] ;

[0086] in, These represent the sub-problems and their corresponding answers in the parallel inference path, respectively. Then, the evaluator performs cross-validation thinking, as shown in the following formula (12):

[0087] ;

[0088] in, It is evaluated by the intelligent agent The evaluation function is executed. This function outputs a graph. question and set As input, to obtain credible evidence. Output The filtered set of reliable supporting evidence includes the following, as shown in formula (13):

[0089] ;

[0090] Here, the final set of verified evidence is presented. Candidates that meet the following conditions composition:

[0091] (1) Present evidence Relevant to the given task;

[0092] (2) Present evidence It is consistent with other options in terms of logic and perception, avoiding conflicts in reasoning logic and visual interpretation; (3) This indicates that when the sub-problems and answers in the auxiliary reasoning path are integrated into the main reasoning process to solve the original problem. At that time, evidence It remains valid. This formalization ensures that only relevant, consistent, and cross-validated evidence is retained to support reliable and illusion-resistant reasoning.

[0093] Finally, the evaluators reconsidered the initial task and based on... and Perform stepwise reasoning. This process is formalized as follows, formula (14):

[0094] ;

[0095] Here, Indicates that the evaluation agent The final inference function executed. The refined answer. This reflects the conclusions after model correction under enhanced evidence alignment.

[0096] To evaluate the two reasoning paths and To ensure consistency between the two answers, the evaluator first compares their respective answers. If the two answers are the same, the evaluator performs cross-validation on one of the reasoning paths. Otherwise, independent cross-validation is performed on each path. If the results of the cross-validation converge to a consistent conclusion, the result is considered reliable and accepted as the final answer; otherwise, the evaluator enters the iterative self-optimization phase for further optimization. The entire process is defined by the following formula (15):

[0097] ;

[0098] in, and This indicates that the reasoning comes from two different paths. and The answer. This indicates the corrected result after single-path cross-validation. and The corrected result was obtained through dual-path cross-validation. This indicates the final accepted answer after the consistency assessment. Iterative self-optimization: During the optimization phase, the evaluator will consider the two reasoning paths... and As input, they are compared from both logical and visual information perspectives. The evaluator extracts consistency and discrepancy information from these two reasoning processes, including logical reasoning and visual evidence, and organizes it into an iterative guidance set. .

[0099] This set is then provided as supplementary clues to the decision-making and reasoning agent in the next round: consistent information is considered more reliable and is adopted critically, while differing information is marked as potential error points that need to be carefully checked and reviewed. The focus is on extracting points of consistency and difference from the reasoning path without making explicit judgments on correctness. This design prevents the introduction of potential illusions or evaluation errors, avoids their propagation in subsequent iterations, thereby avoiding the cascading of reasoning errors and achieving more stable and reliable iterative optimization. The iterative self-optimization process is formalized as follows: (16)

[0100] ;

[0101] in, Indicates that the evaluation agent The executed iteration function The extracted consistency and difference information serves as a guide for iterative reasoning optimization.

[0102] The evaluation agent module is designed to support dynamic consistency verification and feedback-driven optimization based on path differences. It mitigates the illusion problem in thought chain reasoning by employing cross-validation thinking, thereby improving its stability. This is achieved by introducing a consistency evaluation module executed by the evaluation agent for consistency verification, which includes cross-validation thinking and iterative self-optimization. This collaborative interaction enables the agent to reflect on and correct potential flaws, comprehensively identifying errors, contradictions, and visual recognition problems in the reasoning path.

[0103] The CoCo-CoT framework establishes a closed-loop reasoning process that integrates decision generation, reasoning, and evaluation. In this process, the perceptual decision generation module provides rich multimodal contextual information to support reasoning; the reasoning agent module constructs parallel and complete reasoning paths; and the evaluation agent module enhances the reliability of these reasoning paths through cross-validation and iterative self-optimization.

[0104] To fully understand the proposed CoCo-CoT framework, the entire inference workflow is summarized in Algorithm 1 and Algorithm 2.

[0105]

[0106] Algorithm 1 outlines the complete iterative optimization and collaborative consensus multimodal thought chain process, combining the decision-reasoning-evaluation architecture in the multi-agent and multi-path paradigm.

[0107] Algorithm 2 details the cross-validation thinking mechanism, which performs fact checking and cross-validation between inference paths, and demonstrates the detailed implementation of the cross-validation thinking process invoked in Algorithm 1.

[0108] Experiments were conducted on this invention, and the experimental process, data, and results are as follows:

[0109] 1 / Datasets: The proposed method was evaluated on two widely used visual reasoning benchmarks, which serve as standard evaluation tools for multimodal reasoning performance. The ScienceQA dataset is a large-scale multimodal benchmark designed to evaluate the ability to answer science questions across textual and visual modalities. It contains 21,208 multiple-choice questions divided into three main domains: Natural Sciences (NAT), Social Sciences (SOC), and Linguistic Sciences (LAN). The ScienceQA test set used in the experiments included 4,241 questions, of which 2,017 were accompanied by images. Accuracy was used as the primary evaluation metric. The MathVista dataset is a challenging multimodal benchmark that combines multiple mathematical reasoning tasks within a visual context, requiring fine-grained image understanding and combinatorial mathematical thinking. It includes multiple-choice questions and free-form answer questions, such as integer, floating-point, or list answers. In the experiments, a testmini subset of MathVista, containing 1,000 image-question pairs, was used.

[0110] Table 1. Main results (%) on the ScienceQA test set, where bolded items indicate the best results and underlined items indicate the second best methods. There are 8 question categories: Natural Sciences (NAT), Social Sciences (SOC), Language Sciences (LAN), Text Context (TXT), Image Context (IMG), No Context (NO), Grades 1-6 (G1-6), and Grades 7-12 (G7-12).

[0111]

[0112] 2 / Models: Experiments used Gemini 2.0-Flash and GPT-3.5 as baseline models to implement and evaluate our framework. Gemini 2.0-Flash is a state-of-the-art multimodal large language model capable of processing and reasoning in both textual and visual modalities. It was used to explore methods leveraging the advanced perceptual understanding capabilities of MLLMs to enhance their multimodal reasoning abilities in complex visual-language scenarios. GPT-3.5 is a robust text-only model used to validate the generalization ability of our CoCo-CoT framework across different model architectures. It was used to evaluate the impact of incorporating multimodal information on LLM reasoning capabilities in visual tasks. Based on these two models, we explored a more effective multimodal reasoning collaborative agent paradigm by utilizing coordination between MLLMs and interactions between MLLMs and LLMs.

[0113] 3 / Implementation Details: Two versions of CoCo-CoT were implemented using the official APIs of Gemini 2.0-Flash and GPT-3.5. For CoCo-CoT (Gemini), Gemini 2.0-Flash was used as the decision-making, reasoning, and evaluation agent, covering three stages. This configuration aims to fully exploit its multimodal perception and reasoning capabilities. For CoCo-CoT (GPT-3.5), GPT-3.5 was used as the decision-making, reasoning, and evaluation agent in the experiments. To accommodate multimodal requirements, following Cantor's experimental setup, we decoupled the visual analysis module from the decision generation stage: visual information is first processed by Gemini 2.0-Flash and then incorporated into the context of GPT-3.5 to support decision generation. During reasoning, Gemini 2.0-Flash acts as the reasoning agent for visual sub-problems, facilitating collaborative multimodal reasoning. To ensure fair evaluation of our framework, we avoided adding contextual examples or generating image captions to enhance context or visual information, especially for GPT-3.5. We compared these two versions with representative multimodal CoT benchmarks and fine-tuned models publicly available on ScienceQA and MathVista leaderboards.

[0114] 4 / Baseline Methods: To comprehensively evaluate the effectiveness and generalization ability of the CoCo-CoT framework of this invention, it is compared with representative prompt-based zero-shot multimodal thought chain (CoT) methods and robust fine-tuning models using the ScienceQA and MathVista datasets.

[0115] Table 2. Main results (%) on the testmini subset of MathVista, where bold entries indicate the best results and underlined entries indicate the second best methods. Inputs include the question (Q), the image (I), and the image caption. and OCR text Task types include Graphical Question Answering (FQA), Geometry Problem Solving (GPS), Mathematical Verbal Problems (MWP), Textbook Question Answering (TQA), and Visual Question Answering (VQA). Mathematical reasoning categories include Algebra (ALG), Arithmetic (ARI), Geometry (GEO), Logic (LOG), Common Sense Number Reasoning (NUM), Science (SCI), and Statistics (STA). "ALL" indicates overall accuracy.

[0116]

[0117] CCoT is the first thought chain method to introduce scene graphs as structured intermediate representations in multimodal reasoning. CCoT prompts the model to generate scene graphs in JSON format and then use them in the context, thus improving performance without training. In experiments, for the CCoT (GPT-3.5) variant, Gemini-2.0 was used to generate scene graphs because of its enhanced multimodal capabilities. CaCo-CoT: This is the first work to introduce causal consistency into thought chain reasoning. CaCo-CoT employs a multi-agent framework where a dedicated evaluator examines the reasoning chain from a counterfactual perspective, enhancing the credibility and robustness of the reasoning process. Cantor: It introduces a perception-decision architecture to multimodal thought chain reasoning to enhance the reasoning capabilities of MLLMs. The Cantor prompts the model to decompose complex problems into multiple sub-tasks and assigns four expert modules, each represented by an MLLM, responsible for handling these sub-tasks. Finally, Cantor integrates these results with the original information and performs reasoning.

[0118] Fine-tuning the Model: For the ScienceQA dataset, CoCo-CoT is compared to several high-performing fine-tuned models reported on the official leaderboard, including Multimodal-T-SciQ, MC-CoT, Multimodal-CoT, other CoT variants, and other well-known models. For the MathVista dataset, CoCo-CoT is compared to leading MLLMs on the official leaderboard, such as Vision-R1 and Ovis.

[0119] 5 / Experimental Results:

[0120] The main experimental results on the ScienceQA dataset are shown in Table 1. The CoCo-CoT (Gemini) framework achieved an accuracy of 96.36%, significantly outperforming all existing multimodal CoT methods, and even surpassing the best-performing fine-tuned model (Wang et al., 2024b) in the zero-shot setting. Furthermore, CoCo-CoT (GPT-3.5) improved accuracy by +5.86% compared to the standard CoT-hinted GPT-3.5 (84.17% vs. 78.31%), also outperforming other CoT methods. The experimental results demonstrate the superior performance and strong generalization ability of the CoCo-CoT framework of this invention. The adopted decision-reasoning evaluation architecture and collaborative reasoning mechanism enhance the structured reasoning capabilities of MLLMs and LLMs, improve the quality of multimodal reasoning, and reduce susceptibility to illusions. Notably, CoCo-CoT (Gemini) and CoCo-CoT (GPT-3.5) significantly outperform other CoT methods in the IMG category. For example, CoCo-CoT (GPT-3.5) achieved an accuracy of 83.49%, while Cantor (GPT-3.5) achieved 77.54%, and GPT-3.5 CoT achieved 67.92%. These results demonstrate that CoCo-CoT effectively enhances visual reasoning capabilities. This improvement can be attributed to the combination of structured perceptual decision-making and collaborative consistency assessment. Visual analysis and sub-question answers provide supporting evidence during decision generation, while cross-validation and iterative self-optimization help detect and correct errors in visual interpretation. Results for the MathVista dataset are shown in Table 2. The CoCo-CoT (Gemini) framework achieved an accuracy of 77.4%, surpassing the Gemini benchmark (72.2%). Meanwhile, CoCo-CoT (GPT-3.5) achieved a score of 68.0%, significantly exceeding the GPT-3.5 benchmark (33.2%). MathVista involves vision-driven mathematical reasoning, requiring fine-grained perception and precise symbolic understanding. The robust performance of CoCo-CoT in this invention demonstrates its ability to perform accurate and interpretable reasoning in complex visual mathematical contexts. In particular, the results of CoCo-CoT (GPT-3.5) show that the collaborative multimodal framework of this invention effectively combines the advanced visual cognition of MLLMs with the textual reasoning capabilities of LLMs, highlighting the advantages of our agent collaboration paradigm.

[0121] 6 / Ablation Experiment:

[0122] Table 3. Ablation studies of CoCo-CoT (Gemini) on the ScienceQA and MathVista datasets. All results are reported as accuracy (%).

[0123]

[0124] Ablation studies of different modules: Ablation experiments were conducted on the ScienceQA and MathVista datasets using Gemini 2.0-Flash to evaluate the contribution of each method in the CoCo-CoT framework of this invention. The results are summarized in Table 2, showing that each method significantly improves the overall performance of the framework. Removing visual perception methods (including visual analysis and sub-question generation) leads to a decrease in performance on the visual task. Specifically, the accuracy of the IMG category in ScienceQA decreased by 1.09%, and the accuracy of VQA in MathVista decreased by 1.11%, indicating that the method helps MLLMs acquire richer visual evidence, enhancing their understanding and reasoning abilities regarding images. Removing evaluation methods (including cross-validation thinking and iterative self-optimization) leads to the most significant performance decrease, with a 1.58% decrease in ScienceQA accuracy and a 3.9% decrease in MathVista accuracy. This highlights the importance of evaluation methods in maintaining overall performance. Based on the ablation results of cross-validation thinking, it is clear that this method plays a key role in improving inference accuracy by ensuring consistency between inference paths. Meanwhile, iterative self-optimization had a more significant impact on MathVista, with performance decreasing by 3.3% compared to ScienceQA. This indicates that MathVista's visual context is more complex, presenting a greater challenge and requiring more refined visual evidence and logical understanding to support deeper reasoning.

[0125] 7 / Quantitative Analysis:

[0126] Table 4. Impact of visual information intensity on the performance of the CoCo-CoT (Gemini) and Gemini baseline models on image-related subsets of the ScienceQA and MathVista datasets. All results are reported as accuracy (%).

[0127]

[0128] Exploring the Importance of Visual Information: We investigated and compared the impact of different visual information intensities on the overall performance of CoCo-CoT (Gemini) and its Gemini-2.0-Flash benchmark model on image-related subsets of the ScienceQA and MathVista datasets. The results on the ScienceQA and MathVista datasets are summarized in Table 4, from which several key findings were obtained: (1) Under all visual information conditions, the CoCo-CoT framework consistently outperforms the Gemini benchmark, demonstrating the overall advantage of our framework in utilizing multimodal information. (2) When we introduce coarse image captions, the performance of both CoCo-CoT (Gemini) and the Gemini benchmark decreases. This indicates that low-quality or blurry visual descriptions may introduce noise during inference. (3) Except under the coarse caption condition, the performance advantage of CoCo-CoT (Gemini) becomes more pronounced as the richness of visual information increases. This indicates that the CoCo-CoT framework of this invention can more effectively utilize detailed and comprehensive visual cues to enhance multimodal inference. (4) Overall, the performance of both CoCo-CoT (Gemini) and the Gemini benchmark improved as visual information became richer, confirming that detailed and complete visual input significantly improves inference quality in multimodal tasks. Iterative self-optimization analysis. Figure 3 shows a successive comparison of CoCo-CoT and the benchmark model on the same subset of questions in the ScienceQA and MathVista datasets, where CoCo-CoT solves these questions in each iteration. At all iteration levels, CoCo-CoT consistently outperformed the benchmark model, demonstrating the effectiveness of the overall collaborative and consistent multimodal CoT framework of this invention. Most questions could be answered correctly without additional optimization, i.e., CoCo-CoT achieved very high accuracy and solved most test cases in the first iteration. This indicates that the structured perceptual decision process and collaborative consistency assessment significantly enhance the initial inference quality. As the iteration depth increased, the number of questions requiring further iterative optimization decreased sharply (according to our complete iteration log), and the accuracy naturally decreased as the task difficulty increased. Only a small fraction of challenging cases required iterative optimization, and very few questions required more than two rounds of optimization. Even under these more challenging conditions, CoCo-CoT still significantly outperforms the baseline model, demonstrating that the iterative self-optimization mechanism effectively guides the model toward convergence and provides reliable improvements on complex multimodal reasoning tasks.

[0129] 8 / Visualization:

[0130] Figure 3 shows a successful case of CoCo-CoT on the ScienceQA dataset.

[0131] Experiments demonstrate some comparative examples of CoCo-CoT and Cantor on the ScienceQA dataset, using Gemini 2.0-Flash as the underlying model. The visualization results are shown in Figure 3. In Example 1, Cantor incorrectly identified the number of particles in solution A on the left as nine, while CoCo-CoT correctly identified eight. This indicates that CoCo-CoT provides more reliable performance in such visually demanding tasks. One possible reason is that CoCo-CoT employs a multi-path collaborative reasoning mechanism. When one reasoning path makes an incorrect visual judgment, CoCo-CoT can effectively detect and correct the error through consistency evaluation. In Example 2, the task involved analyzing nutrient levels in an ecosystem; Cantor misunderstood soil conditions, while the CoCo-CoT of this invention provided the correct judgment. This result shows that the CoCo-CoT of this invention effectively supports high-level understanding and analysis of complex visual scenes by MLLMs. This is attributed to CoCo-CoT's flexible perceptual decision-making, comprehensively providing the basic visual evidence needed for interpretation.

[0132] 9 / Experiments prove:

[0133] This invention enhances the multimodal reasoning capabilities of MLLMs and mitigates the illusion problem in CoT reasoning under a zero-shot setting. It proposes a collaboratively consistent multimodal CoT framework, named CoCo-CoT. This framework constructs clear perceptual decisions through visual analysis and sub-problems, and enhances the reliability of reasoning through cross-validation thinking and iterative self-optimization-enabled collaborative consistency evaluation. Extensive experiments on the ScienceQA and MathVista benchmarks further validate the effectiveness and generalization ability of CoCo-CoT. These results highlight its ability to improve the robustness and interpretability of multimodal reasoning, providing a promising direction for advancing reliable multimodal CoT reasoning.

[0134] In all respects, the embodiments should be regarded as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Therefore, it is intended that all variations falling within the meaning and scope of the equivalents of the application be included within the present invention.

[0135] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. An iterative comparative reasoning method based on MLLMs, characterized in that, Includes the following steps: Step S1: Under zero-sample settings, multimodal data input is obtained through the perception decision unit, and the decision intelligent agent module independently analyzes the problem and image, constructs the reasoning process and proposes sub-problems, and constructs a clear perception decision, that is, formulates parallel decision; Step S2: The reasoning agent module answers the sub-questions sequentially to support decision-making, and then constructs a collaborative reasoning path based on the corresponding decisions; Step S3: Evaluate the agent module to perform cross-validation between inference paths. If a consensus is reached, the answer will be accepted; otherwise, iterative self-optimization is triggered to form a closed-loop inference process.

2. The iterative comparative thought chain reasoning method based on MLLMs according to claim 1, characterized in that, In step S1, the decision-making agent module independently analyzes the problem and image features to formulate parallel decisions, including the following steps: Step S11: The decision-making agent module includes multiple decision-making agent units, which perform structured analysis on multimodal data and task problems. Each decision-making agent unit can independently analyze problems and images, and generate reasoning arguments by performing context analysis. Step S12: The decision-making agent module performs structured analysis on the image. Based on the generated rationale and problem analysis, the decision-making agent module raises a question to obtain the theoretical or conceptual knowledge required to solve the task. Step S13: The decision-making agent module proposes specific sub-problems based on reasoning and visual analysis in order to flexibly and comprehensively obtain the visual evidence required for reasoning.

3. The iterative comparative thought chain reasoning method based on MLLMs according to claim 2, characterized in that, In step S1, the reasoning process is constructed and sub-problems are proposed. This includes guiding the model to explicitly analyze visual information through implicit design hints or system flow, proposing questions by explicitly analyzing visual information, and decomposing complex problems into a series of sub-problems for gradual solution. The sub-problems may involve identifying object attributes or analyzing spatial relationships. The explicit and implicit information of the sub-problems are crucial for solving the task. The visual sub-problems are defined by formula (6): ； in, Activate the decision-making agent Generate the visual subproblem.

4. The iterative comparative thought chain reasoning method based on MLLMs according to claim 1, characterized in that, In step S2, the reasoning agent module includes an answering unit and a comprehensive reasoning unit. The answering unit answers sub-questions sequentially to support perceptual decision-making. The specific steps are as follows: Step S21: The reasoning agent module retrieves the problem analysis results and the set of conceptual sub-problems, and the answering unit answers the conceptual sub-problems in sequence, with the answer to the previous sub-problem serving as key supporting evidence for the current sub-problem; Step S22: The answering unit answers the set of visual sub-questions, wherein the visual sub-questions and their corresponding answers are independent of each other and do not depend on the answer of the previous sub-question; Step S23: The integrated reasoning unit integrates the rationale and all relevant information in the plan, including the answers to conceptual and visual sub-questions. Based on these inputs, it constructs a coherent step-by-step reasoning chain to complete the entire original question-answering process.

5. The iterative comparative thought chain reasoning method based on MLLMs according to claim 4, characterized in that, The entire original question-and-answer process in step S23 is formalized as follows: ； in Indicated by the reasoner The inference function executed. Response. Including the constructed inference chain and the final answer The complete reasoning path is represented as ,in This indicates different reasoning paths.

6. The iterative comparative thought chain reasoning method based on MLLMs according to claim 1, characterized in that, The evaluation agent module in step S3 is a consistency evaluation module. The consistency evaluation module includes cross-validation thinking and iterative self-optimization. The evaluation agent module independently evaluates each component of the main inference path and its inference chain, and collaborates and interacts between parallel inference paths to further identify potential errors and extract reliable supporting evidence.

7. The iterative comparative thought chain reasoning method based on MLLMs according to claim 6, characterized in that, The evaluation agent module further includes integrating sub-questions and answers from the auxiliary reasoning path into the main reasoning path, and prompting the evaluator to perform cross-validation thinking. The cross-validation feature between reasoning paths includes the following steps: Step S31: If the two answers are the same in the two reasoning paths, the evaluation agent module performs cross-validation on one of the reasoning paths; otherwise, cross-validation is performed independently on the two paths respectively. Step S32: If the results of cross-validation converge to a consistent conclusion, the result is considered reliable and accepted as the final answer; otherwise, the evaluation agent module enters the iterative self-optimization stage for further optimization.

8. The iterative comparative thought chain reasoning method based on MLLMs according to claim 6, characterized in that, In the cross-validation process, if the results converge to a consistent conclusion, the result is considered reliable and accepted as the final answer; otherwise, the evaluator enters an iterative self-optimization phase for further optimization. The entire process is defined as follows: ; in, and This indicates that the reasoning comes from two different paths. and The answer, This indicates the corrected result after single-path cross-validation. and The corrected result was obtained through dual-path cross-validation. This indicates the final accepted answer after the consensus assessment.

9. The iterative comparative thought chain reasoning method based on MLLMs according to claim 6, characterized in that, The iterative self-optimization extracts consistency and difference points from the inference path without making an explicit judgment on correctness. The evaluation agent will consider the two inference paths... and As input, comparisons are made, and consistency and difference information between the two reasoning processes are extracted, including reasoning logic and visual evidence, and organized into an iterative guidance set. The iterative self-optimization process is formalized as follows: ； in, Indicates that the evaluation agent The executed iteration function The extracted consistency and difference information serves as a guide for iterative reasoning optimization.

10. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores a computer program for execution to implement the iterative comparative thought chain reasoning method based on MLLMs as described in any one of claims 1 to 9.

Images