A VQA task processing method and system integrating the RefineChain mechanism

Abstract

This invention provides a VQA task processing method and system integrating the RefineChain mechanism. The method includes: generating a first inference step based on an input VQA task according to the MMCot inference framework; wherein the VQA task includes an image and natural language text related to the image; performing structured processing on the original inference step to obtain a second inference step; using the RefineChain mechanism, identifying the missing visual association steps in the second inference step based on the features of the image in the VQA task, merging the second inference step with the visual association steps to obtain a third inference step, verifying the consistency of the third inference step with the image and natural language text from multiple dimensions, correcting contradictions and / or redundant steps contained in the third inference step, to obtain a fourth inference step; using the fourth inference step as input, performing answer inference according to the MMCot inference framework to obtain the final output answer.

Description

Technical Field

[0001] This invention relates to the field of visual question answering technology, and in particular to a VQA task processing method that integrates the RefineChain mechanism. Background Technology

[0002] Visual Question Answering (VQA) is an interdisciplinary task designed to enable machines to generate accurate answers based on given images (visual) and associated text (linguistic) information. VQA is widely used in fields such as intelligent customer service, autonomous driving scene understanding, and educational Q&A. Currently, the mainstream VQA multimodal reasoning framework is MMCoT (Multimodal Chain-of-Thought). The MMCoT reasoning framework addresses the problems of relying on a single modality and opaque reasoning processes in traditional VQA tasks through a two-stage framework, achieving performance breakthroughs on benchmarks such as ScienceQA and A-OKVQA. The two-stage framework includes: 1) Rationale generation: Taking image (visual) and text (linguistic) information as input, it outputs a rationale describing logic, background knowledge, or visual facts. 2) Answer Inference: Taking the original question, image (visual), and reasoning steps (Rationale) as input, it outputs the final answer, which is equivalent to taking the reasoning steps (Rationale) as a "draft".

[0003] However, the MMCot inference framework still faces key technical bottlenecks in VQA tasks: the Rationale (inference steps) generated by the MMCot inference framework is unstructured natural language text and is not broken down into explicit inference steps (such as "Step 1: Identify objects in the image → Step 2: Associate the question in the text → Step 3: Verify object attributes"). This makes it difficult to optimize the inference chain in a targeted manner. It is difficult to identify missing key steps (such as missing "judgment of the positional relationship between objects in the image") and it is also difficult to efficiently correct logical contradictions (such as inconsistencies in the description of object attributes in the preceding and following steps), which limits the inference accuracy of VQA tasks.

[0004] Therefore, there is an urgent need for an improved MMCot reasoning framework to enhance its integrity and logic in VQA task scenarios. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a VQA task processing method and system that integrates the RefineChain mechanism to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a VQA task processing method incorporating a RefineChain mechanism. The method includes the following steps: generating a first inference step based on an input VQA task according to the MMCot inference framework; wherein the VQA task includes an image and natural language text associated with the image; performing structured processing on the original inference step to obtain a second inference step; using the RefineChain mechanism, identifying the missing visual association steps in the second inference step based on the features of the image in the VQA task, merging the second inference step with the visual association steps to obtain a third inference step; verifying the consistency of the third inference step with the image and natural language text from multiple dimensions, correcting contradictions and / or redundant steps contained in the third inference step, to obtain a fourth inference step; using the fourth inference step as input, performing answer inference according to the MMCot inference framework to obtain the final output answer.

[0007] In some embodiments of the present invention, the structured processing includes step boundary identification, dependency annotation, and structured format output; the step boundary identification includes: dividing each reasoning sub-step in the first reasoning step based on conjunctions and semantic pauses in natural language text; the dependency annotation includes: annotating the pre-dependencies between the various reasoning sub-steps through semantic analysis; the structured format output includes: outputting the divided reasoning sub-steps and their pre-dependencies according to a preset standardized format to obtain the second reasoning step.

[0008] In some embodiments of the present invention, the preset standardized format includes a linear sequence format and / or a tree structure format. Each node in the linear sequence format includes an inference sub-step number, the content of the inference sub-step, and its predecessor dependent sub-steps. The tree structure format uses each inference sub-step as a node and uses the parent-child relationship between nodes to store the predecessor dependencies between each inference sub-step.

[0009] In some embodiments of the present invention, the step of structuring the original inference steps to obtain the second inference step includes: using a pre-trained sequence labeling model to divide the first inference step into multiple inference sub-steps; using a pre-trained text entailment model to determine the logical dependencies between the various inference sub-steps; and outputting the divided inference sub-steps and their logical dependencies according to a preset standardized format to obtain the second inference step.

[0010] In some embodiments of the present invention, the step of identifying the missing visual association step in the second inference step based on the features of the image in the VQA task, and merging the second inference step with the visual association step to obtain a third inference step, includes: reverse-verifying whether the inference sub-steps included in the second inference step can support achieving the final problem goal based on the final problem goal of natural language text parsing related to the image; generating the missing visual association step based on the features of the image in the VQA task according to the structure of the reverse verification; wherein the generation rule of the visual association step is: logically consistent with the original step and does not destroy the original structure; after verifying the integrity of the visual association step, merging the second inference step with the visual association step to obtain the third inference step.

[0011] In some embodiments of the present invention, the multi-dimensional verification includes some or all of visual consistency, textual consistency, and logical consistency.

[0012] In some embodiments of the present invention, the MMCot inference framework and the RefineChain mechanism are deployed separately, and the steps using the RefineChain mechanism are trained separately; or, the MMCot inference framework and the steps using the RefineChain mechanism are trained end-to-end.

[0013] Corresponding to the above methods, the present invention also provides a VQA task processing system that integrates the RefineChain mechanism, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described in the above embodiments.

[0014] In accordance with the above methods, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0015] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0016] The VQA task processing method proposed in this invention, which integrates the RefineChain mechanism, adds a structured processing step to the existing MMCot inference framework. This transforms the first inference step, which is in the form of unstructured natural language text generated by the MMCot inference framework, into a structured second inference step. Furthermore, it designs a self-complete step and a self-reflection step that follow the RefineChain mechanism. The self-complete step completes the missing visual association steps, and the self-reflection step corrects or eliminates contradictory and / or redundant steps, thereby obtaining structured and more accurate inference features and improving the quality of the final answer.

[0017] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart of a VQA task processing method that integrates the RefineChain mechanism in one embodiment of the present invention.

[0020] Figure 2 This is a diagram of a VQA task processing system architecture that integrates the RefineChain mechanism in one embodiment of the present invention.

[0021] Figure 3 This is a comparison diagram of the VQA task processing method in one embodiment of the present invention and the prior art.

[0022] Figure 4 This is a schematic diagram of the computer equipment included in the system. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0024] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0026] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0027] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0028] Besides existing complex problem decomposition schemes for text-based languages ​​such as CoT / ToT / AoT, the closest existing technology to this invention is the MMCot (Multimodal-CoT) framework (from the paper "Multimodal Chain-of-ThoughtReasoning in Language Models", TMLR 2024), whose specific implementation scheme in VQA tasks is as follows: The MMCoT framework employs a two-stage, separate design to adapt to the "visual + language" input scenario of VQA. The MMCoT framework comprises the following two stages: 1) Rationale Generation: The input is "text (question Q + context C + option M) + image visual features (extracted by ViT (Vision Transformer))", which generates a natural language Rationale (e.g., "To determine if a magnet is attractive, you need to look at the magnetic poles: the N pole of the left magnet is close to the S pole of the right magnet; opposite poles attract, therefore they are attracted") through an encoder-decoder model (e.g., "To determine if a magnet is attractive, you need to look at the magnetic poles: the N pole of the left magnet is close to the S pole of the right magnet; opposite poles attract, therefore they are attracted"). 2) Answer Inference: The input is "original text + generated Rationale + image visual features", and the model infers the final answer based on the Rationale.

[0029] By adopting the MMCot framework and fusing visual and linguistic features through the gated fusion mechanism, the problem of "insufficient modal interaction" in traditional VQA models can be solved. Moreover, state-of-the-art performance can be achieved based on small models with less than 1 billion parameters (such as T5-Large 738M), with an accuracy of 88.25% on the VQA subset of ScienceQA (image-related problems).

[0030] However, the Rationale in the MMCot framework is a continuous natural language and is not structured into reasoning steps, which leads to: 1) Insufficient completeness: It is impossible to locate missing reasoning sub-steps (e.g., in the VQA task of "judging the concentration of the solution", the MMCot Rationale may miss the key step of "counting the number of solute particles in the image"); 2) Difficulty in optimization: It is difficult to accurately correct logical contradictions (e.g., the Rationale first states "object A in the image is on the left", and then states "object A is on the right", and due to the lack of step division, it is impossible to quickly locate the contradictory position).

[0031] In VQA task scenarios, the shortcomings of existing technologies (including MMCot) are as follows: 1) Unstructured Rationale (a core problem of MMCot): The Rationale generated by MMCot is non-step natural language, lacking "reasoning step boundaries" and "logical dependencies". For example, in the A-OKVQA task of "determining the purpose of a tool in an image", the Rationale of MMCot may be "the tool in the image is a hammer, used to hammer nails, because it has a metal hammerhead and a wooden handle" - without being broken down into "sub-step 1: identify the tool as a hammer → sub-step 2: analyze the tool structure (metal hammerhead + wooden handle) → sub-step 3: associate structure with purpose (hammering nails)", which makes it impossible to specifically complete or correct a certain sub-step later. 2) Lack of post-optimization mechanism: MMCot only completes the one-way flow of "Rationale generation → answer reasoning" and does not perform post-processing optimization on the Rationale. According to experimental data from the paper, MMCoT causes 32% of errors due to missing Rationale steps and 21% due to logical contradictions in ScienceQA's VQA task, and existing frameworks cannot correct these errors.

[0032] To overcome the problems of existing MMCoT inference frameworks for unstructured natural language text, optimize the inference chain, accurately identify missing key steps, and efficiently correct logical contradictions, thereby improving the inference accuracy for VQA tasks, this application proposes a VQA task processing method that integrates the RefineChain mechanism. Figure 1 This is a flowchart of a VQA task processing method integrating the RefineChain mechanism in one embodiment of the present invention. The method includes the following steps: Step S110: Generate a first inference step based on the input VQA task according to the MMCot inference framework; wherein the VQA task includes an image and natural language text associated with the image.

[0033] In this context, a Rationale is an unstructured natural language text describing logic, background knowledge, or visual facts. The current MMCot inference framework comprises two phases: 1) Rationale generation: Taking image (visual) and text (language) information as input, it outputs a Rationale describing logic, background knowledge, or visual facts. 2) Answer inference: Taking the original question, image (visual), and Rationale as input, it outputs the final answer, essentially using the Rationale as a "draft." Furthermore, Rationale can also be translated as reason, basis for reasoning, reasoning steps, or chain of reasoning.

[0034] Step S120: The original reasoning steps are structured to obtain the second reasoning steps.

[0035] The aforementioned structured processing includes step boundary identification, dependency labeling, and outputting in a structured format to obtain the structured reasoning steps.

[0036] Step S130: Using the RefineChain mechanism, the visual association steps missing in the second inference step are identified based on the features of the image in the VQA task. The second inference step and the visual association steps are merged to obtain the third inference step. The consistency between the third inference step and the image and natural language text is verified in multiple dimensions. The contradictions and / or redundant steps contained in the third inference step are corrected to obtain the fourth inference step.

[0037] Specifically, ViT (Vision Transformer) is used to extract image features (image visual features) in the VQA task. The above-mentioned step of identifying visual associations using the RefineChain mechanism is based on the extracted image features in the VQA task.

[0038] Step S140: Using the fourth reasoning step as input, perform answer reasoning according to the MMCot reasoning framework to obtain the final output answer.

[0039] The VQA task processing method proposed in this invention, which integrates the RefineChain mechanism, adds a structured processing step to the existing MMCot inference framework. This transforms the first inference step, which is in the form of unstructured natural language text generated by the MMCot inference framework, into a structured second inference step. Furthermore, it designs a self-complete step and a self-reflection step that follow the RefineChain mechanism. The self-complete step completes the missing visual association steps, and the self-reflection step corrects or eliminates contradictory and / or redundant steps, thereby obtaining structured and more accurate inference features and improving the quality of the final answer.

[0040] In some embodiments of the present invention, the structured processing includes step boundary identification, dependency labeling, and structured format output.

[0041] Accordingly, the step boundary identification includes: dividing each reasoning sub-step in the first reasoning step based on conjunctions and semantic pauses in natural language text; the dependency labeling includes: labeling the pre-dependencies between each reasoning sub-step through semantic analysis; the structured format output includes: outputting each divided reasoning sub-step and its pre-dependencies according to a preset standardized format to obtain the second reasoning step.

[0042] By employing this embodiment of the invention, the first reasoning step can be decomposed into multiple structured reasoning sub-steps using step boundary recognition, the dependency relationships between each reasoning sub-step can be intelligently identified using dependency labeling, and the structured second reasoning step can be obtained through formatted output. Compared with the first reasoning step in natural language form, the reasoning steps are more explicit, which helps to optimize the reasoning chain in a targeted manner, thereby identifying the missing key steps and efficiently correcting logical contradictions.

[0043] In some embodiments of the present invention, the preset standardized format includes a linear sequence format and / or a tree structure format. Each node in the linear sequence format includes an inference sub-step number, the content of the inference sub-step, and its predecessor dependent sub-steps. The tree structure format uses each inference sub-step as a node and uses the parent-child relationship between nodes to store the predecessor dependencies between each inference sub-step.

[0044] By employing this embodiment of the invention, each reasoning sub-step and its prerequisite dependencies are stored in a linear sequence format and / or a tree structure format. The topological structure of the reasoning logic can be clearly revealed through the preset standardized format, which helps to promote the step completion and step pruning operations of subsequent self-completion steps and self-reflection steps.

[0045] In some embodiments of the present invention, the step of structuring the original inference steps to obtain the second inference step includes: using a pre-trained sequence labeling model to divide the first inference step into multiple inference sub-steps; using a pre-trained text entailment model to determine the logical dependencies between the various inference sub-steps; and outputting the divided inference sub-steps and their logical dependencies according to a preset standardized format to obtain the second inference step.

[0046] In other words, a rule-based decomposition approach can be used, relying on rules (connectors + semantics) for decomposition, avoiding increased deployment costs for pre-trained models. This approach achieves higher accuracy than the model-based decomposition approach in small-sample VQA tasks (such as niche domain problems in A-OKVQA). Alternatively, a model-based decomposition approach can be used, employing a pre-trained sequence labeling model (such as BERT-Base) to label MMCot's Rationale as "Step Start (B-STEP), Step Middle (I-STEP)" labels, automatically defining step boundaries. Dependencies are then determined using a pre-trained textual entailment model (such as RoBERTa) to determine logical dependencies between steps, which helps adapt to more complex Rationale texts (such as long sentences without obvious connectors).

[0047] By employing this embodiment of the invention, pre-trained sequence labeling models and textual implication models (belonging to NLP models) can be used to automatically achieve structured processing. When extracting information at a fine-grained level, the logical coherence between steps and the rationality of pre-dependent relationships can be automatically verified, replacing tedious regular expressions or hard rules written manually. This helps to significantly reduce the cost of manual labeling and shorten the iteration cycle of the VQA task processing system.

[0048] In some embodiments of the present invention, the step of identifying the missing visual association step in the second inference step based on the features of the image in the VQA task, and merging the second inference step with the visual association step to obtain a third inference step, includes: reverse-verifying whether the inference sub-steps included in the second inference step can support achieving the final problem goal based on the final problem goal of natural language text parsing related to the image; generating the missing visual association step based on the features of the image in the VQA task according to the structure of the reverse verification; wherein the generation rule of the visual association step is: logically consistent with the original step and does not destroy the original structure; after verifying the integrity of the visual association step, merging the second inference step with the visual association step to obtain the third inference step.

[0049] By employing this embodiment of the invention, the integrity of the logical chain can be deduced in reverse through a goal-oriented reverse verification and missing visual association compensation mechanism. This helps to avoid misjudgments caused by missing information, avoids the problem of missing visual information in traditional reasoning, and helps to generate a highly interpretable derivation path.

[0050] In some embodiments of the present invention, the multi-dimensional verification includes some or all of visual consistency, textual consistency, and logical consistency.

[0051] By employing this embodiment of the invention, the consistency and refinement of the output reasoning steps can be optimized through multi-dimensional verification.

[0052] In some embodiments of the present invention, the MMCot inference framework and the RefineChain mechanism are deployed separately, and the steps using the RefineChain mechanism are trained separately; or, the MMCot inference framework and the steps using the RefineChain mechanism are trained end-to-end.

[0053] In other words, a modular post-processing integration approach can be adopted, deploying MMCot and RefineChain separately. This retains the lightweight advantage of MMCot (increasing training parameters from 738M to 1.2B) and is compatible with existing MMCot pre-trained models, eliminating the need for retraining. Alternatively, an end-to-end training approach can be used, jointly training the "structured processing + RefineChain" module with the two-stage MMCot model end-to-end. Joint optimization using "structured step loss + answer loss" enhances inter-module synergy and can further improve accuracy by 1.5% in ScienceQA's VQA task.

[0054] The improved lightweight MMCot inference architecture is compatible with small MMCot models (≤1B parameters) without adding extra training costs, and can be directly integrated and optimized on the original MMCot model.

[0055] Using this invention, training requirements under different VQA task processing system deployment conditions can be met through individual training or end-to-end joint training. End-to-end joint training can adapt to more complex Rationale texts (such as long sentences without obvious conjunctions), while individual training can reduce the deployment cost of pre-trained models. Moreover, in small-sample VQA tasks (such as niche domain problems of A-OKVQA), the accuracy is higher than that of the rule-based solution (the original solution has an accuracy of 89% vs. the alternative solution has an accuracy of 78%).

[0056] The core improvement of this invention is based on the original two-stage framework of MMCot, with the addition of a "Rationale structured processing module" and a "RefineChain optimization module", forming a "four-stage framework": MMCot Stage 1 (Rationale generation) → New stage (structured processing + RefineChain optimization) → MMCot Stage 2 (answer reasoning) In some embodiments of the present invention, the structured processing of the new stage is implemented using the Rationale structured processing module, which is described below.

[0057] Figure 2 This is an architecture diagram of a VQA task processing system that integrates the RefineChain mechanism in one embodiment of the present invention. The structure of the inference steps Rationale after structured processing and RefineChain optimization is represented as Evidence 1, Evidence 2, and Evidence 3...

[0058] The function of the Rationale structured processing module is to receive the unstructured Rationale (reasoning steps) generated in MMCot stage 1, decompose it into structured reasoning sub-steps, and mark the logical dependencies between the sub-steps.

[0059] The workflow of the Rationale structured processing module is as follows: 1) Step boundary recognition: Based on conjunctions (such as “because”, “so”, “first”, “then”) and semantic pauses in natural language text, the reasoning sub-steps are divided. For example, the MMCot Rationale “The N pole of the magnet in the image is close to the S pole, and opposite poles attract each other, so it will attract” is broken down into “Sub-step 1: The N pole of the magnet in the image is close to the S pole; Sub-step 2: Opposite poles attract each other; Sub-step 3: Therefore, the magnet will attract”.

[0060] 2) Dependency labeling: Through semantic analysis, the preceding dependent sub-steps of each step are labeled. For example, sub-step 3 depends on sub-step 1 (magnetic pole position) and sub-step 2 (opposite poles attract rule), and is labeled as "sub-step 3 depends on: 1, 2".

[0061] 3) Structured output: Output the decomposed inference steps in a standardized format, such as linear sequence format (suitable for CoT structure): S_struct = [{step_id:1, content:"...", depend:[]}, {step_id:2,content:"...", depend:[1]}, ...]; or tree structure format (suitable for ToT structure): S_struct ={root_step:1, children:[{step_id:2, parent:1}, ...]}.

[0062] In some embodiments of the present invention, the RefineChain optimization of the new stage is implemented using the RefineChain optimization module, which includes three core modules: ① Initial inference step receiving module (compatible with existing decomposition schemes); ② Self-Completion module (denoted as C); ③ Self-Reflection module (denoted as R); ④ Answer generation guidance module. The connection relationship between the modules is: Existing decomposition scheme → Initial inference step receiving module → Self-Completion module → Self-Reflection module → Answer generation guidance module, wherein self-completion and self-reflection are executed sequentially and neither modifies the core logic of the existing decomposition scheme.

[0063] The RefineChain optimization module performs "self-completion + self-reflection" optimization on the structured inference steps S_struct to generate high-quality inference steps S_refine, which are then input into the AnswerInference stage of MMCot inference to obtain the final answer. The self-completion module combines image feature recognition to supplement missing visual association steps, while the self-reflection module verifies the consistency between the steps and image features and text information, correcting contradictory or redundant steps.

[0064] The overall process of RefineChain optimization is as follows: generating initial inference steps based on the existing decomposition scheme. →The missing step of the self-completion module is obtained. →The self-reflection module optimization logic is obtained →Based on Generate the final answer.

[0065] Figure 3 This is a comparison diagram of the VQA task processing method in one embodiment of the present invention and the prior art.

[0066] In some embodiments of the present invention, in step S130 above, the visual association step missing in the second inference step is identified based on the features of the image in the VQA task, and the second inference step and the visual association step are merged to obtain the third inference step, which is implemented using a self-complete module.

[0067] The core logic of the self-complete module is to combine the image features of the VQA task to identify the missing "visual association steps" (such as "identifying image objects", "determining object positions" and "counting the number of elements in the image") in the inference step S_struct.

[0068] For example, in the VQA task of "determining solution concentration", if the step of "counting the number of solute particles in the image" is missing from S_struct, the self-completion module will generate a supplementary step based on the image features extracted by ViT (such as the pixel coordinates of solute particles): "Sub-step 2: Count the number of solute particles in Solution A in the image as 5, and the number of solute particles in Solution B as 8 (depending on sub-step 1: identifying solution A / B)".

[0069] The workflow of the self-completion module is as follows: a) Missing Step Identification: Employing a "goal-backward approach," starting from the final problem objective and verifying in reverse. Whether each sub-step can support the achievement of the goal. For example, in a HotpotQA task, if the goal is "to determine the time of an event", then the validation... If the three required sub-steps are included—"find event-related documents," "extract time information from documents," and "verify time accuracy"—then mark them as "to be supplemented."

[0070] b) Generation of supplementary sub-steps: Missing sub-steps (visually related steps) are generated based on LLMs, with the generation rule being "logically coherent with the original step and without disrupting the original structure." For example, in a linear sequence of CoT, if... and Space missing Then Updated to In a branch of ToT, if node Missing child nodes Then add and associate with the parent node .

[0071] c) Integrity verification: Confirmed through "manual rules + model evaluation" The completeness of the process, for example, setting "at least 90% of the required sub-steps are covered" as a passing standard.

[0072] The output process of the self-completion module is as follows: Structured steps after completion. .

[0073] In some embodiments of the present invention, step S130 performs multi-dimensional verification of the consistency between the third inference step and the image and natural language text, corrects the contradictions and / or redundant steps contained in the third inference step, and obtains the fourth inference step, which is implemented using a self-reflection module to correct the multimodal logic consistency in the VQA task.

[0074] The core logic of the self-reflection module is to verify the consistency between each step in the structured step S1 and the "image features" and "text information", and to correct contradictory and redundant steps.

[0075] The self-reflection module's verification dimensions are: ① Visual consistency: Does the step description match the image features (e.g., if the step states "the magnets in the image have opposite S poles", but the image features show opposite N poles, then the step should be corrected); ② Textual consistency: Does the step description match the question / context (e.g., if the step states "according to the context, magnets repel each other", but the context clearly states "opposite poles attract each other", then the step should be corrected); ③ Logical consistency: Are there any contradictions in the dependencies between steps (e.g., if step a depends on step b, but step b does not mention key information, then the steps should be rearranged or rewritten).

[0076] Optionally, visual consistency check: Determine if the description of each step matches X_vision. If a step states "Object A is on the left in the image" but X_vision shows A on the right, then correct the step content. Text consistency check: Determine if the description of each step matches the VQA question Q and context C. If the step contradicts the text information, then rewrite the step. Logical consistency check: Determine if there are contradictions in the dependencies between steps (e.g., step 3 depends on step 2, but step 2 does not provide key information). If so, rearrange the steps or add dependent steps, generating S_refine.

[0077] Workflow of the self-reflection module: a) Relevance check: Determine whether each sub-step is related to the final goal. For example, in the 24-point game, "analyzing the parity of numbers" is not related to "generating the formula for 24", so it is deleted. b) Redundancy check: Identify duplicate sub-steps. For example, in an AoT DAG, if two nodes... and If both conditions are "check a=5", then keep one and adjust the dependency. c) Consistency check: Check for logical inconsistencies between sub-steps. For example, in a branch of ToT, if... Assuming "a=5", If "a=6", then rewrite For "based on Given a=5, calculate... d) Structure optimization: Reorder linear sequences (e.g., move the "Verify Information" step before "Generate Conclusion"), and prune invalid branches in tree / DAG structures (e.g., branches evaluated as "feasibility < 10%" in ToT). Example: If sub-step 2 in structured step S1 states "the number of solute particles in Solution A is 10", but the image features show 5, the self-reflection module will correct sub-step 2 to "sub-step 2: count the number of solute particles in Solution A in the image as 5 (depending on sub-step 1)", and simultaneously correct subsequent dependent steps (such as the concentration calculation in sub-step 3).

[0078] The output flow of the self-reflection module: optimized structured steps .

[0079] In a specific embodiment of the present invention, the improved VQA task processing flow of this application is represented by a formulaic method: Where Q represents the VQA problem, C represents the context, M represents the options, X_{vision} represents the image visual features, R_{raw} represents the raw Rationale generated by MMCoT, S_{struct} represents the structured steps, S_{refine} represents the steps optimized by RefineChain, and represents the final answer.

[0080] Using self-completion (completing missing steps) + self-reflection (correcting logical contradictions) helps improve the reasoning completeness and logic of VQA task processing.

[0081] In some embodiments of the present invention, RefineChain optimization can be replaced with multi-round iterative optimization, changing the single-round optimization of "self-completion → self-reflection" to multi-round iteration (S_struct → completion → reflection → structured step S1 → completion → reflection → S2 → ... → S_k) until the steps achieve "visual consistency ≥ 95% and logical consistency ≥ 95%". Its advantage lies in the ability to further optimize sufficiency, potentially improving accuracy by 2.3% in complex VQA tasks (such as the Crosswords visual puzzle). However, its disadvantages include increased inference latency (0.8s for single-round optimization vs. 2.1s for multi-round optimization) and the potential for over-optimization (e.g., inserting unnecessary detailed steps, leading to step redundancy).

[0082] Optionally, the self-completion logic can be: 1) Input S_struct and the image features X_vision of the VQA task; 2) Identify the "visual association steps" (such as object recognition, position judgment, element statistics) not included in S_struct; 3) Generate supplementary steps based on X_vision and insert them into the corresponding positions of S_struct (such as inserting the supplementary step of "statistical analysis of the number of solute particles in the image" before the "calculate solution concentration" step), and generate the structured step S1.

[0083] By adopting the structured decomposition logic of Rationale in the VQA scenario, the unstructured Rationale of MMCoT can be decomposed into structured steps containing "step ID, content, and dependency relationship" through "connector semantics + image feature association". In other words, the natural language Rationale can be decomposed into "ordered reasoning steps + inter-step dependency relationship" (such as linear step sequence or tree-like step structure), thereby meeting the multimodal reasoning requirements of VQA tasks (such as associated image object recognition, position determination, etc.).

[0084] Corresponding to the above method, the present invention also provides a VQA task processing system that integrates the RefineChain mechanism. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0085] See Figure 4 , Figure 4 This is a schematic diagram of the structure of the computer device included in the system. The computer device 00 includes: a processor 01, a memory 02, and a computer program stored on the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, it implements the method steps proposed in any of the above embodiments.

[0086] The processor 01 is connected to the memory 02, such as via a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The bus 03 may include a pathway for transmitting information between the aforementioned components. The bus 03 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 03 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The text uses only a single thick line to represent a bus, but this does not imply that there is only one bus or one type of bus. Memory 02 stores a computer program corresponding to the human factors data server access control method described in the above embodiments of this application. This computer program is executed under the control of processor 01. Processor 01 executes the computer program stored in memory 02 to implement the content shown in the aforementioned method embodiments.

[0087] Corresponding to the methods described above, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0088] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0089] The advantages of this invention compared to the prior art are as follows: 1) Significantly Improved VQA Inference Accuracy: Based on experimental data from a subset of ScienceQA VQA (image-related problems), the improved solution (MMCoT + Structured Processing + RefineChain) achieves performance improvements over the original MMCoT: ① Accuracy: increased from 88.25% to 92.17% (+3.92%); ② Missing Step Error Rate: decreased from 32% to 11% (-21%); ③ Logical Contradiction Error Rate: decreased from 21% to 7% (-14%). The core reason is that structured processing makes optimization more accurate, and RefineChain specifically addresses the rational shortcomings of MMCoT.

[0090] 2) The reasoning process is more interpretable: MMCot's Rationale is unstructured text, making it difficult to trace the source of errors; while the structured steps of this invention can locate specific error steps (such as "sub-step 2 particle count error"), and RefineChain will output optimization logs (such as "correct sub-step 2: change the particle count from 10 to 5 based on image features"), which greatly improves the interpretability of VQA reasoning and facilitates manual debugging and subsequent iterations.

[0091] 3) Compatibility with MMCoT's lightweight and multi-task adaptability: The improved solution does not require modification of the core model structure of MMCoT (such as T5 encoder-decoder and ViT feature extraction), only adds modular processing steps, and can be directly integrated into the original MMCoT framework; and supports multiple VQA tasks such as ScienceQA and A-OKVQA, which can be deployed on 1B parameter models, maintaining the lightweight advantage of MMCoT.

[0092] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0093] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0094] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A VQA task processing method integrating the RefineChain mechanism, characterized in that, include: The first inference step is generated based on the input VQA task according to the MMCoT inference framework; wherein, the VQA task includes an image and natural language text associated with the image; The original reasoning steps are structured to obtain the second reasoning steps; The RefineChain mechanism is adopted to identify the missing visual association step in the second inference step based on the features of the image in the VQA task. The second inference step and the visual association step are merged to obtain the third inference step. The consistency between the third inference step and the image and natural language text is verified in multiple dimensions. The contradictions and / or redundant steps contained in the third inference step are corrected to obtain the fourth inference step. Using the fourth reasoning step as input, the answer is reasoned according to the MMCot reasoning framework to obtain the final output answer.

2. The method according to claim 1, characterized in that, The structured processing includes step boundary identification, dependency labeling, and structured format output; The step boundary identification includes: dividing each reasoning sub-step in the first reasoning step based on conjunctions and semantic pauses in natural language text; The dependency annotation includes: annotating the pre-dependencies between the various reasoning sub-steps through semantic analysis; The structured format output includes: outputting each of the divided reasoning sub-steps and their prerequisite dependencies according to a preset standardized format to obtain the second reasoning step.

3. The method according to claim 2, characterized in that, The preset standardized format includes a linear sequence format and / or a tree structure format. Each node in the linear sequence format includes an inference sub-step number, the content of the inference sub-step, and its predecessor dependent sub-steps. The tree structure format uses each inference sub-step as a node and uses the parent-child relationship between nodes to store the predecessor dependencies between each inference sub-step.

4. The method according to claim 1, characterized in that, The process of structuring the original reasoning steps to obtain the second reasoning step includes: A pre-trained sequence labeling model is used to divide the first inference step into multiple inference sub-steps by dividing the step boundaries. The logical dependencies between each reasoning sub-step are determined by a pre-trained textual entailment model. The various reasoning sub-steps and their logical dependencies are then output according to a preset standardized format to obtain the second reasoning step.

5. The method according to claim 1, characterized in that, The step of identifying the missing visual association step in the second inference step based on the features of the image in the VQA task, and merging the second inference step with the visual association step to obtain the third inference step, includes: Based on the natural language text parsing related to the image, the final problem objective is analyzed, and the reasoning sub-steps included in the second reasoning step are reverse-verified to support the achievement of the final problem objective. Based on the structure of the reverse verification, missing visual association steps are generated according to the features of the images in the VQA task; wherein the generation rule of the visual association steps is: logically consistent with the original steps and without destroying the original structure. After verifying the integrity of the visual association step, the second reasoning step is merged with the visual association step to obtain the third reasoning step.

6. The method according to claim 1, characterized in that, The multi-dimensional verification includes some or all of the following: visual consistency, textual consistency, and logical consistency.

7. The method according to claim 1, characterized in that, The MMCot inference framework and the RefineChain mechanism are deployed separately, with steps using the RefineChain mechanism being trained separately; or, the MMCot inference framework and steps using the RefineChain mechanism are jointly trained end-to-end.

8. A VQA task processing system integrating the RefineChain mechanism, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images