A method to reduce the computational resource consumption of training multimodal large models

Abstract

This invention relates to a method for reducing the computational resource consumption during training of a large multimodal model. The method includes: evaluating the large multimodal model to be trained using an evaluation sample set; collecting erroneous samples of incorrect answers generated by the model, and identifying the causes of these errors as visual perception defects or logical reasoning errors; generating corresponding evaluation samples for each visual perception defect and logical reasoning error, and updating the evaluation sample set; for erroneous samples, using correct reference thought chains as training samples to optimize the large multimodal model, wherein the format of the reference thought chain is image description – reasoning process – final answer; and using the updated evaluation sample set to perform a next round of evaluation on the trained large multimodal model. Optimizing the model using correct thought chains for erroneous samples can achieve efficient performance improvements with minimal data usage, thus reducing computational resource consumption.

Description

Technical Field

[0001] This invention relates to the field of multimodal large language model technology, and in particular to a method for reducing the computational resource consumption during training of multimodal large models. Background Technology

[0002] In recent years, multimodal large language model technology has developed rapidly, and multimodal mathematical reasoning has become an important benchmark for measuring the advanced cognitive ability of models. This type of task is geared towards mathematical scenarios that combine text and vision, requiring models to jointly conduct mathematical understanding, logical deduction, and numerical calculation based on textual problems and image information. It not only requires models to have the logical reasoning ability to process textual information, but also to have fine-grained visual perception ability. Since the key known conditions, spatial relationships, functional laws, and statistical characteristics of the problem are often implicit in geometric figures, function graphs, and data statistics, the above visual carriers contain the core constraints and numerical relationships required for reasoning. They are an important foundation for building mathematical models and completing solutions. Therefore, models need to accurately identify and understand the key information in these visual carriers to provide reliable premises for subsequent textual logical reasoning and mathematical calculations, thereby completing the complete multimodal mathematical reasoning process.

[0003] Current mainstream technical approaches mostly adopt a post-training optimization paradigm, which can be mainly divided into two categories: 1. Supervised fine-tuning paradigm: As a fundamental implementation method, this method constructs data pairs of "image-question-answer" and their reasoning thought chains, and trains the model based on maximum likelihood estimation, enabling the model to learn standardized reasoning patterns.

[0004] 2. Reinforcement Learning Optimization Paradigm: To further improve the depth and logic of reasoning, related research has introduced feedback-based reinforcement learning strategies. Reward signals are constructed through result feedback or process feedback, and optimization algorithms such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) are used to incentivize the model to explore better reasoning paths.

[0005] However, existing technologies generally couple visual perception with mathematical reasoning processes for joint optimization, leading to attribution ambiguity: when a model makes a reasoning error, it cannot effectively distinguish whether the error originates from a visual perception defect ("misreading") or a logical reasoning fallacy ("miscalculation"), making it difficult to accurately locate and target model defects. This deficiency results in model optimization heavily relying on large-scale, highly redundant training data. This extensive model optimization approach causes significant computational and data resource waste, representing a core pain point in the current technological system. Model training and iteration require repeated calculations and updates of multimodal data and all model parameters. The current training method, which relies on highly redundant training data, consumes substantial computational resources, resulting in high resource consumption. Moreover, this existing training method is inefficient in data utilization; only a portion of the training data effectively improves model performance, leaving a significant amount of redundant training data. Summary of the Invention

[0006] To address at least some of the aforementioned problems in the prior art, the present invention provides a method for reducing the computational resource consumption during model training, comprising: The evaluation sample set is used to evaluate the multimodal large model to be trained, and the error samples of the multimodal large model to be trained generating incorrect answers are collected. At the same time, the reasons for the incorrect answers are determined to be visual perception defects or logical reasoning errors. For each visual perception defect and logical reasoning error, a corresponding evaluation sample is generated, and the evaluation sample set is updated. Generating the corresponding evaluation sample for each visual perception defect and logical reasoning error includes: generating visual upgrade samples or reasoning upgrade samples using a generator and a code drawing tool, wherein the generator generates drawing code and text questions based on prompts, and the code drawing tool draws images using the drawing code; using isomorphic generalization technology, an isomorphic generalization variant is generated for each error sample, wherein the isomorphic generalization variant maintains the same visual complexity and reasoning complexity as the corresponding error sample, only equivalently replacing the numerical distribution or semantic background of the original error sample; the visual upgrade samples, reasoning upgrade samples, and isomorphic generalization variants all include questions and correct answers, with questions including image and text questions; For incorrect samples, the correct reference thought chain is used as the training sample to optimize the multimodal large model to be trained. The format of the reference thought chain is image description - reasoning process - final answer. The trained multimodal large model is evaluated again using the updated evaluation sample set.

[0007] Furthermore, the evaluation sample set is used to evaluate the multimodal large model to be trained, and erroneous samples of incorrect answers generated by the multimodal large model to be trained are collected. The causes of incorrect answers are identified as visual perception defects or logical reasoning errors, including: The multimodal large model to be trained is used as the student model, and the trained multimodal large model is used as the teacher model. An evaluation sample set containing multiple samples is input into the student model and the teacher model respectively. Each sample in the evaluation sample set contains a question and the correct answer. The questions include image and text questions. For each sample in the evaluation sample set, the student model outputs a student thought chain, and the teacher model outputs a standard reference thought chain. The analyzer aligns and compares the student's thought process with the reference thought process, identifies the student's thought process with incorrect answers, and diagnoses the cause of the error as a visual perception defect or a logical reasoning error. The format of the student's thought process is image description - reasoning process - final answer. Collect samples of incorrect answers generated by the student model as error samples.

[0008] Furthermore, the student model is forced to follow the following generation sequence during reasoning: image description → reasoning process → final answer; The complete reasoning process of the student model is formalized as a sampled structured trajectory. ,in , and These represent the generated image description, the reasoning process, and the predicted answer, respectively.

[0009] Furthermore, the analyzer aligns and compares the student's thought process with the standard reference thought process based on the cue words, including: Align and compare the final answer in the student's thought process with that in the reference thought process. If they are different, further investigate the reasons for the error, including: First, a visual perception check is performed: the analyzer aligns and compares the student's thought chain generated by the learned model with the image description in the reference thought chain. If semantic illusion, topological error or omission of basic visual premise is detected, the comparison is terminated and marked as a visual perception defect. After passing the visual perception check, the remaining errors were attributed to errors in the reasoning process and marked as logical reasoning errors.

[0010] Furthermore, for erroneous samples, the reference thought chain of the teacher model is used as a training sample to optimize the large multimodal model to be trained.

[0011] Furthermore, the visual complexity of the images in the visual upgrade samples is higher than that of the images in the error samples with visual perception defects. The reasoning complexity of the text problems in the upgraded reasoning samples is higher than that of the text problems in the error samples with logical reasoning errors.

[0012] Furthermore, a visual escalation method is used to generate visually upgraded samples, including: keeping the logical reasoning path and mathematical principles of the questions unchanged; based on the current visual complexity level, the generator uses prompts to dynamically adjust the drawing generation logic to increase the visual difficulty and obtain visually upgraded samples. The reasoning escalation method is used to generate reasoning escalation samples, including: keeping the visual image and geometric topology of the question unchanged; and using a generator to generate reasoning escalation samples based on clue words.

[0013] Furthermore, the evaluation sample set is updated by adding visual upgrade samples, inference upgrade samples, and isomorphic generalization variants to the evaluation sample set, and the trained model is evaluated in the next round.

[0014] Furthermore, when the accuracy of the multimodal large model in generating answers exceeds a threshold, the model optimization is stopped.

[0015] The present invention has at least the following beneficial effects: The method of reducing the computational resource consumption of multimodal large model training in this invention first evaluates the accuracy of the multimodal large model's generated answers, identifies samples with incorrect answers, and uses the correct reference thought chain to optimize the model in a targeted manner, achieving precise compensation for the model's capability gaps. It can achieve efficient improvement in model performance with a very small amount of training data. Compared with traditional training methods that require tens of thousands to hundreds of thousands of data points, the method of this invention can achieve the same or even better performance with only a few thousand data points, greatly saving computational resources and improving training efficiency. It avoids the inefficient optimization and large computational resource consumption caused by traditional random augmentation or extensive data augmentation, and reduces data construction costs.

[0016] The method of reducing the computational resource consumption of multimodal large model training in this invention identifies the cause of model-generated erroneous results as visual perception defects in image content recognition or logical reasoning errors based on images and text. This solves the problem of ambiguous error attribution in existing methods, provides a clear basis for subsequent targeted optimization, and thereby improves the accuracy of the model in recognizing and reasoning about multimodal data (images and text), as well as the targeting and interpretability of the training process.

[0017] This invention employs a closed-loop mechanism of "model evaluation - synthesizing evaluation samples, obtaining training samples - model evolution". In this mechanism, model evaluation identifies erroneous samples and diagnoses the causes of errors. Correct thought processes are used as training samples to optimize the model and achieve model purification. Synthesized evaluation samples are used to update the evaluation sample set for the next round of model evaluation. This closed-loop mechanism enables continuous monitoring, targeted repair, and iterative optimization of the model's capability status, breaking through the traditional training mode that relies on static large-scale data expansion, and enabling the model to achieve adaptive capability evolution during the training process.

[0018] The method of this invention effectively alleviates the "barrel effect" problem in traditional training methods. By decoupling the diagnosis of visual perception defects and logical reasoning errors, it can accurately locate the model's shortcomings and generate corresponding types of evaluation samples. Through a cycle of evaluation diagnosis, model optimization, and targeted evaluation sample synthesis, it achieves targeted capability enhancement. Compared with the indiscriminate global optimization strategy, this method significantly improves the efficiency of training resource utilization and the targeting of capability enhancement, and can greatly improve the performance of the model on multimodal mathematical reasoning tasks. Attached Figure Description

[0019] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the embodiments of the invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0020] Figure 1 A flowchart illustrating a method for reducing computational resource consumption during training of large multimodal models according to an embodiment of the present invention is shown.

[0021] Figure 2 Pseudocode is shown for a method to reduce computational resource consumption during training of large multimodal models according to an embodiment of the present invention.

[0022] Figure 3 A computer system 100 for implementing the method according to the invention is shown. Detailed Implementation

[0023] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0024] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0025] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0026] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0027] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0028] It should also be noted that in the description of this invention, the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not explicitly or implicitly suggest that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0029] Furthermore, the embodiments of the present invention describe the process steps in a specific order. However, this is only for the convenience of distinguishing each step, and is not a limitation on the order of each step. In different embodiments of the present invention, the order of each step can be adjusted according to the process.

[0030] Multimodal large models receive image data and text questions to perform multimodal mathematical reasoning, and can output the answer and reasoning process.

[0031] To enhance the multimodal mathematical reasoning capabilities of large multimodal models, existing technical solutions mainly improve upon these capabilities from three dimensions: data scale, data quality, and data scheduling.

[0032] Large-scale instruction fine-tuning and knowledge distillation: The core strategy of this approach is to improve the model's generalization ability by expanding the scale and diversity of data. Technically, this relies heavily on collecting massive amounts of heterogeneous multimodal data, or using thought chain data generated by more powerful closed-source teacher models (such as GPT-4V) for distillation training. Large-scale datasets are constructed by integrating millions of multimodal instruction data or employing variance-aware sampling strategies. While this approach improves model performance to some extent, it is extremely dependent on the amount of data and, due to a lack of diagnosis of specific cognitive shortcomings in the model, results in low data utilization and difficulty in achieving efficient capability enhancement.

[0033] Instruction Evolution and Synthesis: This approach aims to automatically increase the complexity of training data through algorithms to construct a higher-quality fine-tuning dataset. Technical Principle: Typically, a large language model is used as a rewriter to rewrite the original seed problem. This is achieved by increasing the text length of the problem, adding constraints, or introducing more complex logic to increase the data difficulty; alternatively, geometric image data can be generated through programmatic code. For example, it borrows evolutionary strategies from the pure text domain, iteratively rewriting multimodal instructions to increase data difficulty. However, this instruction evolution is usually static or based on heuristic rules, failing to decouple visual and logical difficulty and unable to generate instructions in a targeted manner based on the model's real-time capability boundaries.

[0034] Traditional curriculum learning and data scheduling: This approach focuses on training data scheduling strategies, attempting to provide data to the model in order of increasing difficulty to optimize the training process. Technical implementations typically rely on static difficulty definitions (such as rule-based heuristic hierarchies) or simple adaptation criteria (such as perplexity or loss values) to arrange the order of training samples. Representative works include heuristic hierarchical methods or adaptive methods based on perplexity feedback. However, these methods mostly rely on predefined static difficulty or fixed datasets, failing to track the capability boundaries of model evolution in real time. Furthermore, they are often based on a single scalar difficulty, unable to distinguish specific difficulties in vision and reasoning, resulting in insufficient training targeting.

[0035] The existing method has the following three drawbacks: Attribution ambiguity: Existing methods optimize vision and reasoning as a whole, failing to distinguish between perceptual defects (such as misreading charts) and logical fallacies (such as incorrect reasoning steps). Lack of targeted remedies: Existing course learning methods are often based on static difficulty heuristics, which cannot dynamically generate the training samples most needed by the model according to its real-time capability boundaries.

[0036] Inefficient data processing and blind expansion: Because the "weak link" cannot be identified, the model can only improve performance by blindly increasing the amount of data, resulting in wasted computing resources and slow convergence.

[0037] Figure 1 A flowchart is shown of a diagnostic optimization method for reducing computational resource consumption during training of multimodal large models according to an embodiment of the present invention.

[0038] like Figure 1 As shown, a method to reduce the computational resource consumption for training large multimodal models includes: Step 1: Use the evaluation sample set to evaluate the multimodal large model to be trained, collect erroneous samples of incorrect answers generated by the multimodal large model to be trained, and determine the cause of the incorrect answer as a visual perception defect or a logical reasoning error.

[0039] Step 2: Generate corresponding evaluation samples for each visual perception defect and logical reasoning error.

[0040] Step 3: For erroneous samples, use correct reference thought chains as training samples to train the multimodal large model to be trained, so as to improve the model's ability to process and reason about multimodal mathematical problems. The thought chain format is image description - reasoning process - final answer.

[0041] Step 4: Use the updated evaluation sample set to perform the next round of evaluation on the trained multimodal large model.

[0042] Repeat steps 1 through 4 to form a diagnosis-synthesis-evolution cycle until the model's accuracy reaches the required level. Stop optimizing the model when the accuracy of the trained multimodal large model in generating answers exceeds a threshold. The order of steps 2 and 3 can be interchanged. Steps 1 through 3 are described in detail below.

[0043] Step 1: Use the evaluation sample set to evaluate the multimodal large model to be trained, collect erroneous samples of incorrect answers generated by the multimodal large model to be trained, and determine the cause of the incorrect answer as a visual perception defect or a logical reasoning error.

[0044] In one embodiment, the multimodal large model to be trained is used as the student model, and the trained multimodal large model is used as the teacher model. An evaluation sample set containing multiple samples is input into the student model and the teacher model respectively. Each sample in the evaluation sample set contains a question and a correct answer; the question includes image and text questions. For each sample in the evaluation sample set, the student model outputs a student thought chain, and the teacher model outputs a reference thought chain. The analyzer aligns and compares the student thought chain of the student model and the reference thought chain of the teacher model, identifies the student thought chain with incorrect answers, and diagnoses the error as a visual perception defect or a logical reasoning error. The thought chain format is image description - reasoning process - final answer.

[0045] Collect samples of incorrect answers generated by the student model as error samples to form an error set.

[0046] In the error set, there are usually both error samples corresponding to visual perception defects and error samples corresponding to logical reasoning errors. In step 1, to resolve attribution ambiguity, the student model (the large multimodal model to be trained) is forced... During inference, it follows a specific generative sequence: (image description) → <reasoning>(Reasoning process) → <answer>(Final Answer). The complete reasoning process of the student model is formalized as a sampled structured trajectory. ,in , and These represent the generated image description, the reasoning process, and the predicted answer, respectively. Following an autoregressive generation method, the joint distribution is explicitly decomposed into two stages: visual perception and logical reasoning. In other words, the joint distribution of the generated answer is clearly divided into two stages: visual perception and logical reasoning. Its mathematical expression is as follows: , In the In a round-robin cycle, the probability distribution of the whole is true. It is explicitly decomposed into three consecutive stages: the first is by The modeling process involves a visual perception stage, followed by a logical reasoning stage based on the perception results. And ultimately through The mapping is to the probability of generating the answer.

[0047] Autoregressive models are the most common generation method for large models, where each generation step depends on previously generated content. In this invention, the model first identifies the image to obtain an image description, then performs reasoning based on the image description and the question, and finally generates the answer.

[0048] To implement accurate error attribution, this invention introduces a comparative diagnostic mechanism for teacher guidance. Specifically, it utilizes a higher-performance multimodal large model as the teacher model (denoted as...). This process generates a reference thought chain. During this process, it is mandatory that the answers generated by the teacher's model strictly adhere to the standard answers to ensure the reliability of the reference thought chain.

[0049] The teacher model can be a more robust closed-source model such as GPT-5.2, Claude 4.5, or Sonnet, or it can be a manually annotated correct path.

[0050] The student thought process chain generated by the student model and the reference thought process chain generated by the teacher model are imported into the diagnostic system in parallel to identify the incorrect answers generated by the student model (i.e., At the same time, the cause of the error was diagnosed.

[0051] The diagnostic system utilizes an analyzer integrated with structured verification instructions. (Based on the LLM as Judge mechanism, using a large language model as an external judge for evaluation, to assess the output quality of the model), a rigorous two-stage comparative diagnosis is performed on the student's thought process and the reference thought process to accurately locate the failure link.

[0052] Align and compare the final answer in the student's thought process with that in the reference thought process. If they are different, further investigate the reasons for the error, including: Visual perception check: The analyzer first aligns and compares the student's thought chain generated by the learned model with the image description in the standard reference thought chain. If semantic illusions (e.g., confusion of geometric categories), topological errors (e.g., missing key points), or omissions of basic visual premises (e.g., failure to recognize specific symbols) are detected, the process is immediately terminated and marked as a visual perception defect. ).

[0053] Logical consistency check: After passing the visual perception check, any remaining errors are attributed to errors in the reasoning process through elimination. Since the visual perception has been judged to be accurate, the failure at this point is essentially due to a breakdown in logical reasoning. Therefore, these instances are marked as logical reasoning errors (…). ).

[0054] By utilizing the decoupling and comparison structure described above, the root cause of the error can be precisely isolated, enabling the analyzer to output a clear attribution vector. These correspond to visual perception defects and logical reasoning errors, respectively.

[0055] The analyzer's structured verification instructions are as follows: Role and Task: You are an expert evaluator analyzing students' mistakes in math problems. Your task is to identify the root cause of the errors when students solve problems incorrectly.

[0056] Error category: Image caption error: Students misunderstand or misdescribe visual elements in an image.

[0057] Reasoning Error: A student makes a logical or calculation error in the reasoning process.

[0058] Analysis process: 1. If the question includes an image, you can view it and analyze it in conjunction with the student's description.

[0059] 2. Compare the student's image caption with the actual image to check for any visual misunderstandings.

[0060] 3. Analyze the student's reasoning steps to check for logical or calculation errors.

[0061] 4. Determine which error is more fundamental: If the image description is incorrect, even perfect reasoning will not yield the correct answer → caption_error; If the image description is correct but the reasoning is flawed → reasoning_error.

[0062] Output format and key requirements: You may provide detailed analysis, but you must end your response with exactly one of the following tags. This tag is mandatory and must appear at the very end of your analysis: <caption_error> If the main problem is a misunderstanding of the visuals / images.

[0063] <reasoning_error> If the main problem lies in logical reasoning or calculation.

[0064] Step 2: Generate corresponding evaluation samples for each visual perception defect and logical reasoning error, update the evaluation sample set, and conduct the next round of evaluation on the trained model.

[0065] In one embodiment, based on the diagnostic results, a generator and code drawing tools are used to generate training samples with increased difficulty on two orthogonal dimensions: visual complexity and reasoning depth, namely visual upgrade samples or reasoning upgrade samples.

[0066] To prevent overfitting, a homogeneous generalization technique is employed to generate a homogeneous generalization variant for each erroneous sample in the error set. Compared to the corresponding erroneous sample, the homogeneous generalization variant maintains the same visual and inference complexity while only equivalently replacing the numerical distribution or semantic background (such as object category or scene task) of the original erroneous sample.

[0067] The generator uses visual ascent, inference ascent, and isomorphic generalization techniques to construct visually upgraded samples, inference upgraded samples, and isomorphic generalization variants.

[0068] This invention leverages a powerful generative large model (generator) to dynamically construct the next round of training data. It employs a procedural generation method, generating Python plotting code to control the visual image and generating textual questions to control the question logic. The Python plotting code can be input into a code-based plotting tool to generate the image.

[0069] In one embodiment, the generator currently generates Python Matplotlib code, but it can also be changed to generate TikZ, SVG code, or other graphics rendering languages, as long as it can achieve parameterized control over image elements.

[0070] This represents a visual upgrade sample. Represents an image. To indicate a problem, Textual problems with reasoning difficulty equivalent to that of erroneous samples (isomorphic reasoning problems). This indicates the correct answer. This represents an upgraded inference sample. Represents an image. The image is the same as the erroneous sample. This represents a problem with increased reasoning complexity. This indicates the correct answer. Indicates an isomorphic generalization variant. Images representing variants, The issue of variants, This indicates the correct answer.

[0071] The visual upgrade samples, inference upgrade samples, and isomorphic generalization variants are merged to obtain the final merged sample set: This is used for the next round of model iteration.

[0072] To address visual perception deficiencies, visual ascent is employed to generate visual upgrade samples, following the rules below: The logical reasoning path and mathematical principles of the problem will remain unchanged. To adapt to the upgraded visual complexity of the images, the numerical values ​​in the problem may be modified. The upgraded images and new problems will be generated by the model together.

[0073] Based on the current level of visual complexity, generative large models utilize prompts to dynamically adjust the drawing generation logic, thereby increasing the visual difficulty. ).

[0074] Compared to the original image, the visual complexity of the generated image increases by at least one level.

[0075] As shown in Table 1, the visual complexity system is subdivided into 7 levels, constructing an progressive gradient from basic perception to high-order reasoning: the starting point is a simple labeling task that only requires recognizing explicit features (level 1), and the endpoint extends to high-order visual cognitive tasks that require performing spatial transformations or mental rotations (level 7).

[0076] Table 1. Definition of visual complexity levels.

[0077] To address logical reasoning errors, reasoning escalation is used to generate reasoning upgrade samples, with the following rules: Keep the visual image and geometric topology of the problem unchanged (fixed) Use the original image or generate a similar image (in some scenarios, the original image may not be suitable for the upgraded problem, so a suitable image needs to be generated).

[0078] Using a high-performance large language model (generator), text question rewriting is driven by prompt words, aiming to induce a dynamic evolution of reasoning depth. Based on the 13 levels of reasoning standards defined in Table 2, the system covers the entire task path from basic formula extraction (Level 1) to multi-dimensional logic chain reconstruction and Olympiad-level strategy planning (Level 13). The prompt words are referenced from the examples of prompt words used to generate the visual upgrade samples mentioned above.

[0079] Table 2. Definition of Reasoning Difficulty Levels.

[0080] Examples of synthesized prompts applicable to visual upgrade samples and inference upgrade samples: System Role and Constraints: You are a professional math problem creator, responsible for creating challenging problems in English. All content must be in English only. Output only raw JSON (do not include Markdown code block tags or explanations).

[0081] Input data template: Original question: {problem} (Please fill in the image and text problem of the error sample here) Original answer: {answer} (Please enter the correct answer corresponding to the incorrect sample here) Dynamic guidance (injected based on upgrade type): Instruction: {difficulty_instruction} (Enter the difficulty name here) Target level: {level_guidance} (Enter the specific level here) Reference scales: {REASONING_DIFFICULTY_TABLE, Reasoning Difficulty Table} / {VISUAL_DIFFICULTY_TABLE, Visual Difficulty Table} Task and key requirements: Create an upgraded version that matches the target reasoning or visual level (approximately one level more difficult than the incorrect sample).

[0082] Keep the same category and general theme (if it is a visual upgrade, keep the same reasoning complexity level; if it is a reasoning upgrade, keep the same visual complexity level).

[0083] Maintain mathematical correctness and ensure solvability.

[0084] Provide clear and correct answers.

[0085] Python code for visualization must always be generated—the image_code field is required and cannot be empty.

[0086] Output and Python coding style: Output a JSON object containing the fields `question`, `answer`, and `image_code`. Use only matplotlib.pyplot and numpy. The code must be self-contained, executable, and end with `plt.savefig('problem.png')`.

[0087] All text in the charts must be in English. Use standard color formatting. Chinese characters are not allowed.

[0088] Important: The image_code field must never be empty. Visualization complexity must adapt to the upgrade type (this mainly refers to image generation; for visual upgrades, images need to be significantly more complex, while for inference upgrades, images should maintain a similar level of complexity).

[0089] Step 3: For error samples, use the reference thought chain of the teacher model as training samples to optimize the multimodal large model to be trained, and use the standard reference thought chain path as a supervision signal to perform supervised fine-tuning of the student model.

[0090] The optimization objective focuses on the student model's current "active failure regions," achieving targeted improvement of the model's capabilities by minimizing the loss function for these specific samples. , in Represents the training sample set, This represents the reference thought chain generated by the teacher model. Represents the error set. Images representing training samples, The text problem representing the training samples, Indicates the first Wheel model parameters, Indicates the learning rate. Represents the gradient of the parameters. This represents the loss function.

[0091] This invention is based on Figure 2 The pseudocode shown demonstrates the feasibility of diagnostic optimization methods.

[0092] This invention establishes a robust learning cycle: the diagnostic phase exposes weaknesses, the evolution phase uses the correct thought process to fix weaknesses, and the synthesis phase prepares the next round of evaluation samples to verify the repair of old errors and the conquest of new heights.

[0093] This invention has been systematically verified through experiments, demonstrating its significant feasibility and superiority. Specific experimental results are as follows: Experimental Setup and Benchmarks: Evaluations were conducted on seven mainstream multimodal leaderboards, including MathVista (visual scene mathematical reasoning benchmark), MathVerse (multimodal mathematical competition benchmark focusing on medium to high difficulty competition problems), and Dynamath (dynamic robustness testing benchmark, the core of which is "dynamic variations of the same problem"). The base models used in the experiments were all initialized from open-source multimodal large models and trained using the method of this invention.

[0094] Extremely high data efficiency: The model trained with only about 7,000 samples achieved an accuracy of 78.6% on MathVista, which is better than the Euclid baseline (75.7%) trained with 30,000 samples and surpasses MMOpenReasoner trained with 800,000 samples.

[0095] Beyond baseline and closed-source models: In the Dynamath test, the model optimized and trained using the method of this invention improved by 12.1% compared to the untrained model (Qwen3-VL-8B-Instruct), and successfully beat top closed-source models such as Gemini 2.5 Flash.

[0096] Cost and efficiency analysis: Compared with traditional methods, this invention significantly reduces the training threshold and resource consumption. Supervised fine-tuning (SFT) training time is reduced by 64%, and computational resource usage is reduced by more than 50%.

[0097] This invention significantly improves the model's inference ability by using only a few thousand high-quality samples through error attribution and a targeted synthesis mechanism of training and evaluation samples. It achieves or outperforms existing models that rely on large-scale data training on multiple public benchmarks, significantly reducing data scale and training resource requirements, saving computing resources, and improving parameter utilization efficiency and overall training efficiency.

[0098] Achieving a Leap in Performance with Small-Parameter Models: This invention, by constructing high-quality, targeted training samples, enables models with smaller parameter sizes (such as 4B and 8B) to achieve significant improvements in inference capabilities. In some benchmark tests, these models can reach or even surpass those of specially optimized larger-scale models or Thinking versions of the same size. The results demonstrate that this invention can achieve a substantial enhancement in model inference capabilities without relying on parameter scaling.

[0099] Each step of the method in this invention requires computer execution. The model processes multimodal information in the specific form of image data (such as geometric figures, function graphs, and statistical data charts) and mathematical text problems, and outputs accurate numerical answers and structured reasoning processes for multimodal mathematical problems. This method improves the accuracy of multimodal mathematical reasoning, significantly saves computational resources, reduces data construction costs, and improves training efficiency compared to traditional optimization methods.

[0100] In addition to mathematical reasoning, the method of this invention can also be applied to multimodal fields that require a combination of strong visual and strong logical reasoning, such as solving physics problems, analyzing chemical molecular formulas, and recognizing engineering drawings.

[0101] This solution addresses the technical problems of ambiguous error attribution, excessive computational resources, and data redundancy in existing multimodal large-scale model training processes. It relies on a computer system to automatically execute a closed loop of "evaluation-diagnosis-synthesis-evolution." Through teacher-student model comparison and structured thought chain alignment technology, it accurately distinguishes between visual perception defects and logical reasoning errors. For visual perception defects and logical reasoning errors, it uses generators and code drawing tools to generate samples for the next round of evaluation. For erroneous samples, it uses correct reference thought chains as training samples to train the multimodal large-scale model to optimize model performance. This process is closely integrated with computer data processing, reducing computational overhead and data construction costs, significantly improving the efficiency of training resource utilization, and achieving adaptive evolution of model capabilities. Because it employs specific technical means and produces objectively verifiable technical effects, it constitutes a technical solution for solving technical problems and meets the requirements for patent protection.

[0102] Figure 3 A computer system 100 for implementing the method according to the invention is shown. Unless otherwise specified, the method according to the invention can be implemented in... Figure 3 The invention may be implemented in the computer system 100 shown to achieve its objectives, or the invention may be implemented in a distributed manner across multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0103] like Figure 3 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a Universal Serial Bus (USB) interface 112. The system bus 101 can be any of several bus architecture types, such as a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. The system bus 101 is used for communication between the various bus devices. In addition to... Figure 3 Other bus devices or interfaces besides those shown are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 may, for example, store basic input / output system (BIOS) data for basic routines used to implement information transfer at startup, while RAM 104 provides the system with faster access speed running memory. Computer system 100 also includes hard disk drive 109 for reading and writing to hard disk 110, optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 may, for example, store operating system and application programs. The drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 100. Computer system 100 may also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may also include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. Furthermore, the computer system 100 may also include a network interface 114 for network connectivity, wherein the network interface 114 can be connected to the Internet 116 via a network device such as a router 115, wherein the connection can be wired or wireless. Additionally, the computer system 100 may also include a Universal Serial Bus (USB) interface 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as microphones, cameras, etc.

[0104] When the present invention is Figure 3 When implemented on the computer system 100, it can improve model optimization efficiency, save computing resources, reduce data construction costs, and shorten model training time.

[0105] Furthermore, the embodiments can be provided as computer program products that may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines such as a computer, computer network, or other electronic equipment, may cause one or more machines to perform operations according to the embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0106] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0107] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.< / answer> < / reasoning>

Claims

1. A method for reducing the computational resource consumption during training of large multimodal models, characterized in that, include: The evaluation sample set is used to evaluate the multimodal large model to be trained, and the error samples generated by the multimodal large model to be trained are collected. At the same time, the cause of the error is determined to be a visual perception defect or a logical reasoning error. For each visual perception defect and logical reasoning error, generate a corresponding evaluation sample and update the evaluation sample set; The process of generating corresponding evaluation samples for each visual perception defect and logical reasoning error includes: generating visual upgrade samples or reasoning upgrade samples using a generator and a code drawing tool, wherein the generator generates drawing codes and text questions based on prompts, and the code drawing tool draws images using the drawing codes; using isomorphic generalization technology, generating isomorphic generalization variants for each error sample, wherein the isomorphic generalization variants maintain the same visual complexity and reasoning complexity as the corresponding error sample, and only make equivalent substitutions to the numerical distribution or semantic background of the original error sample; the visual upgrade samples, reasoning upgrade samples, and isomorphic generalization variants all include questions and correct answers, and the questions include image and text questions; For incorrect samples, the correct reference thought chain is used as the training sample to train the multimodal large model to be trained. The format of the reference thought chain is image description - reasoning process - final answer. The trained multimodal large model is evaluated again using the updated evaluation sample set.

2. The method for reducing computational resource consumption during training of large multimodal models according to claim 1, characterized in that, The training multimodal large model is evaluated using an evaluation sample set. Error samples of incorrect answers generated by the training multimodal large model are collected, and the causes of these errors are identified as visual perception defects or logical reasoning errors, including: The multimodal large model to be trained is used as the student model, and the trained multimodal large model is used as the teacher model. An evaluation sample set containing multiple samples is input into the student model and the teacher model respectively. Each sample in the evaluation sample set contains a question and the correct answer. The questions include image and text questions. For each sample in the evaluation sample set, the student model outputs a student thought chain, and the teacher model outputs a standard reference thought chain. The analyzer aligns and compares the student's thought process with the reference thought process, identifies the student's thought process with incorrect answers, and diagnoses the cause of the error as a visual perception defect or a logical reasoning error. The format of the student's thought process is image description - reasoning process - final answer. Collect samples of incorrect answers generated by the student model as error samples.

3. The method for reducing the computational resource consumption during training of large multimodal models according to claim 2, characterized in that, The student model is forced to follow the following generation sequence during inference: image description → inference process → final answer; The complete reasoning process of the student model is formalized as a sampled structured trajectory. ,in , and These represent the generated image description, the reasoning process, and the predicted answer, respectively.

4. The method for reducing the computational resource consumption for training multimodal large models according to claim 2, characterized in that, The analyzer aligns and compares the student's thought process with the standard reference thought process based on the prompt words, including: Align and compare the final answer in the student's thought process with that in the reference thought process. If they are different, further investigate the reasons for the error, including: First, a visual perception check is performed: the analyzer aligns and compares the student's thought chain generated by the learned model with the image description in the reference thought chain. If semantic illusion, topological error or omission of basic visual premise is detected, the comparison is terminated and marked as a visual perception defect. After passing the visual perception check, the remaining errors were attributed to errors in the reasoning process and marked as logical reasoning errors.

5. The method for reducing the computational resource consumption during training of large multimodal models according to claim 1, characterized in that, For erroneous samples, the reference thought chain of the teacher model is used as a training sample to train the large multimodal model to be trained.

6. The method for reducing computational resource consumption during training of large multimodal models according to claim 1, characterized in that, The visual complexity of the images in the visual upgrade samples is higher than that of the images in the error samples with visual perception defects. The reasoning complexity of the text problem in the reasoning upgrade sample is higher than the reasoning complexity of the text problem in the error sample with logical reasoning error.

7. The method for reducing computational resource consumption during training of large multimodal models according to claim 6, characterized in that, The visual escalation method is used to generate visually upgraded samples, including: keeping the logical reasoning path and mathematical principles of the questions unchanged; based on the current visual complexity level, the generator uses prompts to dynamically adjust the drawing generation logic to increase the visual difficulty and obtain visually upgraded samples. The reasoning escalation method is used to generate reasoning escalation samples, including: keeping the visual image and geometric topology of the question unchanged; and using a generator to generate reasoning escalation samples based on clue words.

8. The method for reducing computational resource consumption during training of large multimodal models according to claim 1, characterized in that, The evaluation sample set is updated by adding visual upgrade samples, inference upgrade samples, and isomorphic generalization variants, and the trained model is evaluated in the next round.

9. The method for reducing the computational resource consumption for training multimodal large models according to claim 8, characterized in that, When the accuracy of the multimodal large model generated by training exceeds a threshold, the model optimization is stopped.

Images