Tool-integrated visual reasoning method, system and electronic device for chart understanding

Abstract

The embodiment of the application provides a tool integrated visual reasoning method, system and electronic equipment for chart understanding. The method comprises the following steps: constructing a mixed chart dataset comprising diversified parameter charts and real charts; generating dimensionally complementary candidate question and answer pairs based on the mixed chart dataset, and constructing a quality reinforced mixed dataset through the candidate question and answer pairs; performing integrated calling reasoning of an image cropping tool for enhancing local visual perception and a code calculation tool for reinforcing precision by a tool agent according to the mixed dataset, generating a cold start track interwoven with vision and calculation, and performing cold start supervision fine tuning on the tool agent by using correct tracks in the cold start track, so that the tool agent can integrate and call tools to perform visual reasoning for chart understanding. The embodiment of the application constructs a high-quality mixed dataset, and uses tool integrated visual reasoning to realize the ability of model planning and self-correction, instead of only imitating human text answers.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a tool-integrated visual reasoning method, system, electronic device, and storage medium for graph understanding. Background Technology

[0002] Graphs are a fundamental medium for presenting structured data in papers and reports, combining textual annotations with structured visual elements to convey quantitative relationships. Although current MLLMs (multimodal large language models) perform well in general visual understanding and reasoning, they still have shortcomings in graph reasoning because graph reasoning requires fine-grained visual perception and precise relational and numerical reasoning on densely structured information.

[0003] To solve the graph reasoning problem, existing techniques typically use general-purpose multimodal large language models: 1. By pre-training on massive amounts of image and text data, the model can directly input chart images and text prompts, and generate answers based on its internal knowledge. Some advanced models (such as GPT-4o) support calling code interpreters, but still rely on overall visual encoding for recognizing specific chart details.

[0004] 2. Chart-specific understanding models: These typically employ methods such as "chart-to-table" or thought chain approaches. They convert charts into text form before reasoning, or use a large number of chart-question-answer pairs for end-to-end training during the fine-tuning phase.

[0005] 3. Model based on "image thinking": Try to introduce visual assistance (such as drawing circles or markings on images) in the reasoning process to help the model focus on key areas by adding visual cues.

[0006] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology: General multimodal large language model: The training data is mostly synthesized by the program, with a single style and simple layout, lacking the visual complexity of real-world charts (such as multiple subgraphs, complex layouts, and non-standard layouts).

[0007] Chart-specific understanding models often rely on program-generated data with limited style and simple layout, lacking the visual complexity of real-world charts. Furthermore, visual attribute information is frequently lost during the conversion of charts into text tables.

[0008] Models based on "image thinking" introduce visual markers but lack deep integration with "external tools" (such as precise calculation tools) and are mostly designed for natural images, performing poorly in scenarios such as charts that require precise coordinates and values.

[0009] In summary, the existing technologies lack precise external tool localization: current methods rely excessively on the model's "internal reasoning" capabilities. For structured data like charts, the model cannot perform precise calculations like a calculator, nor can it observe local details at high resolution like a magnifying glass. There is a discrepancy between training data quality and realism: the synthetic data generation pipeline has limitations, resulting in charts that lack diversity, are overly simplistic, and cannot simulate the complex noise and diverse layouts of charts in real scientific papers. The training paradigm is also limited: most models only undergo supervised fine-tuning, without using reinforcement learning to specifically optimize strategies for "when to use tools" and "how to plan in multiple steps." Summary of the Invention

[0010] In order to at least address the discrepancies between training data quality and realism, the limitations of training paradigms, and the reliance on models in existing graph inference technologies.

[0011] In a first aspect, embodiments of the present invention provide a tool-integrated visual reasoning method for chart understanding, comprising: Construct a hybrid chart dataset that includes diverse parametric charts and real-world charts; Based on the hybrid graph dataset, candidate question-answer pairs with complementary dimensions are generated, and a quality-enhanced hybrid dataset is constructed using these candidate question-answer pairs. The categories of the candidate question-answer pairs with complementary dimensions include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning. The tool agent performs integrated invocation reasoning based on the hybrid dataset for image cropping tools to enhance local visual perception and code computation tools to improve accuracy, generating a cold start trajectory that interweaves vision and computation. The correct trajectory in the cold start trajectory is used to supervise and fine-tune the cold start of the tool agent, enabling the tool agent to integrate and invoke tools for visual reasoning of graph understanding.

[0012] In a second aspect, embodiments of the present invention provide a tool-integrated visual reasoning system for chart understanding, comprising: The Hybrid Chart Data Building Module is used to construct hybrid chart datasets that include diverse parametric charts and real-world charts. A question-answering agent is used to generate candidate question-answer pairs with complementary dimensions based on the hybrid graph dataset, and to construct a quality-enhanced hybrid dataset through the candidate question-answer pairs. The categories of the candidate question-answer pairs with complementary dimensions include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning. The tool agent is used to perform integrated invocation reasoning based on the hybrid dataset for image cropping tools to enhance local visual perception and code computation tools to improve accuracy, generating a cold start trajectory that interweaves vision and computation. The correct trajectory in the cold start trajectory is used to supervise and fine-tune the cold start of the tool agent, enabling the tool agent to integrate and invoke tools for visual reasoning of graph understanding.

[0013] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the tool-integrated visual reasoning method for graph understanding according to any embodiment of the present invention.

[0014] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the tool-integrated visual reasoning method for graph understanding according to any embodiment of the present invention.

[0015] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the tool-integrated visual reasoning method for graph understanding according to any embodiment of the present invention.

[0016] The beneficial effects of this invention are as follows: This method constructs a dataset containing tool call trajectories, directly supervising the training of the agent model. An image cropping tool is introduced to enhance local visual perception, and a code calculation tool is used to ensure the accuracy of numerical calculations. The model can, like a human expert, first zoom in to examine details and then use a calculator to calculate the data. Reinforcement learning training is performed using the GRPO algorithm, rewarding the model for calling the correct tool at the right time, thereby endowing the model with planning and self-correction capabilities, rather than simply imitating human textual responses. This makes the tool integration and visual perception capabilities universally applicable. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a tool-integrated visual reasoning method for chart understanding provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the data synthesis process of CHARTOOL-100k, a tool-integrated visual reasoning method for chart understanding, provided in an embodiment of the present invention. Figure 3 This is a statistical diagram of charts and question-and-answer data in CHARTOOL-100k, a tool for chart understanding and integrated visual reasoning method, provided by an embodiment of the present invention. Figure 4 This is a schematic diagram comparing the quality of synthetic datasets for a tool-integrated visual reasoning method for chart understanding, provided in an embodiment of the present invention. Figure 5 This is a performance comparison diagram of different chart benchmark tests for a tool-integrated visual reasoning method for chart understanding, provided by an embodiment of the present invention (divided into two categories: real-world (CharXiv, ChartQA, ChartQAPro) and synthetic (ChartBench, ChartX, ReachQA). The best result in each column is highlighted in bold, the second best result is indicated by underline, and the relative improvement (Δ) relative to the baseline model is shown in red). Figure 6 This is a schematic diagram of the off-domain visual reasoning benchmark test results of a tool-integrated visual reasoning method for graph understanding provided in an embodiment of the present invention; Figure 7 This is an ablation experiment diagram illustrating the effectiveness of tool integration reinforcement learning in a tool integration visual reasoning method for graph understanding, provided by an embodiment of the present invention. Figure 8 This is a schematic diagram showing the performance comparison of a tool-integrated visual reasoning method for chart understanding using different training data sources, according to an embodiment of the present invention. Figure 9 This is a schematic diagram of an ablation experiment of a training data source for a tool-integrated visual reasoning method for chart understanding, provided in an embodiment of the present invention. Figure 10 This is a schematic diagram of an ablation experiment of a tool-integrated visual reasoning method for chart understanding provided in an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the tool call distribution under different benchmarks of a tool integration visual reasoning method for graph understanding provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of a CHARTOOL example for a tool-integrated visual reasoning method for chart understanding, provided in an embodiment of the present invention; Figure 13This is a schematic diagram of a CHARTOOL example using a cropping tool and a code calculation tool, provided by an embodiment of the present invention, for a tool-integrated visual reasoning method for chart understanding; Figure 14 This is a schematic diagram of the structure of a tool-integrated visual reasoning system for chart understanding provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of an embodiment of an electronic device for integrating visual reasoning tools for graph understanding, provided as an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 The diagram shows a flowchart of a tool-integrated visual reasoning method for chart understanding according to an embodiment of the present invention, including the following steps: S11: Construct a hybrid chart dataset that includes diverse parametric charts and real charts; S12: Generate candidate question-answer pairs with complementary dimensions based on the hybrid chart dataset, and construct a quality-enhanced hybrid dataset through the candidate question-answer pairs. The categories of the candidate question-answer pairs with complementary dimensions include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning. S13: The tool agent performs integrated invocation reasoning based on the hybrid dataset to use image cropping tools for enhancing local visual perception and code computation tools for improving accuracy, generating a cold start trajectory that interweaves vision and computation. The correct trajectory in the cold start trajectory is used to supervise and fine-tune the cold start of the tool agent, enabling the tool agent to integrate and invoke tools for visual reasoning of graph understanding.

[0021] In this embodiment, existing technologies for graph reasoning typically employ methods such as bloating up data volume, increasing model parameters, and OCR post-processing. However, these methods do not improve the generation mechanism, brute-force the reasoning ability, and convert the graph into a pure text problem, ignoring the importance of visual reasoning. Considering these shortcomings, this method utilizes reinforcement learning to unify visual cropping and code computation within the same framework for end-to-end optimization, while simultaneously integrating the accuracy of synthetic data with the complexity of real data through dual-source multi-agent collaboration.

[0022] For step S11, this method utilizes two complementary sources (e.g., diverse parametric charts and real charts) to construct a hybrid chart dataset.

[0023] like Figure 2 Part A, Hybrid Chart Data Construction, includes coded and rendered synthetic charts for controlled diversity and accurate labeling, as well as real-world charts mined from literature (e.g., books, journals, scientific papers) to represent the visual complexity of the real world.

[0024] As one implementation, the construction of a hybrid chart dataset including diverse parametric charts and visually realistic charts includes: Using a large artificial intelligence model, drawing code is generated based on a variety of set parameters, and a composite chart with diverse parameters is rendered. The diverse parameters include: chart type, layout, and number of sub-charts. Data mining is performed on the literature to incorporate the visual complexity, layout variations, and domain diversity of the charts, resulting in realistic charts with corresponding contextual text. Visual quality and semantic integrity are filtered for the diverse parameter charts and the real charts to obtain a hybrid chart dataset suitable for chart reasoning.

[0025] The visual quality screening includes: screening for overlapping, misalignment, and readability of text in charts; The semantically complete filtering includes checking whether the self-explanatory elements of the chart are complete, wherein the chart includes axes, legend and labels.

[0026] In this implementation, the method utilizes a large model as a drawing agent to generate drawing code and render a composite chart based on various set parameters (chart type, layout, number of subgraphs). Specifically, to ensure consistency with the distribution of real charts, this method reproduces the real chart using Python code generated by MLLM and manually verifies its consistency with the original chart, thereby constructing a reference code library. For example, in a specific implementation process, Reference code was extracted from a reference code library, along with the necessary concepts (character theme, number of subgraphs, chart layout, and type), and input into a drawing agent. The drawing agent then generated 112 reference charts, including single-chart and multi-subgraph layouts. To promote thematic diversity, this method employs a role-based sampling strategy. Under the conditions of sampling attributes such as the number of subgraphs, layout, and chart type, the drawing agent generates drawing code. This code is executed in a sandbox environment and iteratively corrected and improved until a valid chart is generated. This process enables scalable synthesis of diverse charts while maintaining the authenticity of the chart structure and the accuracy of the underlying data.

[0027] This method also mines real-world charts from literature (e.g., using arXiv papers for the illustration) and extracts the corresponding contextual text. Specifically, to expose the agent's large model to realistic chart layouts and visual diversity, this method mines charts covering eight domains from literature, such as arXiv papers. MinerU (an open-source intelligent data extraction tool) is used to extract the chart images and their associated textual context.

[0028] The above steps, namely "synthetic charts based on code rendering," ensure diversity and label accuracy, while "realistic charts based on literature mining" ensure visual authenticity.

[0029] After obtaining the synthesized, diverse parametric charts and real charts, a high-quality chart library is selected based on two criteria: visual quality (clarity, overlap) and semantic integrity. This filtering process ensures both visual usability and semantic integrity of the charts. Specifically, each candidate chart is scored based on two aspects using MLLM's evaluation criteria: (1) visual quality, deducting points for severely overlapping, misaligned, and unreadable text; and (2) semantic integrity, checking whether the chart possesses the basic elements required for self-explanation, such as axes, legends, and labels. Charts that do not meet either criterion are eliminated, thus generating the final high-quality chart image database. This hybrid chart dataset can then be applied to the chart reasoning of this method.

[0030] In step S12, after obtaining a high-quality graph and image database, this method also needs to generate high-quality question-answer pairs to address the issues of limited and low-quality training data. Simply put, the question-answering agent generates question-answer pairs of varying difficulty and type (including recognition-based and reasoning-based pairs) based on the graph and image metadata. It's important to note that, considering the training of the tool agent in subsequent steps, the generation of question-answer pairs aims to produce complementary recognition-based pairs for element recognition and structure extraction, as well as reasoning-based pairs for visual analysis and numerical reasoning.

[0031] Specifically, such as Figure 2Part B, Question Answer Generation, provides a graph image and its associated context. The multimodal question answering agent generates candidate question-answer pairs across two complementary categories: identification questions for self-explanatory element recognition and structure extraction, and reasoning questions requiring multi-step visual analysis and numerical reasoning. To achieve diversity and complexity in the questions, this method conditions the question answering agent by extracting fine-grained analytical aspects from a question-answer metadata pool, thereby constructing the questions. These aspects explicitly indicate the type of visual or numerical reasoning required, such as legend interpretation, cross-subgraph comparison, trend estimation, or multivariate aggregation. By sampling these fine-grained aspects and conditionally setting the generation process accordingly, the generated questions cover a wide range of reasoning behaviors with varying levels of difficulty. Through these steps, high-quality question-answer pairs are obtained.

[0032] To ensure the accuracy of subsequent reinforcement learning, as one implementation method, the construction of a hybrid dataset for quality reinforcement using the candidate question-answer pairs includes: The candidate question-answer pairs are subjected to quality verification to obtain a hybrid dataset of categories and quality enhancement. The quality verification includes consistency verification, inference verification, voting denoising verification, and difficulty score of recognition inference.

[0033] In this embodiment, after candidate question-answer pairs are generated, a quality-enhanced hybrid dataset is constructed using these candidate pairs. This method employs an inspector to filter out illusions and retain reliable supervisory information, performing four-fold verification: consistency between the QA and the image, inference logic verification, majority voting denoising, and difficulty scoring. Specifically, 1. Consistency alignment of the question-answer images is performed to remove questions that rely on non-visual knowledge or unverified claims; 2. Inference logic verification ensures correct answers and consistency between the inference and graphical evidence; 3. Majority voting is used to discard ambiguous question-answer pairs with low consistency; 4. Difficulty scores are assigned, and questions requiring non-simple visual and numerical reasoning are retained. After the above quality verification, a high-quality question-answer dataset is obtained, which this method refers to as the CHARTOOL-100k dataset. This method... Figure 3 The data statistics are displayed in the document.

[0034] For step S13, this method introduces TIR (tool-integrated reasoning) for graph understanding. It is performed alternately by a multilingual logic model (MLLM) between internal reasoning and external tool invocation. Given an input graph image I and a question q, the model follows a policy. The process iteratively generates inference markers and selects tool actions based on the current context. This interaction continues until the model outputs the final answer. Formally, this generates a trajectory: in, This represents the inference marker for step i. Represents the selected tool action (e.g., image cropping or calculation). y represents the observation returned after the tool executes, and y is the final predicted answer. Observation This may include text output or processed visual input, which will be added to the model context for subsequent inference. In practice, strong models (such as Qwen3-VL-235B) can be used as tool agents to perform graph understanding for tool integration inference.

[0035] This method extracts samples from the CHARTOOL-100k dataset determined in step S12 above to construct a cold-start dataset of executable trajectories. Trajectories with tool calls (clipping, code execution) are generated for this dataset, and binary annotations are applied to these trajectories to distinguish between "successful" and "failed" trajectories. This step is crucial for ensuring data quality and selecting the correct trajectories. This is because, in the absence of historical behavior data of the target user or agent, initial, high-quality interaction trajectories are generated through synthetic techniques to help the model quickly initiate the learning process.

[0036] Specifically, in the initialization of the model cold start, this method extracts samples from CHARTOOL-100k to construct a cold start dataset of executable trajectories (e.g., ...). Figure 2 (C. Cold Start Trajectory in the diagram). For each graph problem, the system prompts the tool agent to solve the problem step by step and invokes image cropping or computation tools as needed. Each tool invocation is executed in a code sandbox, and the output is appended to the context for subsequent inference. The complete sequence of inference steps, tool invocations, and observations constitutes a trajectory. This method only retains those trajectories that execute successfully and ultimately provide the correct answer, resulting in approximately 9000 reliable demonstration data points for cold start supervised fine-tuning, enabling the tool agent to integrate tool invocations for visual inference in graph understanding. It is worth noting that this method integrates the two most critical tools for graph understanding: an image cropping tool for achieving fine-grained visual perception, and a code computation tool for performing explicit numerical operations such as aggregation and statistical analysis. This design allows the model to seamlessly interweave visual localization and precise computation during inference.

[0037] As can be seen from this implementation method, this method constructs a dataset containing tool call trajectories, directly supervises the training of the agent model, and fine-tunes the tool agent model through supervision, enabling the tool agent model to learn the syntax format of tool calls and attempt to write code or crop images. The training speed is fast and the resource consumption is small.

[0038] As one implementation method, in order to make the tool agent of this method more generalizable, in this implementation method, the tool agent is integrated with reinforcement learning for calling tools by using group relative strategy optimization, so as to improve the tool agent's tool usage efficiency and multi-step reasoning ability.

[0039] The reward function of the group relative strategy optimization ,in, To score the accuracy of the reasoning answer, Score based on structure and format. The score is based on the use of the tool. The indicator function is set to 1 when the reasoning answer is correct, and 0 otherwise. and Preset weights.

[0040] In this embodiment, after cold-start supervised fine-tuning, the method further utilizes reinforcement learning optimization strategies. This aims to improve tool usage efficiency and multi-step reasoning capabilities. The goal is to maximize expected reward along the trajectory. in, Used to evaluate the final answer y relative to the true answer. The quality of the reward is defined at the trajectory level based on the correctness of the answer, while intermediate inference and tool use are implicitly optimized via policy gradients.

[0041] Furthermore, this method employs GRPO (Group Relative Policy Optimization). Let... This represents the set of tag indices corresponding to inference steps, tool calls, and final answer tags that are not masked during RL (Reinforcement Learning) training. Tags outside of M (such as cue or observation tags) are masked during training.

[0042] In each update step, this method changes from the previous strategy. A set of G trajectories was sampled. The GRPO objective is defined as follows: in, , Indicates the clipping threshold. This represents the word-level advantage calculated by normalizing the trajectory-level reward within each group: For the training reward, the reward function designed in this method consists of three parts: answer accuracy ( (For example, the error between the predicted inference result output by the tool agent and the baseline inference result can be determined), format ( (e.g., bonus points for the format of the generated chart structure) and bonus points for tool usage ( (For example, the percentage of points awarded for correctly using the corresponding tool). The total reward is defined as: in, This is an indicator function that sets its value to 1 when the answer is correct and 0 otherwise, to ensure that the tool is only encouraged when the answer is correct. In experiments, this method empirically... Set it to 0.1, Set it to 0.2.

[0043] In this way, the aforementioned group-relative policy optimization algorithm further reinforces the learning of the tool agent through rewards. The policy planning ability brought by reinforcement learning further improves the generalization of the tool agent's understanding of charts.

[0044] This implementation demonstrates that image cropping tools are introduced to enhance local visual perception, while code-based computation tools ensure the accuracy of numerical calculations. The model can, like a human expert, first zoom in to examine details and then use a calculator to calculate the data. Reinforcement learning training using the GRPO algorithm rewards the model for calling the right tool at the right time, thus endowing it with planning and self-correction capabilities, rather than simply mimicking human textual responses. This makes the tool integration and visual perception capabilities universally applicable.

[0045] This paper presents experimental results demonstrating the effectiveness of our proposed method. We comprehensively evaluate our approach on CHARTOOL (Tool-Integrated Visual Reasoning for Chart Understanding) from three perspectives. First, we assess the quality of the synthetic dataset CHARTOOL-100k. Second, we evaluate its performance on a range of chart reasoning benchmarks and out-of-domain visual reasoning tasks. Third, we conduct an ablation study to analyze the impact of key design components.

[0046] Regarding the quality of synthetic datasets, such as Figure 3As shown, the dataset used in this method covers a variety of chart layouts, chart types, and question types. This method further evaluates the quality of the synthetic dataset CHARTOOL-100k through quantitative evaluation and model-based evaluation. Specifically, this method reports the average entropy of the chart images and uses GPT-5.2 to evaluate four key metrics: visual quality, question-answer-image alignment, answer correctness, and reasoning difficulty. For comparison, this method benchmarks CHARTOOL-100k against widely adopted synthetic chart datasets, including the ReachQA dataset and the ECD dataset.

[0047] like Figure 4 As shown, CHARTOOL-100k consistently outperforms ReachQA and ECD across all evaluation metrics. Notably, it achieves higher image entropy, indicating the presence of more complex layouts. Furthermore, it demonstrates superior visual quality and includes a larger proportion of challenging, high-difficulty reasoning problems. These results demonstrate that CHARTOOL-100k more accurately reflects the complexity and diversity of real-world graph analysis tasks.

[0048] Regarding the experimental setup, this method implements CHARTOOL using the Qwen2.5-VL architecture (including 3B and 7B parameter variants). Training is divided into two phases: first, SFT (supervised fine-tuning) is performed on the cold start trajectory, and then reinforcement learning is performed on CHARTOOL-100k.

[0049] This method comprehensively evaluates CHARTOOL across a range of different standard chart benchmarks, which are divided into: (1) Real-world benchmarking, including charts extracted from real-world sources such as the CharXiv dataset, ChartQAPro dataset and ChartQA dataset; (2) Synthetic benchmarks, including synthetic images or queries: ChartBench dataset, ChartX dataset, and ReachQA dataset. In addition, to evaluate the out-of-domain generalization ability of this method, three visual mathematical reasoning benchmarks were also included: MathVista, WeMath, and MathVerse.

[0050] Regarding the baseline for comparison in the experiment, this method compares CHARTOOL with three types of models: (1) proprietary and general-purpose multilingual large-scale language models, including GPT-4o, Claude 3.5 Sonnet, InternVL3-8B, Qwen3-VL-8B and Qwen2.5-VL series; (2) chart-specific models, including TinyChart, ChartGemma, ChartMoE and ECD-Qwen2.5-VL-7B; (3) image-based thinking models, including DeepEyes and Thyme.

[0051] To comprehensively evaluate the CHARTOOL of this method, this method, in the following aspects... Figure 5 The paper demonstrates its performance across benchmarks covering various chart types. CHARTOOL exhibits significant advantages across various benchmarks and model sizes. Based on quantitative results, the key observations of this approach are as follows: CHARTOOL achieves significant improvements over baseline methods in all tests. Compared to the Qwen2.5-VL backbone model, CHARTOOL achieves significant improvements across all evaluation metrics. On the challenging real-world benchmark CharXiv, CHARTOOL-3B and CHARTOOL-7B achieve absolute improvements of 10.1% and 8.0% in inference accuracy, respectively. Compared to previous chart-specific models, CHARTOOL demonstrates superior performance on real-world datasets and achieves significant improvements in average accuracy. These continuous improvements confirm that the tool ensemble approach of this method effectively enhances the model's interpretability and inference capabilities for chart data.

[0052] Despite using only 7 billion parameters, CHARTOOL-7B's performance is comparable to advanced proprietary models and large open-source models. Figure 5 As shown, CHARTOOL-7B achieved an average score of 66.52, outperforming GPT-4o (61.52), Claude 3.5 Sonnet (63.62), and the larger Qwen2.5-VL-72B (66.02). These results demonstrate that, compared to simply scaling up the model, the tool-integrated inference framework of this method provides a more parameter-efficient approach for graph inference.

[0053] CHARTOOL outperforms baseline "image-based thinking" models such as DeepEyes and Thyme. This advantage can be attributed to the high-quality training data built for our data engine, specifically tailored to the unique characteristics of scientific charts, such as high information density and multi-subgraph layouts. Unlike general inference models that treat charts as standard images, CHARTOOL benefits from fine-tuned instructions that explicitly model the structural relationships within complex subgraphs, enabling more accurate visual localization and inference.

[0054] To verify CHARTOOL's generalization ability beyond the chart domain, the proposed model was also evaluated on three common visual reasoning benchmarks: MathVista, WeMath, and Math-Verse. These benchmarks involve a combination of visual understanding and multi-step numerical and logical reasoning, and are not limited to chart-based inputs.

[0055] like Figure 6 As shown, despite not being trained on these tasks, CHARTOOL demonstrates performance improvements across multiple benchmarks. These results indicate that CHARTOOL's tool-integrated inference framework enables explicit numerical computation and sophisticated visual perception, effectively extending its application to visual mathematical reasoning. By invoking code execution for structured computation, CHARTOOL achieves better performance on out-of-domain visual reasoning benchmarks.

[0056] This method also includes ablation studies on several key components in CHARTOOL.

[0057] Regarding the impact of tool integration reinforcement learning, such as Figure 7 Qwen2.5-VL-3B was evaluated using different inference enhancement methods. Using only thought chain cues failed to improve performance and even reduced description accuracy, indicating that simple cue engineering cannot provide the model with sufficient graph understanding capabilities. Pure text reinforcement learning yielded only minor gains because language-level optimizations cannot address errors in fine-grained perception and numerical computation. In contrast, CHARTOOL's tool-integrated inference showed significant improvements across all metrics, particularly on inference-intensive benchmarks such as CharXiv and ChartQAPro. These results demonstrate that explicit tool invocation is crucial for robust graph inference.

[0058] To evaluate the effectiveness of the CHARTOOL-100k training data synthesized by our method, and in comparison with other synthetic training data, the same training process was used to train CHARTOOL on the ReachQA and ECD datasets. Figure 8As shown, the model trained on CHARTOOL-100k outperforms models trained on ReachQA or ECD in all benchmark tests. This highlights the superior quality and complexity of CHARTOOL-100k, enabling more robust and generalizable graph reasoning capabilities.

[0059] The impact of data composition. This method, in situations such as... Figure 9 The impact of different training data sources was analyzed. Training using only real-world arXiv chart data significantly improved descriptive performance on CharXiv, reflecting the benefits of engaging with diverse layouts and visual elements in scientific charts. In contrast, incorporating code-synthesized data was crucial for inference, as procedural generation problems involving trend analysis and precise numerical comparisons significantly boosted CharXiv's inference score to 41.40. Combining these two data sources enabled CHARTOOL to learn robust visual generalization and accurate computational inference together.

[0060] This method also analyzes the contributions of the two main tools used in CHARTOOL: image cropping and code computation. For example... Figure 10 As shown, both tools improve overall performance and demonstrate complementary advantages. The full CHARTOOL was compared to single-tool variants (clipping only and code-only computation) and the base model. Results show that using either tool alone provides an improvement over the baseline, while integrating both tools achieves the best average performance by jointly supporting fine-grained visual perception (clipping) and multi-step logic and numerical reasoning (code).

[0061] This method analyzes the distribution of tool calls to characterize the dynamic tool selection behavior of CHARTOOL. For example... Figure 11 As shown, CharXiv's layout is more complex, thus primarily triggering clipping tool calls. In contrast, benchmarks with simpler layouts but higher numerical complexity rely more heavily on calculation tool calls; on ChartQA, calculation tool calls account for 60.43% of all actions, while on MathVista this proportion increases to 77.17%. These results indicate that CHARTOOL dynamically adjusts tool usage based on the structural complexity of the task.

[0062] For problems that require numerical calculations, such as Figure 12 As shown (an example of mathematical visual reasoning), CHARTOOL invokes code-based calculations to ensure the answer is accurate and reliable. Figure 13 The demonstration showcased the combined use of two tools, where the model first applied a cropping tool to extract visual information, and then used code to calculate rigorous conclusions.

[0063] This structured approach highlights that CHARTOOL goes beyond simple pattern matching, employing a proactive, progressive reasoning workflow. The model exhibits unique and interpretable reasoning patterns that closely align with human cognitive processes.

[0064] In summary, this method designs a multi-agent data synthesis pipeline that, after rigorous validation and filtering, generates a diverse and challenging dataset, thus overcoming the limitations of existing synthetic data. By combining tool-based reasoning with image cropping and code computation, CHARTOOL supports both sophisticated visual perception and precise numerical reasoning. These capabilities are crucial for interpreting complex real-world graphs. Extensive experiments on graph benchmarks and out-of-domain visual mathematical reasoning tasks demonstrate sustained performance improvements and strong generalization capabilities.

[0065] like Figure 14 The diagram shown is a schematic diagram of a tool-integrated visual reasoning system for chart understanding provided in an embodiment of the present invention. The system can execute the tool-integrated visual reasoning method for chart understanding described in any of the above embodiments and is configured in a terminal.

[0066] This embodiment provides a tool-integrated visual reasoning system 10 for graph understanding, which includes: a hybrid graph data construction module 11, a question-answering agent 12, and a tool agent 13.

[0067] The hybrid chart data construction module 11 is used to construct a hybrid chart dataset including diverse parameter charts and real charts; the question-answering agent 12 is used to generate dimensionally complementary candidate question-answer pairs based on the hybrid chart dataset, and construct a quality-enhanced hybrid dataset through the candidate question-answer pairs, wherein the categories of the dimensionally complementary candidate question-answer pairs include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning; the tool agent 13 is used to perform integrated invocation reasoning based on the hybrid dataset for image cropping tools for enhancing local visual perception and code calculation tools for enhancing accuracy, generate a cold start trajectory intertwined with vision and computation, and use the correct trajectory in the cold start trajectory to supervise and fine-tune the cold start of the tool agent, so that the tool agent can integrate and invoke tools for visual reasoning of chart understanding.

[0068] Furthermore, the system also includes: The reinforcement learning module is used to optimize the tool agent's integrated tool invocation through group-relative policy optimization, so as to improve the tool agent's tool usage efficiency and multi-step reasoning ability.

[0069] Furthermore, the system also includes: An intelligent agent is used to perform quality verification on the candidate question-answer pairs to obtain a hybrid dataset of categories and quality enhancement, wherein the quality verification includes: consistency verification, inference verification verification, voting denoising verification, and difficulty score of recognition inference.

[0070] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the tool-integrated visual reasoning method for graph understanding in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: Construct a hybrid chart dataset that includes diverse parametric charts and real-world charts; Based on the hybrid graph dataset, candidate question-answer pairs with complementary dimensions are generated, and a quality-enhanced hybrid dataset is constructed using these candidate question-answer pairs. The categories of the candidate question-answer pairs with complementary dimensions include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning. The tool agent performs integrated invocation reasoning based on the hybrid dataset for image cropping tools to enhance local visual perception and code computation tools to improve accuracy, generating a cold start trajectory that interweaves vision and computation. The correct trajectory in the cold start trajectory is used to supervise and fine-tune the cold start of the tool agent, enabling the tool agent to integrate and invoke tools for visual reasoning of graph understanding.

[0071] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the tool-integrated visual reasoning method for graph understanding in any of the above method embodiments.

[0072] Figure 15 This is a schematic diagram of the hardware structure of an electronic device integrating a visual reasoning method for graph understanding, as provided in another embodiment of this application. Figure 15 As shown, the device includes: One or more processors 1510 and memory 1520, Figure 15 Taking a processor 1510 as an example, the device for integrating visual reasoning methods for graph understanding may also include an input device 1530 and an output device 1540.

[0073] The processor 1510, memory 1520, input device 1530, and output device 1540 can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0074] The memory 1520, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the tool-integrated visual reasoning method for graph understanding in the embodiments of this application. The processor 1510 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 1520, thereby implementing the tool-integrated visual reasoning method for graph understanding described in the above embodiments.

[0075] The memory 1520 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1520 may optionally include memory remotely located relative to the processor 1510, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0076] Input device 1530 can receive input numerical or character information. Output device 1540 may include display devices such as a display screen.

[0077] The one or more modules are stored in the memory 1520, and when executed by the one or more processors 1510, they execute the tool-integrated visual reasoning method for graph understanding in any of the above method embodiments.

[0078] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0079] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the tool-integrated visual reasoning method for graph understanding according to any embodiment of this invention.

[0081] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0082] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0083] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0084] (4) Other electronic devices with data processing functions.

[0085] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tool-integrated visual reasoning method for graph understanding, comprising: Construct a hybrid chart dataset that includes diverse parametric charts and real-world charts; Based on the hybrid graph dataset, candidate question-answer pairs with complementary dimensions are generated, and a quality-enhanced hybrid dataset is constructed using these candidate question-answer pairs. The categories of the candidate question-answer pairs with complementary dimensions include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning. The tool agent performs integrated invocation reasoning based on the hybrid dataset for image cropping tools to enhance local visual perception and code computation tools to improve accuracy, generating a cold start trajectory that interweaves vision and computation. The correct trajectory in the cold start trajectory is used to supervise and fine-tune the cold start of the tool agent, enabling the tool agent to integrate and invoke tools for visual reasoning of graph understanding.

2. The tool-integrated visual reasoning method according to claim 1, characterized in that, After using the correct trajectory in the cold start trajectory to supervise and fine-tune the cold start of the tool agent, the method further includes: The tool agent is integrated with reinforcement learning for tool invocation using group-relative strategy optimization, which improves the tool agent's tool usage efficiency and multi-step reasoning ability.

3. The tool-integrated visual reasoning method according to claim 2, characterized in that, The reward function of the group relative strategy optimization ,in, To score the accuracy of the reasoning answer, Score based on structure and format. The score is based on the use of the tool. The indicator function is set to 1 when the reasoning answer is correct, and 0 otherwise. and Preset weights.

4. The tool-integrated visual reasoning method according to claim 1, characterized in that, The constructed hybrid chart dataset, comprising diverse parametric charts and visually realistic charts, includes: Using a large artificial intelligence model, drawing code is generated based on a variety of set parameters, and a composite chart with diverse parameters is rendered. The diverse parameters include: chart type, layout, and number of sub-charts. Data mining is performed on the literature to incorporate the visual complexity, layout variations, and domain diversity of the charts, resulting in realistic charts with corresponding contextual text. Visual quality and semantic integrity are filtered for the diverse parameter charts and the real charts to obtain a hybrid chart dataset suitable for chart reasoning.

5. The tool-integrated visual reasoning method according to claim 4, characterized in that, The visual quality screening includes: screening for overlapping, misalignment, and readability of text in charts; The semantically complete filtering includes checking whether the self-explanatory elements of the chart are complete, wherein the chart includes axes, legend and labels.

6. The tool-integrated visual reasoning method according to claim 1, characterized in that, The construction of a quality-enhanced hybrid dataset through the candidate question-answering includes: The candidate question-answer pairs are subjected to quality verification to obtain a hybrid dataset of categories and quality enhancement. The quality verification includes consistency verification, inference verification, voting denoising verification, and difficulty score of recognition inference.

7. A tool-integrated visual reasoning system for graph understanding, comprising: The Hybrid Chart Data Building Module is used to construct hybrid chart datasets that include diverse parametric charts and real-world charts. A question-answering agent is used to generate candidate question-answer pairs with complementary dimensions based on the hybrid graph dataset, and to construct a quality-enhanced hybrid dataset through the candidate question-answer pairs. The categories of the candidate question-answer pairs with complementary dimensions include: recognition-type question-answer pairs for element recognition and structure extraction, and reasoning-type question-answer pairs for visual analysis and numerical reasoning. The tool agent is used to perform integrated invocation reasoning based on the hybrid dataset for image cropping tools to enhance local visual perception and code computation tools to improve accuracy, generating a cold start trajectory that interweaves vision and computation. The correct trajectory in the cold start trajectory is used to supervise and fine-tune the cold start of the tool agent, enabling the tool agent to integrate and invoke tools for visual reasoning of graph understanding.

8. The tool-integrated visual reasoning system according to claim 7, characterized in that, The system also includes: The reinforcement learning module is used to optimize the tool agent's integrated tool invocation through group-relative policy optimization, so as to improve the tool agent's tool usage efficiency and multi-step reasoning ability.

9. The tool-integrated visual reasoning system according to claim 7, characterized in that, The system also includes: An intelligent agent is used to perform quality verification on the candidate question-answer pairs to obtain a hybrid dataset of categories and quality enhancement, wherein the quality verification includes: consistency verification, inference verification verification, voting denoising verification, and difficulty score of recognition inference.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-6.

11. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

12. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

Images