Chart editing framework based on visual prompt

Abstract

The invention discloses a chart editing framework based on visual cues. The chart editing framework comprises a chart editing mechanism guided by the visual cues and a large-scale high-quality chart editing training data set construction method. In a chart editing framework provided by the invention, a user can generate a corresponding visual prompt (such as a bounding box) in an image through an automatic script based on an editing requirement only through more convenient interaction operation (such as simple clicking and target area selection), and a direct and accurate editing instruction is provided for a model. According to the method, the editing accuracy is remarkably improved, and the method is more suitable for the characteristic of a visual form with a strict semantic structure of a chart. According to the method, on one hand, the burden of a user can be remarkably reduced, and on the other hand, clearer and more structured guidance is provided for the model.

Description

Technical Field

[0001] The present invention relates to the technical field of automated chart editing, and in particular to a chart editing framework based on visual cues. Background Art

[0002] Charts are a form of data visualization widely used in scientific research and industry, often used for data analysis and results presentation. Automated chart editing technology aims to intelligently modify existing charts based on user instructions. It integrates multidisciplinary technologies such as computer vision, natural language processing, and data visualization. This capability enables AI models to understand user intent and precisely adjust the chart's structure, style, or data content.

[0003] Automatic chart editing based on user intent holds significant practical value in data analysis and visualization tools, and can be widely applied in a variety of scenarios, including business analysis, scientific research, and education and training. By improving the efficiency and accuracy of chart adjustments, this technology helps enhance the clarity and impact of data presentation, thereby more effectively supporting data-driven decision-making.

[0004] Image editing technology has evolved from pixel-level tasks like image restoration and style transfer to semantic-level editing tasks based on text descriptions or drag-and-drop operations. Existing research on semantic-level image editing primarily relies on generative models such as diffusion models and generative adversarial networks (GANs). These tasks essentially encode image pixels into vectors, then use a large model to directly operate on these encodings in the latent space to achieve image editing and generation.

[0005] Existing research on charts primarily focuses on chart understanding and chart generation. On the one hand, tasks like chart question answering (ChartQA) are being used to improve models' ability to extract and understand information from charts. On the other hand, research is focused on enabling models to generate corresponding chart code based on natural language descriptions, thus achieving automatic chart generation. In contrast, research on chart editing is relatively limited, with only a limited amount of work focusing on how to use natural language instructions to guide models to edit existing charts.

[0006] Although current mainstream image editing methods have made significant progress in visual content generation, their essence is still based on pixel-level operations. This processing method has obvious shortcomings when dealing with visual objects such as charts. As a highly structured form of data visualization, charts have strict semantic and logical constraints. For example, there is a strict alignment relationship between each column and the coordinate axis and numerical labels in a bar chart. Therefore, any modification to the chart requires a high degree of precision at the visual level to ensure its semantic integrity. However, existing image editing technologies based on pixel operations often cannot meet this precision requirement and are prone to structural damage, which in turn undermines the information expression ability carried by the chart.

[0007] Current research in charting primarily focuses on understanding chart content, lacking the ability to model and reason about the underlying structure (such as rendering code) in a fine-grained manner. Empirical results show that when editing charts based on user intent, existing methods often suffer from "editing bias," where the resulting charts are inconsistent with the user's intended goals. Furthermore, while some work has attempted to implement chart editing, it still suffers from three significant drawbacks: 1) Dependence on external tools: Some methods require additional plugins or specialized interfaces to complete the editing process, reducing the system's versatility and integration efficiency; 2) Limited task coverage: The number of editable chart types is limited, and the user's available modification operations are relatively limited, making it difficult to meet the diverse needs of real-world application scenarios; 3) Inefficient natural language interaction: When modifying specific elements in a chart (such as a curve in a line chart), users must precisely describe the target object using complex natural language. This not only increases interaction costs but can also lead to model misjudgments due to language ambiguity, compromising editing quality and user experience.

[0008] In summary, achieving high-quality automatic editing of charts requires breaking through the technical bottlenecks of existing image editing and chart understanding methods, and conducting systematic optimization from multiple levels such as semantic modeling, structure preservation, and human-computer interaction.

[0009] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0010] The purpose of this invention is to provide a visually-cued chart editing framework for more effectively guiding large models in chart editing operations. Compared to traditional natural language-based approaches, this approach offers significant advantages in terms of interaction efficiency and modification accuracy: it significantly reduces user burden while providing clearer, more structured guidance for models.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions: The present invention provides a visual cue-based chart editing framework, including a visual cue-guided chart editing mechanism and a method for constructing a large-scale, high-quality chart editing training dataset. Specifically, the visual cue-guided chart editing mechanism comprises the following steps: given a chart to be edited, a user provides visual cues (e.g., by clicking) based on their editing intentions; upon receiving the user's click, an internal automated Python script annotates the chart and provides this as a visual cue to a model; the model executes the chart editing task based on the visual cue and outputs the modified results. The method for constructing a large-scale high-quality chart editing training dataset is as follows: first, a certain amount of original chart data is collected from the Internet to construct a chart pool, and then the editing pool is further constructed by combining the Artist package in Python's matplotlib and the modification method artificially constructed for specific chart types; at the beginning of the pipeline, a chart is randomly selected from the chart pool, and according to the corresponding type of the chart, the corresponding modification instruction and the Python function required to execute the modification instruction are selected from the specific editing pool; then the chart drawing code, Python function, and modification instruction are input into the large model together, so that the large model can be used in the detailed The modification operation is performed under detailed and intuitive guidance, and the big model is allowed to select the modification object by itself. The big model will output the modified code and the modified object; then the modified code is run and the big model is allowed to test it to check whether the modified chart has any visual discomfort and filter out the data that does not meet the conditions; after that, the modification instructions are rewritten, keeping the original meaning unchanged but replacing it with different expressions to achieve diversified expressions of the modification instructions; then the chart is visually annotated using an automated script, and the final data includes: the original chart, the code for drawing the original chart, the rewritten modification instructions, the modified code, the specific description of the modification object, and the original chart with visual annotations.

[0012] By adopting the above technical solution, the present invention has the following beneficial effects: 1. Efficient and convenient human-computer interaction: This invention provides users with an efficient chart editing interaction mechanism. It only requires a combination of simple visual prompts (such as simple click operations) and natural language descriptions to complete the targeted modification of complex charts, significantly reducing user operation costs and being closer to the needs of actual data analysis and visualization application scenarios.

[0013] 2. Guidance mechanism significantly improves editing accuracy: By introducing a clear visual annotation mechanism (such as bounding boxes), it effectively alleviates the ambiguity in natural language expressions and significantly reduces the risk of user instructions misleading the model, making chart editing results more accurate and in line with user expectations.

[0014] 3. Driving Breakthroughs in Open-Source Multimodal Model Performance: This application constructed a high-quality chart editing dataset of 20,000 records and used this dataset to fine-tune an open-source multimodal model, significantly improving its performance on chart editing tasks. Experiments demonstrated that the fine-tuned open-source model surpassed closed-source large-scale models such as GPT-4 on multiple evaluation metrics, effectively promoting the development and practical application of the open-source large-scale multimodal model community. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a complete technical process for editing charts provided by an embodiment of the present invention; Figure 2 A schematic diagram of a method for constructing a large-scale, high-quality chart editing training dataset provided by an embodiment of the present invention; Figure 3 A schematic diagram comparing the traditional natural language-based chart editing provided in an embodiment of the present invention and the visual prompt-based chart editing proposed in this application. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0019] To facilitate understanding of this application, the following technical terms involved in this application are explained: Charts: Charts are a type of visual representation displayed on screen using a graphical structure. They are widely used to present statistical attribute information such as temporal and quantitative information. As a key vehicle for knowledge mining and information communication, charts are intuitive, vivid, and easy to understand, making them a key means of visualizing object attribute data.

[0020] Multimodal Large Models: Multimodal Large Models are deep learning models that can simultaneously process and integrate multiple data modalities (such as text, images, audio, and video). Through cross-modal understanding and interaction capabilities, MLLMs can perform complex intelligent tasks such as visual question answering, image-text generation, and speech synthesis, becoming a key research direction in the field of artificial intelligence.

[0021] Large-scale model training: Large-scale model training is the process of iteratively optimizing deep neural networks with parameters in the billions or even tens of billions, using high-performance computing resources on large-scale labeled or unlabeled datasets. This process enables the model to learn abstract patterns and regularities in the data, which can then be applied to various intelligent tasks such as language understanding and image recognition.

[0022] End-to-end learning: End-to-end learning is a holistic modeling paradigm that emphasizes learning directly from raw input to final output, eliminating the handcrafted intermediate processing modules found in traditional systems. This approach integrates multiple subtasks within a unified model structure, improving the system's integrated expressiveness and task generalization capabilities. It has been widely used in fields such as speech recognition and image generation.

[0023] Supervised fine-tuning: Supervised fine-tuning is the process of further training a pre-trained model using manually annotated task-specific data. This strategy enables the model to better adapt to specific application scenarios and improves performance in downstream tasks. Supervised fine-tuning is typically performed after the pre-training phase and is a key method for achieving customization for large models.

[0024] Matplotlib: is a powerful plotting library for Python that is designed to help users create static, dynamic, and interactive graphics. It is widely used for data visualization in data science, engineering, finance, and other fields. Matplotlib supports a variety of chart types, including line charts, bar charts, scatter plots, pie charts, histograms, heat maps, and more. Through its pyplot module, users can draw high-quality graphics in a simple way. Matplotlib's artist package is one of its core concepts, mainly used to manage and manipulate graphic elements. In Matplotlib, "artist" refers to any visual object in a graphic, and all visual elements (such as lines, text, titles, axes, etc.) can be regarded as "artists." The artist package provides many classes and methods to help users create, modify, and control these graphic elements.

[0025] The current chart editing technology still has significant deficiencies in feasibility, applicability and interaction efficiency, which seriously restricts its application value in actual scenarios. To this end, this application proposes a chart editing framework based on visual cues to more effectively guide large models to perform chart editing operations. Compared with the traditional method that relies on natural language, this method has significant advantages in interaction efficiency and modification accuracy: on the one hand, it can significantly reduce the burden on users, and on the other hand, it provides clearer and more structured guidance for the model.

[0026] Specifically, the visual cue-based chart editing framework proposed in this application mainly includes two major innovative technologies: a visual cue-guided chart editing mechanism and a method for constructing a large-scale high-quality chart editing training dataset: Visual cue-guided chart editing mechanism: e.g. Figure 1 As shown in the figure, given a chart that needs editing, the user can provide visual cues based on their editing intent: for example, they can click to select the chart object to be modified and enter the desired modification instructions. After the system receives the user's click, an internal automated Python script annotates the chart (for example, generating a bounding box) and provides this visual cue to the model. Based on this information, the model performs the chart editing task and outputs the modification results.

[0027] Method for constructing large-scale high-quality graph editing training dataset: Figure 2As shown, first, about 10k of original chart data are collected from the Internet to build a chart pool, and then the editing pool is further constructed by combining the Artist package in Python's matplotlib and artificially constructing some modification methods for specific chart types. At the beginning of the pipeline, a chart is randomly selected from the chart pool. According to the corresponding type of the chart, this application selects the corresponding modification instruction and the python function needed to execute the modification instruction from the specific editing pool. This application inputs the code for drawing the chart, the python function, and the modification instruction into the large model together, allowing the large model to perform the modification operation under detailed and intuitive guidance, and allowing the large model to select the modification object by itself. The large model will output the modified code and the modified object, and then this application runs the modified code and lets the large model perform a test, mainly to check whether the modified chart has any visual discomfort (such as overlapping elements, exceeding the canvas, etc., which causes visual discomfort), and filter out the data that does not meet the conditions. This application then rewrites the modification instructions, keeping the original meaning unchanged but changing the expression to achieve diversified expression of the modification instructions; then uses an automated script to visually annotate the chart, and the final data obtained includes: the original chart, the code for drawing the original chart, the rewritten modification instructions, the modified code, the specific description of the modification object, and the original chart with visual annotations.

[0028] like Figure 3 As shown in the figure, when users use natural language to describe the objects that need to be modified in the chart, large models are easily affected by vague or ambiguous expressions, causing the generated results to deviate from the user's expectations. In the framework proposed in this application, users only need to use more convenient interactive operations (for example, by simply clicking to select the target area), and this application can generate corresponding visual cues (such as bounding boxes) in the image through automated scripts, providing direct and precise editing instructions for the model. This method not only significantly improves the accuracy of editing, but also better fits the characteristics of charts, a visual form with a rigorous semantic structure.

[0029] In addition, this application constructs a high-quality dataset covering a variety of chart types and modification operations for training and evaluating visual annotation-based chart editing models. This dataset not only improves the feasibility and generalization ability of the model on this task, but also provides a solid data foundation for related research.

[0030] Compared with natural language, the method of this application is more intuitive and precise, effectively reducing the model's understanding deviation, thereby significantly improving the accuracy and robustness of chart editing.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A chart editing framework based on visual cues, characterized in that: This paper includes a visual cue-guided chart editing mechanism and a method for constructing a large-scale, high-quality chart editing training dataset. Specifically, the visual cue-guided chart editing mechanism involves the following steps: given a chart to be edited, the user provides visual cues based on their editing intentions. After the system receives a click from the user, an internal automated Python script annotates the chart and provides this as a visual cue to the model. The model then executes the chart editing task based on the editing requirements and visual cues, and outputs the modified results. The method for constructing a large-scale high-quality chart editing training dataset is as follows: first, a certain amount of original chart data is collected from the Internet to construct a chart pool, and then the editing pool is further constructed by combining the Artist package in Python's matplotlib and the modification method artificially constructed for specific chart types; at the beginning of the pipeline, a chart is randomly selected from the chart pool, and according to the corresponding type of the chart, the corresponding modification instruction and the Python function required to execute the modification instruction are selected from the specific editing pool; then the chart drawing code, Python function, and modification instruction are input into the large model together, so that the large model can be used in the detailed The modification operation is performed under detailed and intuitive guidance, and the big model is allowed to select the modification object by itself. The big model will output the modified code and the modified object; then the modified code is run and the big model is allowed to test it to check whether the modified chart has any visual discomfort and filter out the data that does not meet the conditions; after that, the modification instructions are rewritten, keeping the original meaning unchanged but replacing it with different expressions to achieve diversified expressions of the modification instructions; then the chart is visually annotated using an automated script, and the final data includes: the original chart, the code for drawing the original chart, the rewritten modification instructions, the modified code, the specific description of the modification object, and the original chart with visual annotations.

Images