Report chart generation method and device based on multi-modal large model

Abstract

The invention relates to a report chart generation method and device based on a multi-modal large model, and the method comprises the steps: obtaining chart drawing data provided by a user through a natural language, calling a first large language model to analyze the chart drawing data, and generating a structured data set; calling the first large language model to generate a chart of an initial version in combination with the structured data set and user requirements; calling a multi-modal generation large model to evaluate and correct the image of the initial version chart to obtain an automatically corrected chart; the user inputs a modification instruction to modify the automatically modified chart to obtain a modified chart; and copying and embedding the modified chart into a specified position of the target report document. Through an original'generation-verification-correction 'closed loop, common problems such as chart layout and label overlapping are automatically detected and corrected by using the multi-modal visual model, the professional quality and readability of the chart are remarkably improved, and manpower is liberated from tedious adjustment.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a method and apparatus for generating report charts based on multimodal large models. Background Technology

[0002] In many fields such as scientific research, business analysis, and education, data reports are a key carrier for information transmission and decision support, and statistical charts (such as bar charts, line charts, pie charts, etc.) are the core of data visualization, which can intuitively and clearly show the relationships, trends and distributions between data.

[0003] The main methods for generating statistical charts and graphs are as follows:

[0004] Manual software generation: Users use office software such as Microsoft Excel and WPS to manually select data, chart types, and adjust styles through a graphical user interface (GUI). While this method is flexible, it is cumbersome, time-consuming, and requires a high level of operator proficiency for creating complex or large numbers of charts.

[0005] Programming code generation: Professional analysts or developers use programming languages ​​such as Python (e.g., Matplotlib, Seaborn, Plotly libraries) and R (e.g., ggplot2 library) to generate charts by writing code. This method allows for high customization and automation, but it has a high technical threshold and is not suitable for ordinary users. Furthermore, once a chart is generated, fine-tuning (e.g., "moving this label a little to the right") often requires modifying complex code parameters, creating a "semantic gap" and resulting in inefficient adjustments.

[0006] Business Intelligence (BI) Tools: These tools, such as Tableau and Power BI, allow users to create interactive dashboards through drag-and-drop operations. While powerful, these tools are often cumbersome and primarily used for exploratory analysis. They export individual charts and seamlessly embed them into report documents, but format compatibility and subsequent editability are often limited.

[0007] It is evident that the above technology has the following defects and shortcomings:

[0008] 1. Accuracy and reliability issues: Charts (image format) directly generated by multimodal large models are essentially "drawn" rather than accurately rendered based on data. This can easily lead to serious errors such as data misinterpretation, inaccurate scales, and mismatch between numerical values ​​and graphs, failing to meet the rigor requirements of statistical charts.

[0009] 2. The "Semantic Round-tripping" problem in modification and iteration: For charts generated by code, simple visual adjustment requests from users (such as "make the font bigger" or "separate these two labels a bit") are difficult to directly translate into precise code parameter modifications. Users need to understand the code logic and make multiple attempts, resulting in a time-consuming and laborious modification process, i.e., "semantic round-tripping".

[0010] 3. Automation layout and aesthetic issues: Automatically generated charts often have problems such as overlapping labels, overlapping legends, unreasonable element layout, and unprofessional color schemes, requiring a lot of manual post-processing to improve them, which affects the professionalism and readability of the report.

[0011] 4. Barriers to format and subsequent editing: The charts generated by the model are mostly static images (such as PNG, JPG). Once inserted into a report (such as a Word document), it is no longer possible to edit the source data behind them or adjust the chart type, thus losing the linkage between data and graphics.

[0012] Therefore, there is an urgent need for a new chart generation method that can combine the understanding of large models, the precise drawing capabilities of traditional office software, and native format compatibility, and can be efficiently and accurately adjusted through automated verification and natural language interaction. Summary of the Invention

[0013] To address the aforementioned problems, the purpose of this invention is to provide a method and apparatus for generating report charts based on a multimodal large model.

[0014] A method for generating report charts based on a multimodal large model includes:

[0015] Step 1: Obtain the chart drawing data provided by the user using natural language, call the first language model to parse the chart drawing data and generate a structured dataset;

[0016] Step 2: Use the first major language model to generate an initial version of the chart by combining the structured dataset and user requirements;

[0017] Step 3: Use the multimodal large model to evaluate and correct the image of the initial version chart to obtain the automatically corrected chart;

[0018] Step 4: The user inputs modification commands to modify the automatically corrected chart to obtain the modified chart;

[0019] Step 5: Copy the modified chart and embed it in the specified location of the target report document.

[0020] Preferably, in step 1, the first major language model is called to parse the chart drawing data to generate data processing code, and the data processing code is executed to generate a structured dataset.

[0021] Preferably, in step 2, the first large language model is invoked to generate macro instructions that are executed in the target application by combining the structured dataset and user requirements, and the macro instructions are executed to generate an initial version of the chart in the target application.

[0022] Preferably, step 3: calling a multimodal large-scale model to evaluate and correct the image of the initial version chart to obtain an automatically corrected chart includes:

[0023] Step 3.1: Capture an image of the initial version of the chart;

[0024] Step 3.2: Call the multimodal generation model to analyze the image according to the preset rule set to obtain the evaluation results, and generate a structured text feedback report based on the evaluation results;

[0025] Step 3.3: Input the structured text feedback report and the current macro instructions into the first language model to generate the revised new macro;

[0026] Step 3.4: Execute the revised macro to generate a new chart, then return to Step 3.2 until the evaluation result of the new chart meets the preset conditions.

[0027] Preferably, in step 3.2, a loss function is used to quantify the degree of non-compliance of the chart:

[0028]

[0029] in, It is a rule The weight of each rule This corresponds to a check item for chart quality. It is an evaluation function, when the graph image Violation of rules When it is 1, its value is 1; otherwise it is 0. If the value exceeds the preset threshold ϵ, the chart is considered unqualified.

[0030] Preferably, in step 3.2, a structured text feedback report is generated based on the evaluation results. Describe all rule violations and their specific manifestations:

[0031]

[0032] in, It is a natural language description of the violation.

[0033] The present invention also provides a report chart generation device based on a multimodal large model, comprising:

[0034] The user interaction module is used to obtain chart drawing data provided by users using natural language, call the first language model to parse the chart drawing data and generate a structured dataset;

[0035] The data generation module is used to call the first major language model to combine structured datasets and user requirements to generate an initial version of the chart;

[0036] The chart generation module is used to call a multimodal large model to evaluate and correct the image of the initial version chart, resulting in an automatically corrected chart.

[0037] The visual verification module is used to modify the automatically corrected chart by the user's input of modification commands to obtain the modified chart;

[0038] The document output module is used to copy and embed the modified charts into a specified location in the target report document.

[0039] The present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, characterized in that the computer program, when executed by the processor, implements the steps in the above-described method for generating report charts based on a multimodal large model.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps in the above-described method for generating report charts based on a multimodal large model.

[0041] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0042] This invention relates to a report chart generation method based on a multimodal large model. Compared with the prior art, this invention uses an original "generation-verification-correction" closed loop to automatically detect and correct common problems such as chart layout and label overlap using a multimodal visual model, which significantly improves the professional quality and readability of the charts and frees manual labor from tedious adjustments.

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a diagram illustrating the initial version of the chart generation process provided by the present invention.

[0046] Figure 2 A diagram illustrating the optimization process of the initial version of the diagram provided by this invention;

[0047] Figure 3 This is a schematic diagram of the report chart generation device based on a multimodal large model provided by the present invention. Detailed Implementation

[0048] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0049] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0050] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0051] Please see Figure 1-2 A method for generating report charts based on a multimodal large model, comprising:

[0052] Step 1: Obtain the chart drawing data provided by the user using natural language, call the first language model to parse the chart drawing data and generate a structured dataset;

[0053] In step 1, the user provides raw data or a data description in natural language. The system calls the primary language model (text generation model). It parses and generates a piece of data processing code. (e.g., Python scripts) are used to generate or organize a structured dataset D.

[0054] This process can be formally expressed as:

[0055]

[0056]

[0057] Here, Exec() represents the operation of code C_data. The dataset D is saved in a common format (such as CSV or JSON) as the data basis for subsequent chart plotting.

[0058] Step 2: Use the first major language model to generate an initial version of the chart by combining the structured dataset and user requirements;

[0059] In step 2, the user specifies the chart type and initial requirement P_chart. The system calls LLM_T, which, combined with the dataset D and the user requirement P_chart, generates a macro instruction M_0 (such as a VBA script) that can be executed in the target application (such as Microsoft Excel).

[0060]

[0061] Execute macro M_0 to generate an initial version of graph G_0 in the target application.

[0062] Step 3: Use the multimodal large model to evaluate and correct the image of the initial version chart to obtain the automatically corrected chart;

[0063] Step 3 includes:

[0064] 1. Chart Visual Capture: The system captures the currently generated chart. The image of (k is the iteration number, initially k=0) .

[0065] 2. Rule-based visual verification: The system has a built-in configurable set of verification rules. Each of the rules This corresponds to a chart quality check item (e.g., whether data labels overlap, whether axis ticks are uniform, whether the legend is clear, whether there is element occlusion, etc.). It calls a multimodal generator to create a large model. (e.g., the Qwen2-vl model) for images Perform analysis and evaluate based on rule set R.

[0066] 3. Quantitative Evaluation and Feedback Generation: Define a loss function. To quantify charts The degree of non-compliance:

[0067]

[0068] in, It is a rule The weight, It is an evaluation function, when the graph image Violation of rules When it is true, its value is 1; otherwise, it is 0. If the value exceeds the preset threshold ϵ, the chart is considered unqualified. Based on the evaluation results, the system generates a structured text feedback report. Describe all rule violations and their specific manifestations.

[0069]

[0070] in It is a natural language description of the violation, such as: "The data labels for 'Q1' and 'Q2' overlap in the chart".

[0071] 4. Iterative Correction: Feedback Report As a new instruction, along with the macros from the previous round... The input is given to LLM_T, which generates a new macro M_{k+1} that has been corrected.

[0072]

[0073] implement Generate new chart And repeat steps 1-4 above until the loss function is calculated. Less than or equal to the threshold ϵ, or the maximum number of iterations is reached.

[0074] Step 4: The user inputs modification commands to modify the automatically corrected chart to obtain the modified chart;

[0075] In step 4, after the automatic correction cycle ends, the system presents the user with the current optimal chart. Users can input further modification instructions through a natural language interactive window. For example, "Change the title font to blue" or "Set the maximum value of the Y-axis to 200".

[0076] The system called again Based on the final macro and user instructions Generate fine-tuned macros Then update the chart. This process can be repeated multiple times until the user is satisfied.

[0077] Step 5: Copy the modified chart and embed it in the specified location of the target report document.

[0078] In step 5, when the user confirms the final chart The system then copies and embeds the chart object in its native format (such as ExcelChartObject) into a specified location in the target report document (such as Word). Because it is in native format, users can still double-click the chart in Word to activate the Excel editing environment and directly modify the source data or make style adjustments.

[0079] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0080] Compared with the prior art, the advantages of this application are as follows:

[0081] 1. High accuracy and reliability: Charts are generated by executing macro commands in professional software environments such as Excel, rather than by directly generating images, which fundamentally ensures the accuracy of chart data and the reliability of rendering.

[0082] 2. Automated quality assurance: The unique "generation-verification-correction" closed loop uses a multimodal visual model to automatically detect and correct common problems such as layout and label overlap, significantly improving the professional quality and readability of charts and freeing manual labor from tedious adjustments.

[0083] 3. Efficient and intuitive editing experience: Completely solves the "semantic detour" problem. Users can make complex chart adjustments using simple natural language, making the operation intuitive, the editing efficiency high, and achieving "what you say is what you get".

[0084] 4. Seamless workflow integration: The final output charts are editable native objects that are perfectly embedded in the report document, preserving the synergy between data and graphics. This greatly facilitates the later maintenance and data updates of the report, meeting the needs of actual workflows.

[0085] 5. Combining flexibility and ease of use: It combines the flexibility of code generation, automation, and the intuitiveness of GUI operation, allowing ordinary users to easily create professional, high-quality statistical charts.

[0086] Please see Figure 3 The present invention also provides a report chart generation device based on a multimodal large model, comprising:

[0087] User interaction module 101 is used to obtain chart drawing data provided by the user using natural language, call the first language model to parse the chart drawing data and generate a structured dataset;

[0088] Data generation module 102 is used to call the first major language model to generate an initial version of the chart by combining the structured dataset and user requirements;

[0089] The chart generation module 103 is used to call a multimodal large model to evaluate and correct the image of the initial version chart to obtain an automatically corrected chart.

[0090] The visual verification module 104 is used to modify the automatically corrected chart by the user inputting modification instructions to obtain the modified chart;

[0091] The document output module 105 is used to copy and embed the modified charts into a specified location in the target report document.

[0092] Compared with the prior art, the beneficial effects of the report chart generation device based on a multimodal large model provided by the present invention are the same as the beneficial effects of the report chart generation method based on a multimodal large model described in the above technical solution, and will not be repeated here.

[0093] The present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. When the computer program is executed by the processor, it implements the steps in the above-described method for generating report charts based on a multimodal large model. Compared with the prior art, the beneficial effects of the electronic device provided by the present invention are the same as those of the above-described method for generating report charts based on a multimodal large model, and will not be elaborated here.

[0094] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps in the above-described method for generating report charts based on a multimodal large model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the above-described method for generating report charts based on a multimodal large model, and will not be elaborated here.

[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating report charts based on a multimodal large model, characterized in that, include: Step 1: Obtain the chart drawing data provided by the user using natural language, call the first language model to parse the chart drawing data and generate a structured dataset; Step 2: Use the first major language model to generate an initial version of the chart by combining the structured dataset and user requirements; Step 3: Use the multimodal large model to evaluate and correct the image of the initial version chart to obtain the automatically corrected chart; Step 4: The user inputs modification commands to modify the automatically corrected chart to obtain the modified chart; Step 5: Copy the modified chart and embed it in the specified location of the target report document.

2. The report chart generation method based on a multimodal large model according to claim 1, characterized in that, In step 1, the first major language model is called to parse the chart drawing data to generate data processing code, and the data processing code is executed to generate a structured dataset.

3. The report chart generation method based on a multimodal large model according to claim 1, characterized in that, In step 2, the first major language model is invoked to generate macro instructions that are executed in the target application, based on the structured dataset and user requirements. Executing the macro instructions generates an initial version of the chart in the target application.

4. The report chart generation method based on a multimodal large model according to claim 1, characterized in that, Step 3: The multimodal large-scale model is used to evaluate and correct the image of the initial version of the chart, resulting in an automatically corrected chart, including: Step 3.1: Capture an image of the initial version of the chart; Step 3.2: Call the multimodal generation model to analyze the image according to the preset rule set to obtain the evaluation results, and generate a structured text feedback report based on the evaluation results; Step 3.3: Input the structured text feedback report and the current macro instructions into the first language model to generate the revised new macro; Step 3.4: Execute the revised macro to generate a new chart, then return to Step 3.2 until the evaluation result of the new chart meets the preset conditions.

5. The report chart generation method based on a multimodal large model according to claim 4, characterized in that, In step 3.2, a loss function is used to quantify the degree of non-compliance of the chart: in, It is a rule The weight of each rule This corresponds to a check item for chart quality. It is an evaluation function, when the graph image Violation of rules When it is 1, its value is 1; otherwise it is 0. If the value exceeds the preset threshold ϵ, the chart is considered unqualified.

6. The report chart generation method based on a multimodal large model according to claim 5, characterized in that, In step 3.2, a structured written feedback report is generated based on the evaluation results. Describe all rule violations and their specific manifestations: in, It is a natural language description of the violation.

7. A report chart generation device based on a multimodal large model, characterized in that, include: The user interaction module is used to obtain chart drawing data provided by users using natural language, call the first language model to parse the chart drawing data and generate a structured dataset; The data generation module is used to call the first major language model to combine structured datasets and user requirements to generate an initial version of the chart; The chart generation module is used to call a multimodal large model to evaluate and correct the image of the initial version chart, resulting in an automatically corrected chart. The visual verification module is used to modify the automatically corrected chart by the user's input of modification commands to obtain the modified chart; The document output module is used to copy and embed the modified charts into a specified location in the target report document.

8. An electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, characterized in that, When the computer program is executed by the processor, it implements the steps in the report chart generation method based on a multimodal large model as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the report chart generation method based on a multimodal large model as described in any one of claims 1-6.

Images