Chart generation method, multimodal data platform, computing device and computer-readable storage medium

Abstract

Embodiments of this specification provide a chart generation method, a multimodal data platform, a computing device, and a computer-readable storage medium. The chart generation method includes: obtaining a task description text and original data of a chart generation task; parsing the chart generation task based on the task description text and the original data to obtain a generation scheme for the chart generation task; retrieving a reference example chart from an example chart database based on the generation scheme; and using the multimodal generation model to perform the chart generation task on the original data based on the reference example chart and the generation scheme to generate a target chart. This ensures that even for uncommon chart types, a target chart with high accuracy and high applicability can be generated, effectively improving the diversity and quality of chart generation.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of artificial intelligence, and particularly to a method for generating charts, a multi-modal data platform, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence technology, through chart generation models, such as large language models or multi-modal models dedicated to chart drawing, automatically generating charts can effectively improve the efficiency of chart drawing.

[0003] Currently, through a chart generation model, under the guidance of the task description text of the input chart generation task, the input original data is used for chart generation, the chart data is output, and the front-end rendering of the chart data is performed to obtain the target chart.

[0004] However, due to the limitations of model training data and capabilities, the types of charts supported are few, and it completely depends on the knowledge that the chart generation model has learned. In the case where the training set of the chart generation model does not cover all possible chart styles and formats, for some uncommon chart types, it is very easy to make mistakes during generation, resulting in insufficient accuracy and applicability of the generation results. Therefore, there is an urgent need for a more flexible and extensible chart generation method. Summary of the Invention

[0005] In view of this, the embodiments of this specification provide a method for generating charts. One or more embodiments of this specification also relate to a multi-modal data processing platform, a chart generation device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0006] One aspect of the embodiments of this specification provides a method for generating charts, including: obtaining the task description text and the original data of the chart generation task; parsing the chart generation task based on the task description text and the original data to obtain a generation scheme for the chart generation task; retrieving a reference example chart from the example chart database based on the generation scheme; and performing the chart generation task on the original data based on the reference example chart and the generation scheme through a multi-modal generation model to generate the target chart.

[0007] In one embodiment of this specification, by combining the parsing of task description text and raw data, a specific generation scheme is generated, and reference sample charts are retrieved from the sample chart database to supplement the knowledge limitations of the multi-modal generation model, getting rid of the dependence on the knowledge of pre-trained models. Additionally, through diverse chart instances that actually exist, the styles and formats of chart generation are enriched, solving the problems of the accuracy and applicability of generation results caused by insufficient coverage of the model training set. Guided by the generation scheme, the multi-modal generation model generates charts according to the reference samples, ensuring that even for uncommon chart types, target charts with high accuracy and high applicability can be generated, effectively improving the diversity and quality of chart generation. Description of the Drawings

[0008] Figure 1 is a flowchart of a chart generation method provided by an embodiment of this specification;

[0009] Figure 2 is a schematic diagram of a target chart in a chart generation method provided by an embodiment of this specification;

[0010] Figure 3 is a schematic flow diagram of a chart generation method provided by an embodiment of this specification;

[0011] Figure 4 is a processing procedure flowchart of a chart generation method applied to a product operation scenario provided by an embodiment of this specification;

[0012] Figure 5 is a schematic structural diagram of a multi-modal data processing platform provided by an embodiment of this specification;

[0013] Figure 6 is a schematic structural diagram of a chart generation device provided by an embodiment of this specification;

[0014] Figure 7 is a structural block diagram of a computing device provided by an embodiment of this specification. Detailed Embodiments

[0015] Many specific details are set forth in the following description in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of this specification. Therefore, this specification is not limited by the specific embodiments disclosed below.

[0016] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the", and "said" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.

[0017] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, usually including hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than one quadrillion model parameters. A large model can also be referred to as a foundation model. Through the pre-training of a large model with a large amount of unlabeled corpus, a pre-trained model with more than one billion parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability, such as large language models (LLMs), multi-modal pre-training models, etc.

[0020] When large models are applied in practice, only a small number of samples are needed to fine-tune the pre-trained models for use in different tasks. Large models can be widely applied in fields such as natural language processing (NLP), computer vision, etc. Specifically, they can be applied to tasks in the field of computer vision such as visual question answering (VQA), image captioning (IC), image generation, etc., as well as tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, machine translation, etc. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0021] First, explain the noun terms involved in one or more embodiments of this specification.

[0022] Retrieval-augmented Generation (RAG for short): A method that combines information retrieval and generation technologies. Specifically, RAG first retrieves document fragments related to the input query from a large-scale knowledge database, and then inputs these contents together with the query into the generation model to assist in generating more accurate and information-rich content.

[0023] Large Language Model (LLM for short): A large language model refers to a language processing artificial intelligence model with a large number of parameters, capable of understanding and generating high-quality human language text, and it realizes the understanding of language structure and semantics by learning a vast amount of Internet text data.

[0024] Visual Language Model (VLM for short): A visual language model is a deep learning model used to process the relationship between images and text. It can understand and generate text describing the content of an image, or generate the corresponding image according to the text description.

[0025] Chart Generation: Chart generation is an inference task that converts data into visual charts (such as bar charts, line charts, pie charts, etc.) to more intuitively analyze and understand data trends and relationships. It is an important link in data analysis and report production.

[0026] Chain of Thought (CoT for short): The chain of thought is a reasoning method that gradually decomposes complex problems into a series of simple intermediate steps to more systematically solve problems or derive conclusions. This method helps to improve the transparency and accuracy of complex problem handling.

[0027] Prompt: A prompt in an AI conversation refers to the prompt or input data used to guide the model to generate a response. By carefully designing the prompt, it is possible to effectively guide the model to output results that meet expectations.

[0028] JavaScript Object Notation (JSON): A lightweight data interchange format that represents structured data in a human-readable text form, consisting of key-value pairs and nested structures.

[0029] Vega-Lite: A high-level visualization specification language based on JSON syntax, focused on quickly generating interactive charts through a concise declarative syntax.

[0030] Comma-Separated Values (CSV): A common file format for storing tabular data in plain text form, widely used for data exchange and storage.

[0031] Portable Network Graphics (PNG): A bitmap image format that uses a lossless compression algorithm, supports transparency channels (Alpha channels), and various color depths (such as 24-bit true color).

[0032] Scalable Vector Graphics (SVG): A two-dimensional vector graphics format based on XML syntax, which defines geometric shapes such as points, lines, and curves through mathematical formulas and can be scaled without loss of quality.

[0033] HTML5 Canvas: An element in the HTML5 standard for dynamically drawing bitmaps through JavaScript scripts, providing a pixel-level graphics operation interface. Based on an immediate rendering mode, it is suitable for generating high-performance interactive content such as games, dynamic charts, and image editing tools.

[0034] ECharts: A data visualization library based on JavaScript, providing a rich variety of chart types (such as line charts, heatmaps, 3D globes) and interactive features (such as dragging, zooming, data filtering).

[0035] Transformer: A deep learning model architecture based on the self-attention mechanism.

[0036] Currently, chart generation can convert data into visual charts (such as bar charts, line charts, pie charts, etc.). These charts can effectively display the relationships and trends between data, helping users more intuitively understand complex information. In the information age, the importance of data visualization has become even more prominent. It not only improves the efficiency of data interpretation but also provides an intuitive basis for decision-making. In the traditional chart-drawing process, generally, users have already thought out the corresponding drawing plan, and then through tools such as Excel and Tableau, they specify the chart type, confirm which fields in the data are mapped to the x-axis, y-axis, or other visual channels, and then manually adjust the chart layout and the styles and color schemes of various chart elements (such as titles, legends, background grids, other marks) until it meets the user's aesthetics and intentions. Drawing a chart requires a relatively high time cost.

[0037] Through a chart generation model, under the guidance of the task description text of the input chart generation task, the input raw data is used for chart generation, and chart data is output. The chart data is front-end rendered to obtain the target chart. However, due to the limitations of model training data and capabilities, the number of supported chart types is small, completely relying on the knowledge that the chart generation model has learned. In the case where the training set of the chart generation model does not cover all possible chart styles and formats, for some uncommon chart types, it is very easy to make mistakes during generation, resulting in insufficient accuracy and applicability of the generation results. Therefore, there is an urgent need for a more flexible and extensible chart generation method.

[0038] To solve the above problems, in one or more embodiments of this specification, an automated chart generation method based on multi-modal retrieval enhancement generation is proposed. By combining large models and multi-modal retrieval enhancement generation technologies, the task objective requirements and data characteristics are analyzed, so as to achieve automated and high-quality chart generation. This specification also relates to a multi-modal data processing platform, a chart generation device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail one by one in the following embodiments.

[0039] See Figure 1 , Figure 1 shows a flowchart of a chart generation method provided by an embodiment of this specification, including the following specific steps:

[0040] Step 102: Obtain the task description text and raw data of the chart generation task.

[0041] The embodiments of this specification are applied to applications, websites, or system platforms with multi-modal data generation capabilities, and are suitable for scenarios such as data analysis, work reports, and academic research.

[0042] The chart generation task is an inference task that transforms raw data into a visual chart according to the requirements of the task objective. For example, a chart generation task for the event occurrence frequency in the past year applied to product operation and maintenance. The task description text of the chart generation task is the chart generation requirement described in natural language, including the chart type, data mapping rules, style constraints, etc., and is used to describe the expectation for the generated target chart. For example, "Help me draw the relationship between xx variable and xx variable". The raw data of the chart generation task is structured or semi-structured input data, such as CSV, JSON, database tables, etc. The raw data can come from the data table file in the local database or be obtained from the remote database, which is not limited here.

[0043] Exemplarily, on a large language model platform with the ability to generate multi-modal data, the user inputs the task description text of the chart generation task at the front end: "Use a line chart to show the number of events per month", and imports the raw data of the data table file CSV from the local database, as shown in Table 1:

[0044]

[0045] Table 1

[0046] By obtaining the task description text and raw data of the chart generation task, it provides support for the task objective requirement description and task object data for subsequent parsing of the chart generation task and generation of chart data.

[0047] Step 104: Based on the task description text and raw data, parse the chart generation task to obtain the generation scheme of the chart generation task.

[0048] The generation scheme of the chart generation task is an intermediate instruction generated by parsing the task description and raw data, including the chart type, data field mapping logic, style configuration, etc. The generation scheme details how to process the raw data, select the appropriate chart type, configure visual elements, etc. For example, use a line chart to represent time series data and suggest color and title styles.

[0049] Based on the task description text and the original data, parse the chart generation task to obtain the generation scheme for the chart generation task. One optional method is: based on the task description text and the original data, conduct intention analysis and requirement analysis on the chart generation task to obtain the generation scheme for the chart generation task. Among them, the intention analysis and requirement analysis include, but are not limited to, at least one of data analysis, visualization requirement analysis, and picture style analysis. Another optional method is: based on the task description text and the original data, conduct intention analysis on the chart generation task to obtain the generation scheme for the chart generation task. Another optional method is: based on the task description text and the original data, conduct requirement analysis on the chart generation task to obtain the generation scheme for the chart generation task, which is not limited here.

[0050] Exemplarily, based on the task description text "Use a line chart to display the number of events each month" and the original data in Table 1 above, conduct visualization requirement analysis and data analysis on the chart generation task respectively to obtain the generation scheme for the chart generation task:

[0051] Detailed design specifications: - Chart type and layout: Use a line chart and display it in a single chart. - Color scheme and style: Use blue as the line color, white as the background, and gray as the border. - Axis configuration and ratio: The X-axis is for months (1 - 12), the Y-axis is for the number of events, and the ratio is automatically adjusted according to the data range. - Labels, titles, and annotations: The X-axis label is "Month", the Y-axis label is "Number of Events", and the chart title is "Trend of Monthly Event Quantity Changes". - Legend position and format: No legend is required. - Gridlines and other reference elements: Fine gray gridlines are enabled for both the X-axis and Y-axis.

[0052] Based on the task description text and the original data, parsing the chart generation task to obtain the generation scheme for the chart generation task provides a retrieval basis for subsequent retrieval and provides scheme support for subsequent generation of chart data.

[0053] Step 106: Based on the generation scheme, retrieve a reference sample chart from the sample chart database.

[0054] The sample chart database is a database that stores sample chart templates, including chart configurations (such as Vega-Lite / JSON), visual styles, applicable scenario labels, etc. The reference sample chart is a chart template retrieved from the sample chart database that matches the generation scheme, including chart configurations (such as Vega-Lite specifications), visual styles (color matching, layout), applicable scenario labels, etc.

[0055] Based on the generation scheme, retrieve reference sample charts from the sample chart database. An optional way is: based on the generation scheme, retrieve reference sample charts from the sample chart database through multimodal retrieval. Another optional way is: based on the generation scheme, retrieve reference sample charts from the sample chart database through heuristic retrieval. Another optional way is: based on the generation scheme, retrieve reference sample charts from the sample chart database through cross-modal retrieval, which is not limited here.

[0056] Exemplarily, adopt a hybrid retrieval scheme of multi-way recall. Based on the generation scheme, retrieve reference sample charts that meet the preset retrieval conditions from the sample chart database.

[0057] Based on the generation scheme, retrieve reference sample charts from the sample chart database to supplement the knowledge limitations of the multimodal generation model, get rid of the dependence on the knowledge of pre-trained models, and also pass through the diverse chart instances that actually exist, thereby enriching the styles and formats of chart generation and solving the problems of the accuracy and applicability of the generation results caused by insufficient coverage of the model training set.

[0058] Step 108: Through the multimodal generation model, based on the reference sample charts, perform a chart generation task on the original data using the generation scheme to generate the target chart.

[0059] The multimodal generation model is a deep learning model with the ability to generate multimodal data, which can process text, structured data, and visual style information simultaneously, including but not limited to large language models or deep learning models based on the Transformer architecture. For example, the multimodal generation model supports end-to-end generation of text-to-chart configuration (such as converting the instruction "line chart" into Vega-Lite code), and can also parse the style features in the reference samples for transfer learning. The target chart is the visualization result of the rendered chart data, and the target chart can be a static image (PNG / SVG) or an interactive component (HTML5 Canvas).

[0060] Through the multimodal generation model, based on the reference sample charts, perform a chart generation task on the original data using the generation scheme to generate the target chart. An optional way is: through the multimodal generation model, based on the reference sample charts, perform a chart generation task on the original data using the generation scheme to directly generate the target chart. Another optional way is: through the multimodal generation model, based on the reference sample charts, perform a chart generation task on the original data using the generation scheme to generate chart data, and generate the target chart based on the chart data, which is not limited here.

[0061] Exemplarily, through the large language model, using the above reference sample charts as supplementary knowledge for retrieval-enhanced generation, perform a chart generation task on the original data in Table 1 using the above generation scheme to obtain chart data:

[0062] import pandas as pd # Import the pandas library for data processing.

[0063] import seaborn as sns # Import the seaborn library for advanced data visualization.

[0064] import matplotlib.pyplot as plt # Import the matplotlib.pyplot module for plotting charts.

[0065] plt.rcParams['font.sans-serif'] = ['SimHei'] # Use the SimHei font to support Chinese display.

[0066] plt.rcParams['axes.unicode_minus'] = False # Solve the problem of negative sign display.

[0067] def preprocess(data): # Ensure that the Month column is of string type for display in the chart.

[0068] data['Month'] = data['Month'].astype(str) # Convert the data type of the 'Month' column to string.

[0069] return data # Return the processed data.

[0070] def plot(data):

[0071] data = preprocess(data) # Call the preprocessing function to process the data.

[0072] sns.set_style("whitegrid") # Set the background style to white grid lines.

[0073] plt.figure(figsize=(10, 6)) # Set the chart size to 10 inches in width and 6 inches in height.

[0074] ax = sns.lineplot(data=data, x='Month', y='Event_Count',

[0075] marker='o', markersize=8, linewidth=2,

[0076] color='#1f77b4') # Use seaborn to draw a line chart, specifying parameters such as data, X-axis, Y-axis, markers, etc.

[0077] plt.title('Monthly Event Quantity Variation Trend', pad = 20, fontsize = 16) # Set the chart title and set the spacing and font size.

[0078] plt.xlabel('Month', fontsize = 14) # Set the X-axis label text and font size.

[0079] plt.ylabel('Event Quantity', fontsize = 14) # Set the Y-axis label text and font size.

[0080] plt.xticks(rotation = 45) # Rotate the X-axis labels by 45 degrees to prevent label overlap.

[0081] for x, y in zip(data['Month'], data['Event_Count']):

[0082] plt.annotate(f'{int(y)}', # Add annotations at each data point to display the event quantity.

[0083] (x, y), # The position of the annotation (data point position).

[0084] textcoords = "offset points", # Specify the unit of the position offset of the annotation relative to the point.

[0085] xytext = (0, 10), # The offset of the annotation relative to the data point.

[0086] ha = 'center', # Set the horizontal alignment to center alignment.

[0087] fontsize = 12) # Set the annotation font size.

[0088] plt.tight_layout() # Automatically adjust the subplot parameters to fill the entire image area.

[0089] plt.savefig('plot.png', dpi = 300, bbox_inches = 'tight') # Save the chart as a PNG file with a resolution of 300 dpi to ensure all content is within the image.

[0090] plt.close() # Close the current figure and release memory.

[0091] Convert the chart data into the ECharts standard configuration, and call the front-end rendering engine of ECharts in the HTML page to dynamically render an interactive line chart. This line chart natively supports displaying values on mouse hover and triggering events by clicking on data points (the drill-down analysis logic can be extended). Render visual features such as the blue line, gray grid lines, and rotated labels defined in the generation scheme. The size of the chart container automatically adjusts with the browser window, and the spacing is automatically compressed when displayed on mobile devices to prevent element overlap. The above interactive line chart is suitable for dashboard scenarios that need to be displayed online, and users can explore data details by zooming and dragging. The specific target chart is as Figure 2 shown Figure 2 shows a schematic diagram of the target chart in a chart generation method provided by an embodiment of this specification:

[0092] The horizontal axis represents months, and the vertical axis represents the number of events. Connecting each data point with a line chart represents the changing trend of the number of events per month.

[0093] In the embodiments of this specification, by combining the analysis of the task description text and the original data, a specific generation scheme is generated, and a reference sample chart is retrieved from the sample chart database to supplement the knowledge limitations of the multi-modal generation model, getting rid of the dependence on the knowledge of pre-trained models. It also passes through diverse chart instances that actually exist, thus enriching the styles and formats of chart generation, and solving the problems of the accuracy and applicability of the generation results caused by insufficient coverage of the model training set. Guided by the generation scheme, the multi-modal generation model generates charts according to the reference samples, ensuring that even for uncommon chart types, target charts with high accuracy and high applicability can be generated, effectively improving the diversity and quality of chart generation.

[0094] In an optional embodiment of this specification, step 104 includes the following specific steps: perform data analysis on the original data to obtain the data element information of the original data; based on the task description text and the data element information, jointly analyze the chart generation task to obtain the generation scheme of the chart generation task.

[0095] The data element information of the original data is a set of statistical features and structural features extracted from the original data. The data element information of the original data includes but is not limited to: field names, data types (numeric, categorical, etc.), missing value situations, distribution characteristics (such as mean, median, mode, standard deviation, etc.), and correlation coefficients between fields (such as Pearson correlation coefficient).

[0096] Perform data analysis on the original data to obtain the data element information of the original data. An optional method is to perform data analysis on the original data in terms of statistical feature dimensions and sampling feature dimensions to obtain the data element information of the original data. Another optional method is to perform data analysis on the original data in terms of statistical feature dimensions to obtain the data element information of the original data. Another optional method is to perform data analysis on the original data in terms of sampling feature dimensions to obtain the data element information of the original data, which is not limited here.

[0097] Exemplarily, perform data analysis on the original data in Table 1 above in terms of statistical feature dimensions and sampling feature dimensions to obtain the data element information of the original data: "Field type": {"Month": {"Type": "Discrete integer", "Range": [1, 12], "Number of enumerated values": 12}, "Number of events": {"Type": "Continuous numerical value", "Range": [17, 40], "Standard deviation": 8.2}}; "Data distribution": {"Month": {"Periodicity": "Monthly cycle (12 months)", "Interval unit": 1}, "Number of events": {"Range": 23, "Mean": 28.1}}; "Data quality": {"Missing value ratio": 0%, "Outlier detection": "None"}}.

[0098] Through the large language model, based on the task description text and data element information, jointly analyze the chart generation task: 1. Task analysis: - What is the industry background and target audience? - What insights need to be conveyed? - What are the common visualization methods in this field? 2. Data element information analysis: - What are the key features of the provided data? - What relationships or patterns need to be highlighted? 3. Visualization generation plan, according to the data characteristics and project background: - Which chart types are the most effective? - What alternative options have been considered and why were they rejected? - If necessary, how should multiple elements be combined?

[0099] Obtain the generation plan for the above chart generation task.

[0100] In the embodiments of this specification, by performing data analysis on the data element information of the original data, the deep data characteristics required by the task target for the original data can be captured more accurately, and then a more reasonable and effective chart generation plan can be formulated, which helps to improve the accuracy and applicability of the generated charts.

[0101] In an optional embodiment of this specification, perform data analysis on the original data to obtain the data element information of the original data, including the following specific steps: perform data statistics on the original data to obtain the statistical information of the original data, and extract a preset number of data from the original data as sampling data; combine the statistical information and the sampling data to obtain the data element information of the original data.

[0102] The statistical information of the original data is a set of quantitative features extracted from the original data, including but not limited to: the number, mean, variance, and quantiles of each numerical column, the number of distinct values and high-frequency values of non-numerical columns. The sampled data are typical data entries extracted from the original data according to a preset rule, which is used to assist the model in understanding the data structure (such as the mapping relationship between fields), and the extraction methods include but not limited to random sampling, stratified sampling, and head-tail sampling. For example, N data entries are extracted from the data table as the sampled data.

[0103] Exemplarily, perform data statistics on the original data in Table 1 to obtain the statistical information of the original data, and extract N sampled data from the original data, and combine the statistical information and the sampled data to obtain the data element information of the original data:

[0104] Month: Total quantity: 12; Average value: 6.5; Standard deviation: 3.60555; Minimum value: 1; First quartile: 3.75; Median: 6.5; Third quartile: 9.25; Maximum value: 12. Number of events: Total quantity: 12; Average value: 27.1667; Standard deviation: 7.45695; Minimum value: 17; First quartile: 21.5; Median: 26.5; Third quartile: 32.75; Maximum value: 40.

[0105] In the embodiments of this specification, by combining data statistics and sampling to obtain data element information, it can not only comprehensively capture the distribution characteristics and potential patterns of the data, but also effectively assist the model in understanding the data structure and the mapping relationship between fields, which not only improves the accuracy and efficiency of the chart generation task, but also makes the generated visualization results closer to the requirements of the task target.

[0106] In an alternative embodiment of this specification, before jointly analyzing the chart generation task based on the task description text and the data element information to obtain the generation scheme of the chart generation task, the following specific steps are further included: obtaining a reference picture of the chart generation task; performing picture style analysis on the reference picture to obtain the visual description information of the reference picture; jointly analyzing the chart generation task based on the task description text and the data element information to obtain the generation scheme of the chart generation task, including the following specific steps: jointly analyzing the chart generation task in a multi-modal manner based on the task description text, the data element information, and the visual description information to obtain the generation scheme of the chart generation task.

[0107] The reference picture for the chart generation task is a visual picture used for style transfer in the image generation task. For example, the example picture actively uploaded by the user (such as the historical chart of the same type of project scenario provided by the user), or, for another example, the candidate template picture recommended by the system (a highly relevant example automatically matched according to the task description). The visual description information of the reference picture is the description information of the visual features extracted from the reference picture, including but not limited to: chart type, color scheme and style, axis configuration, labels, title and notes, legend position, etc.

[0108] Perform picture style analysis on the reference picture to obtain the visual description information of the reference picture. One optional method is: use an image parsing tool to perform picture style analysis on the reference picture to obtain the visual description information of the reference picture. Another optional method is: use a deep learning model to perform picture style analysis on the reference picture to obtain the visual description information of the reference picture. This is not limited here.

[0109] Based on the task description text, data element information, and visual description information, perform multimodal joint parsing on the chart generation task to obtain the generation scheme for the chart generation task. One optional method is: based on the task description text, data element information, and visual description information, perform intention analysis and requirement analysis on the chart generation task to obtain the generation scheme for the chart generation task. Among them, the intention analysis and requirement analysis include but are not limited to at least one of data analysis, visualization requirement analysis, and picture style analysis.

[0110] Exemplarily, the user uploads the picture of the reference chart for the chart generation task at the front end, performs picture style analysis on the reference picture, and obtains the visual description information of the reference picture: Chart type: line chart. Main color system (such as HEX color value #2E75B6), background transparency, width-to-height ratio of the chart container. Axes: Tick mark direction (inward / outward), axis line thickness, label rotation angle. Data markers: Line width of the line (2px), shape of the marker points (circle / square), font size of the data labels (12pt). Text layout: Title font (Microsoft YaHei), legend position (upper right corner), note alignment method (center alignment). Hover highlight effect (such as the data points are enlarged by 1.2 times), animation transition duration (300ms).

[0111] Based on the task description text "Use a line chart to display the number of events each month", the original data in Table 1 above, and the visual description information, perform visualization requirement analysis, data analysis, and picture style analysis on the chart generation task respectively to obtain the generation scheme for the above chart generation task.

[0112] In the embodiments of this specification, by introducing a reference picture in the visual modality as a reference for the image generation task and performing multimodal joint parsing in combination with the task description text and data element information, not only is the dimension of demand understanding for chart generation enriched, but also the visual accuracy and style consistency of the generated charts are significantly improved. The data in the generated charts can not only accurately reflect the data characteristics and user intentions, but also match specific visual style requirements, enhancing the professionalism and aesthetics of the target charts and better meeting the diverse needs of users.

[0113] In an alternative embodiment of this specification, the picture style of the reference picture is analyzed to obtain the visual description information of the reference picture, including the following specific steps: The picture style of the reference picture is analyzed through a vision-language model to obtain the visual description information of the reference picture.

[0114] The vision-language model is a deep learning model for processing the relationship between images and texts. The vision-language model can understand and generate texts describing the content of images, or generate corresponding images according to text descriptions.

[0115] In the embodiments of this specification, by analyzing the reference picture through a vision-language model, its visual description information can be accurately extracted, not only improving the accuracy of style transfer, further enhancing the professionalism and aesthetics of the target charts, but also better meeting the diverse needs of users.

[0116] In an alternative embodiment of this specification, step 104 includes the following specific steps: Through a solution generation model, based on the task description text and the original data, the task objective of the chart generation task is parsed using a chain of thought to obtain the task objective analysis result, and based on the task description text and the original data, under the guidance of the task objective analysis result, the task implementation path of the chart generation task is parsed using a chain of thought to obtain the generation solution design result. Based on the task objective analysis result and the generation solution design result, a generation solution for the chart generation task is generated.

[0117] The solution generation model is a deep learning model with the ability to generate solutions, capable of processing text data, including but not limited to large language models or deep learning models based on the Transformer architecture. For example, the solution generation model supports end-to-end generation from text to text. The chain of thought is a logical framework for a multimodal generation model to perform complex reasoning tasks. By explicitly decomposing the relevance between the task objective and the implementation path, a step-by-step reasoning chain is formed. The task objective analysis result is a natural language description of the essence of the task objective requirements. The task implementation path of the chart generation task is a set of execution strategies based on the task objective. The generation solution design result is a structured output after multimodal reasoning.

[0118] Exemplarily, through a large language model, based on the task description text and the original data, using the chain of thought (task analysis: - What is the industry background and target audience? - What insights need to be conveyed? - What are the common visualization methods in this field? Data meta-information analysis: - What are the key features of the provided data? - What relationships or patterns need to be highlighted?) to analyze the task objectives of the chart generation task, and obtain the task objective analysis results: 1. Industry background positioning: Identify the scenario label as the operation and maintenance monitoring field, and the target audience as IT operation and maintenance teams and department managers; the core requirement is to reflect the periodic fluctuation law of the number of events and assist in resource scheduling decisions. 2. Key features of the data: The number of events reaches its peak in September (40 times) and drops to a low point in December (22 times). 3. What relationships or patterns to highlight: The number of events is frequent in the third quarter (possibly related to the increase in server load); the number of events drops significantly at the end of the year (or related to the off-season of the project). 4. Visualization generation scheme adaptation: Priority of applicable chart types: Line chart (emphasizing time trend) > Bar chart (suitable for discrete comparison) > Heat map (requiring multi-dimensional data).

[0119] Based on the task description text and the original data, under the guidance of the task objective analysis results, using the chain of thought (visualization generation scheme, according to data characteristics and project background: - Which chart types are the most effective? - What alternative solutions have been considered and why were they rejected? - How should multiple elements be composed if necessary?) to analyze the task implementation path of the chart generation task, and obtain the generation scheme design results: 1. Data mapping rules: X-axis: Bind the "month" field and set it as a discrete categorical axis (to avoid scale errors caused by automatically filling in the whole year); Y-axis: Bind the "number of events" field and use a linear scale (display the original values without aggregation calculation). 2. Visual enhancement strategy: Peak annotation: Add a red outer frame (HEX #FF4500) to the data point in September and label "Annual peak"; Trend line overlay: Add a gray dashed trend line (quadratic polynomial fitting) to show the long-term change direction. 3. Interactive logic design: Hover feedback: Display the proportion of the event classification in the current month (associated with the sub-dataset); Drill-down function: Double-click on the peak month to jump to the distribution map of the hourly events on that day.

[0120] Based on the task objective analysis results and the generation scheme design results, generate the generation scheme for the above chart generation task.

[0121] In the embodiments of this specification, by using the chain of thought to conduct a detailed analysis of the task objectives and implementation paths, not only is the understanding of the requirements of the chart generation task refined, but also the accuracy and practicality of the generation scheme are significantly improved, enabling the generated chart to accurately reflect the key features of the data and the project background, and enhancing the effectiveness of information transmission and the professionalism of visual expression.

[0122] In an alternative embodiment of this specification, the generation scheme includes at least one of a target chart type, a reference picture of the chart generation task, and visual description information of the reference picture of the chart generation task; step 106 includes at least one of the following:

[0123] Based on the target chart type, perform keyword matching retrieval from the classification label field of the sample database to obtain a reference sample chart, where the classification label field is the field in the sample database used to identify the chart type; based on the semantic vector of the visual description information, perform vector matching retrieval from the preset semantic vectors of the sample database to obtain a reference sample chart, where the preset semantic vector is the semantic coding vector in the sample database used to identify the text description information; based on the visual vector of the reference picture, perform vector matching retrieval from the preset visual vectors of the sample database to obtain a reference sample chart, where the preset visual vector is the visual coding vector in the sample database used to identify the visual description information.

[0124] The sample database contains data tables of several high-quality chart generation schemes, and the data tables are stored in this multi-modal database of the sample database. Each piece of data in the data table mainly includes information such as chart category, sub-category, visualization purpose, visual description, applicable scenario, data characteristics, sample data and code, sample picture vector, etc. The sample data in the sample database can be generated using LLM and verified through manual selection. Optionally, the sample database is open and extensible. For the diverse chart drawing requirements of different industries, only by adding the corresponding sample chart information to the sample database, charts that meet the target chart style can be generated to support more types and more styles of chart generation tasks.

[0125] The classification label field is the field in the sample database used to identify the chart type, which can be an exactly matching keyword or a set of fuzzy extended labels. The preset semantic vector is the semantic coding vector in the sample database used to identify the text description information, and the similarity between different text descriptions can be compared by calculating the similarity between vectors. The preset visual vector is the visual coding vector in the sample database used to identify the visual description information, and the similarity between different visual descriptions can be compared by calculating the similarity between vectors.

[0126] The target chart type is a specific chart type determined according to the task target requirements, such as bar chart, pie chart, line chart, etc. The semantic vector of the visual description information is the semantic coding vector of the visual description information. The visual vector of the reference picture is the visual coding vector of the reference picture.

[0127] Exemplarily, parse the chart_type field in the generation scheme: enhanced_line_chart, extract the keyword "enhanced line chart" of the standardized classification label, perform exact matching and fuzzy expansion of the keyword in the classification label field of the sample database, and recall the reference sample charts. Exemplarily, encode the visual description text "blue line, grey grid lines, peak red box annotation" in the generation scheme into a semantic vector, calculate the cosine similarity between this vector and the semantic vectors in the visual description field of the sample database, and retain the Top-K similar results as the recall results to recall the reference sample charts. Exemplarily, the user uploads a "dark blue gradient line chart with hover tooltip" as a reference picture, encodes the reference picture into a visual vector, calculates the cosine similarity between this vector and the preset visual vectors of the reference sample charts in the sample database, and retains the Top-K similar results as the recall results to recall the reference sample charts.

[0128] In the embodiments of this specification, through the multi-modal retrieval method, the target chart type, semantic vectors, and visual vectors are accurately matched or fuzzily expanded, effectively improving the recall accuracy and diversity of the reference sample charts. This not only enhances the flexibility and customization level of chart generation but also ensures that the generated target charts meet both the data display requirements and the user's visual expectations.

[0129] In an alternative embodiment of this specification, based on the generation scheme, retrieving the reference sample charts from the sample chart database includes the following specific steps: Based on the generation scheme, retrieve multiple recalled sample charts from the sample chart database to obtain the sorting results of the multiple recalled sample charts; use a pre-trained sorting model to sort the multiple recalled sample charts; based on the sorting results of the multiple recalled sample charts, screen out the reference sample charts.

[0130] The recalled sample charts are a set of chart templates with a relatively high matching degree to the task objective requirements initially screened out from the sample chart database during the multi-modal retrieval process based on the generation scheme. The recalled sample charts are a candidate set retrieved from a large number of sample charts through methods such as keyword matching, semantic vector matching, or visual vector matching, and are the preliminary result set close to the task objective requirements of the chart generation task, but may not have been precisely sorted and optimized yet. The pre-trained sorting model is a machine learning model. The sorting model has been pre-trained with a large number of samples and can sort a given set of items (such as the recalled sample charts) according to specific criteria or objectives. This model is usually based on deep learning architectures such as Transformer or RNN. The sorting results of the multiple recalled sample charts are an ordered list obtained after the pre-trained sorting model sorts the multiple recalled sample charts, which contains the results of all the recalled sample charts arranged from high to low according to factors such as their relevance, quality, and applicability to the chart generation task.

[0131] Exemplarily, a pre-trained ranking model (such as a deep neural network based on multi-modal contrastive learning) is used to refine the ranking of hundreds of recalled sample charts retrieved from multiple sources. The ranking model evaluates the matching degree through the following dimensions: 1. Functional adaptation: The compatibility between the chart type and data characteristics (such as whether time series data is suitable for a line chart). 2. Style consistency: The similarity of the color scheme (such as the color difference between the main color and the reference picture) and label layout (such as the rotation angle of axis labels). 3. Interaction logic fit: Whether it supports the interaction requirements defined in the generation scheme (such as hover tips, drill-down analysis).

[0132] After the ranking model outputs the matching scores of each recalled sample chart, they are sorted in descending order of scores, and the top-K are intercepted as reference sample charts.

[0133] In the embodiments of this specification, the refined ranking of the retrieval results improves the relevance, quality, and user satisfaction of the reference sample charts, ensures the matching degree and visual consistency of the generation scheme, optimizes the accuracy and professionalism of chart generation, and effectively improves the chart customization level and user experience.

[0134] In an optional embodiment of this specification, before step 108, the following specific steps are further included: Through a preset prompt information template, the task description text, original data, generation scheme, and reference sample charts are integrated to obtain the prompt information for the chart generation task;

[0135] Correspondingly, step 108 includes the following specific steps: Input the prompt information into a multi-modal generation model to perform the chart generation task on the original data and obtain chart data.

[0136] The preset prompt information template is a pre-set structured text framework for integrating the input information of the chart generation task (such as task description text, generation scheme, reference sample charts, etc.) into prompt information in a unified format that can be parsed by the multi-modal generation model. The prompt information for the chart generation task is multi-modal input data integrated through the preset template, including context information such as task objective requirements, generation scheme, and reference samples. As the input of the multi-modal generation model, it guides the multi-modal generation model on how to convert the original data into visual chart data that meets the user's expectations.

[0137] Through the preset prompt information template, the task description text, original data, generation scheme, and reference sample charts are integrated to obtain the prompt information for the chart generation task. Some of the content has been given in the above examples and will not be elaborated here:

[0138] Exemplarily, the prompt information is: "You are a data visualization expert. Now there is a task of generating Python visualization code. You need to first read the reference sample chart code, and then combine the original data and the task description text, and adopt the recommended visualization generation scheme to implement the visualization code for the original data. ## Reference sample chart starts; Target chart type: Basic line chart; Sample data format:...; Reference sample chart code:...; ## Reference sample chart ends. Next, the characteristics of the user data and the requirements of the task objective are given. ## User data starts: Task description text: Use a line chart to show the number of events per month; Recommended visualization generation scheme:...; Original data:...; Metadata information of the original data:...; # User data ends. Now, please refer to the sample and combine the actual situation of the user data to generate the visualization chart code data that meets the requirements."

[0139] Input the above prompt information into the large language model, and based on the reference sample chart, execute the chart generation task on the original data using the generation scheme to obtain the above chart data.

[0140] In the embodiments of this specification, through the preset prompt information template and the dynamically generated prompt information, the system can deeply integrate the user requirements, the parsing scheme, and the reference samples in the external knowledge base to form high-quality inputs for the multi-modal generation model, significantly improving the accuracy, style richness, and cross-scenario adaptability of chart generation, especially performing outstandingly when dealing with complex interaction requirements or niche chart types.

[0141] In an optional embodiment of this specification, step 108 includes the following specific steps: Through the multi-modal generation model, based on the reference sample chart, execute the chart generation task on the original data using the generation scheme to obtain the chart data; Render the target chart based on the chart data.

[0142] The chart data is the chart code data generated based on the generation scheme and the reference sample template, including complete visualization parameters. For example: 1. Structured configuration: Define the chart type (such as a line chart), data mapping relationship (such as binding the month field to the x-axis), axis range, and scale in the Vega-Lite specification or JSON format. 2. Visual parameters: Include details such as the color scheme (such as HEX color value #1f77b4), line width, marker point style, font size, etc. 3. Interaction logic: If the reference sample includes dynamic interactions (such as hover prompts), inherit its event response configuration.

[0143] Through a multimodal generation model, based on a reference sample chart, a chart generation task is performed on the original data using a generation scheme to obtain chart data. An optional way is: through a multimodal generation model, using the reference sample chart as supplementary knowledge for retrieval-augmented generation, a chart generation task is performed on the original data using a generation scheme to obtain chart data. Another optional way is: through a multimodal generation model, using the reference sample chart as the iterative starting point for iterative optimization generation, a chart generation task is performed on the original data using a generation scheme to obtain chart data, which is not limited here.

[0144] Render the target chart based on the chart data. An optional way is: through a front-end rendering engine, render the target chart based on the chart data. Another optional way is: through a front-end rendering template, render the target chart based on the chart data, which is not limited here.

[0145] Exemplarily, through a large language model, using the above reference sample chart as supplementary knowledge for retrieval-augmented generation, a chart generation task is performed on the original data in Table 1 using the above generation scheme to obtain chart data, and the front-end rendering engine of ECharts is called in the HTML page to dynamically render an interactive line chart.

[0146] Guiding the multimodal generation model to generate charts according to the reference sample through the generation scheme ensures that high-accuracy and highly applicable chart data can be generated even for uncommon chart types. Based on this, the target chart is rendered, which can improve the rendering effect.

[0147] In an optional embodiment of this specification, after rendering the target chart based on the chart data, the following specific steps are further included: if the rendering fails, obtain the error message; through the multimodal generation model, based on the reference sample chart, using the error message as the correction feedback data, perform the chart generation task on the original data again using the generation scheme to obtain updated chart data; re-render the target chart based on the updated chart data.

[0148] The error message is the error diagnosis data returned during the chart rendering process, including key information such as the error type, location description, and context data, which is used to locate logical conflicts or technical defects in the generation scheme or chart configuration.

[0149] Through the multimodal generation model, based on the reference sample chart, using the error message as the correction feedback data, perform the chart generation task on the original data again using the generation scheme to obtain updated chart data. An optional way is: through the multimodal generation model, using the reference sample chart as supplementary knowledge for retrieval-augmented generation, using the error message as the correction feedback data, perform the chart generation task on the original data again using the generation scheme to obtain updated chart data.

[0150] To re-render the target chart based on updated chart data, one optional way is to re-render the target chart through the front-end rendering engine based on the updated chart data. Another optional way is to render the target chart through the front-end rendering template based on the updated chart data. This is not limited here.

[0151] Exemplarily, when an error occurs during the rendering of the generated chart data, the large model will be called for inspection and rewriting. The corresponding prompt information is as follows:

[0152] You are a data visualization expert. Now there is a task of modifying Python visualization code. You need to first read the source code, and then refer to the user's modification suggestions (or error messages) to modify the source code to implement chart drawing:

[0153] Value Error: Cannot convert nontype column to str.

[0154] It is necessary to ensure that the 'Month' column does not cause a 'Value Error' when converted to string type. We can use the `astype(str)` method to ensure the correctness of the conversion. Obtain the modified chart data and re-render the target chart shown below through the front-end rendering engine based on the above updated chart data. Figure 2 the target chart shown.

[0155] In the embodiments of this specification, in the case of rendering failure, based on the error message, adjustment generation and re-rendering are completed. Through the intelligent error diagnosis and correction feedback mechanism, the stability and accuracy of chart generation are significantly improved. It can not only quickly locate and solve logical conflicts or technical defects in the generation scheme, but also automatically optimize the chart configuration to ensure that the final rendering result meets expectations, enhancing the user experience and satisfaction.

[0156] In an optional embodiment of this specification, after rendering the target chart based on the chart data, the following specific steps are further included: If the rendering is successful, through the visual language model, based on the task description text and the generation scheme, analyze whether the target chart meets the task objective requirements and the scheme constraints of the image generation task to obtain the analysis result; if the analysis result is in line, output the target chart to the user terminal; if the analysis result is not in line and includes adjustable parameters of the generation scheme, through the multimodal generation model, adjust the generation scheme based on the adjustable parameters, and based on the reference sample chart, re-execute the chart generation task on the original data using the adjusted generation scheme to obtain updated chart data, and re-render the target chart based on the updated chart data.

[0157] The solution constraints of the generation solution are a set of immutable parameters defined in the generation solution. For example, the chart type (such as a line chart), the key data mapping rules (such as the X-axis binding to the month field), and the style requirements explicitly specified by the user (such as prohibiting the use of red).

[0158] The analysis results include a binary classification conclusion (compliant / non-compliant) output by the visual language model, accompanied by a detailed description of the non-compliant items (such as "The unit is not marked on the Y-axis").

[0159] The adjustable parameters of the generation solution are parameters that allow dynamic changes. For example, style parameters such as color saturation, grid line thickness, and font size that do not affect the expression of core data.

[0160] To re-render the target chart based on the updated chart data, one optional way is: through the front-end rendering engine, re-render the target chart based on the updated chart data. Another optional way is: through the front-end rendering template, render the target chart based on the updated chart data. This is not limited here.

[0161] Exemplarily, if the rendering is successful, through the visual language model, based on the task description text and the generation solution, analyze whether the target chart meets the task objective requirements of the image generation task and the solution constraints of the generation solution, and obtain the analysis results:

[0162] {"Function verification": {"Chart type matching degree": "Line chart (compliant)", "Data mapping correctness": "X-axis bound to month (compliant), Y-axis bound to number of events (compliant)"}, "Style verification": {"Main color": "Detected blue line (HEX #1f77b4, compliant)", "Title text": "Detected title 'Monthly trend of the number of events' (compliant)", "Exception item": "The unit is not displayed on the Y-axis (does not meet the 'unit needs to be marked' constraint in the generation solution)"}}

[0163] Output the analysis result as "non-compliant" and mark the exception items that need to be corrected.

[0164] Adjust the generation solution based on the adjustable parameters through the multi-modal generation model:

[0165] # Y-axis configuration in the original generation solution: "yAxis": {"label": "Number of events", "unit": "" # Unit not defined}

[0166] # Adjusted generation solution: "yAxis": {"label": "Number of events (times)", "unit": "times", "fontSize": 14 # Synchronously enlarge the font size to improve readability}.

[0167] And based on the reference sample chart, the chart generation task is re-executed on the original data using the adjusted generation scheme to obtain updated chart data, and the target chart as shown in Figure 2 is re-rendered based on the updated chart data. After re-rendering, the visual language model detects that the unit has been added to the Y-axis and the style completely conforms to the generation scheme, and the output analysis result is "conforms", and the target chart is displayed through the user terminal.

[0168] In the embodiments of this specification, a post-generation self-reflection logic is introduced. Through the method of multi-round feedback iteration, functions of automatic error repair and automatic optimization of drawing results are realized, further improving the quality of the generated target chart, ensuring that the final rendering result meets the expectations, and enhancing the user experience and satisfaction.

[0169] Corresponding to the above-mentioned multiple embodiments, Figure 3 FIG. shows a schematic flowchart of a chart generation method provided by an embodiment of this specification, as Figure 3 shown:

[0170] Start: The process starts from the "Start" node. User input stage: The user inputs the original data, task description text, and reference picture. Intent analysis and requirement analysis stage: Data analysis is performed on the original data to obtain metadata information, and at the same time, picture style analysis is performed on the reference picture to obtain visual description information. Based on the task description text, metadata information, and visual description information, multi-modal joint parsing of the visualization chart generation task is performed to obtain a generation scheme. Multi-modal retrieval stage of chart category: Based on the generation scheme, multi-modal hybrid retrieval is performed, and the retrieval results are refined to screen out high-quality reference sample charts. Chart generation stage: Based on the metadata information, generation scheme, and reference sample chart, prompt information is integrated, the chart generation task is executed, the generated chart data is chart-rendered. If the rendering fails, an error message is returned and adjustment and generation are performed again. If the rendering is successful, it is checked whether the rendered chart meets the task objective requirements of the image generation task and the scheme constraints of the generation scheme. If it does not meet the requirements, adjustable parameters are returned and adjustment and generation are performed again. When the chart meets the requirements, the process is completed at the "End" node.

[0171] The following combines the attached Figure 4 , taking the application of the chart generation method provided by this specification in the product operation scenario as an example, to further illustrate the chart generation method. Among them, Figure 4 FIG. shows a flowchart of the processing process of a chart generation method applied to the product operation scenario provided by an embodiment of this specification, including the following specific steps:

[0172] Step 402: Obtain the task description text, original data, and reference picture of the generation task of the product operation chart.

[0173] For example, obtain the task description text, original data (such as sales data, user growth data, etc.), and reference pictures (such as charts of previous successful marketing campaigns) for the task of generating product operation charts for product performance, market trends, or user behavior analysis.

[0174] Step 404: Perform data statistics on the original data to obtain statistical information of the original data, extract a preset number of sampling data from the original data, combine the statistical information and the sampling data, and obtain the data element information of the original data.

[0175] For example, count the quarterly repurchase rate and extract sample data to generate data element information.

[0176] Step 406: Through a visual language model, perform picture style analysis on the reference picture to obtain the visual description information of the reference picture.

[0177] For example, through a visual language model, perform picture style analysis on the reference picture to obtain the visual description information of the chart type, color scheme, and label style of the promotion chart.

[0178] Step 408: Through a large language model, based on the task description text and the original data, use the chain of thought to analyze the task objectives of the task of generating product operation charts to obtain the task objective analysis results, and based on the task description text, the original data, and the visual description information, under the guidance of the task objective analysis results, use the chain of thought to analyze the task implementation path of the task of generating product operation charts to obtain the generation scheme design results. Based on the task objective analysis results and the generation scheme design results, generate the generation scheme for the task of generating product operation charts.

[0179] For example, for the task of "analyzing the differences in retention rates among different user segments from Q1 to Q3 in 2024", design a grouped bar chart scheme to compare the quarterly retention rates of young users (18 - 24 years old) and senior users (35+ years old), and mark the significant difference values.

[0180] Step 410: Based on the target chart type, perform keyword matching retrieval from the classification label fields of the sample database to obtain the recalled sample charts. Based on the semantic vectors of the visual description information, perform vector matching retrieval from the preset semantic vectors of the sample database to obtain the recalled sample charts. Based on the visual vectors of the reference picture, perform vector matching retrieval from the preset visual vectors of the sample database to obtain the recalled sample charts. Through a pre-trained ranking model, rank the multiple recalled sample charts, and based on the ranking results of the multiple recalled sample charts, screen out the reference sample charts.

[0181] For example, through keyword "quarterly comparison", semantic vector and visual vector matching, recall the column chart of the promotion season in 2023 from the sample library, and screen out the quarterly sales comparison chart with unified color scheme and clear labels as the reference sample.

[0182] Step 412: Through a preset prompt information template, integrate the task description text, generation scheme and reference sample chart to obtain the prompt information for the generation task of the product operation chart.

[0183] For example, integrate the task description, grouped column chart scheme and reference sample into a prompt: "Generate a two-color grouped column chart, with the X-axis being quarters and the Y-axis being retention rate. Refer to the color scheme of the promotion chart (red / blue) and mark the difference values."

[0184] Step 414: Input the prompt information into the large language model, and based on the reference sample chart, use the generation scheme to perform the generation task of the product operation chart on the original data to obtain chart data.

[0185] For example, input the prompt information "Generate a two-color grouped column chart, with the X-axis being quarters and the Y-axis being retention rate. Refer to the color scheme of the promotion chart (red / blue) and mark the difference values." into the large language model, and based on the style and structure of the reference sample chart, combine with the original data to perform the generation task of the product operation chart to obtain a data chart containing the comparison of quarterly retention rates.

[0186] Step 416: Render the target chart based on the chart data.

[0187] For example, use the front-end rendering engine to render a two-color grouped column chart that clearly shows the comparison of quarterly retention rates between young users and senior users according to the generated chart data, and ensure that the significant difference values are correctly marked on the chart.

[0188] Step 418: If the rendering fails, obtain the error message. Through the large language model, based on the reference sample chart, use the error message as the correction feedback data, and use the generation scheme to re-perform the generation task of the product operation chart on the original data to obtain updated chart data, and re-render the target chart based on the updated chart data, and output the target chart to the user terminal.

[0189] For example, if an error of "unable to recognize color code" occurs during the rendering process, the system will capture this error message and feedback it to the large language model. The model adjusts the color settings according to this feedback, re-performs the generation task of the product operation chart, and ensures that the chart is re-rendered until successful after the colors are correctly applied.

[0190] Step 420: If the rendering is successful, through the vision-language model, based on the task description text and the generation scheme, analyze whether the target chart meets the task objective requirements of the image generation task and the scheme constraints of the generation scheme to obtain the analysis result.

[0191] For example, after rendering is completed, use a visual language model to check whether the generated chart accurately reflects the quarterly retention rate differences among different user groups, including whether elements such as labels, colors, and significance markers meet the expected design, to confirm that the chart meets the product operation analysis requirements.

[0192] Step 422: If the analysis result is in line, output the target chart to the user terminal.

[0193] For example, after inspection, it is confirmed that the chart not only shows the comparison of quarterly retention rates, but also provides in-depth insights through the annotation of significant difference values, fully meeting the requirements of the original task objective. Therefore, it is directly output to the user terminal to provide decision-making support.

[0194] Step 424: If the analysis result is not in line and includes adjustable parameters of the generation scheme, use a large language model to adjust the generation scheme based on the adjustable parameters, and based on the reference sample chart, re-execute the generation task of the product operation chart for the original data using the adjusted generation scheme to obtain updated chart data, re-render the target chart based on the updated chart data, and output the target chart to the user terminal.

[0195] For example, if it is found that the chart fails to highlight the key retention rate differences, the system will adjust the color contrast in the scheme or add more explanatory notes as adjustable parameters, and then re-execute the generation task of the product operation chart until a visualization chart that meets the requirements and is easy to understand is generated.

[0196] In the embodiments of this specification, the user requirement analysis method based on the chain of thought enables the system to deeply understand customer requirements and accurately convert them into specific drawing visual language schemes. This method guides the large language model to think step by step, ensuring that each step from analysis to generation scheme design is more accurate, thereby improving the accuracy of chart generation in subsequent steps. Secondly, introducing a high-quality chart sample database and multi-modal retrieval to enhance the generation link effectively solves the problem that existing large language models are prone to errors when dealing with uncommon or complex chart elements. By supporting multiple ways to match the sample charts in the database and providing sample codes for the large language model to refer to, it not only improves the success rate of chart generation but also enhances the aesthetics of the generated charts. The introduction of the post-generation self-reflection logic realizes the automatic error correction and optimization functions in the chart generation process. Through the multi-round feedback iteration mechanism, the system can automatically correct errors and re-execute the chart generation task after the initial rendering fails until a chart that meets the requirements is successfully output. This self-optimization ability further improves the quality of the finally generated chart, ensuring that it can not only meet the specific needs of customers but also reach a high professional standard, and is applicable to various product operation analysis scenarios. The entire process not only improves work efficiency but also ensures the professionalism and aesthetics of the data visualization results, providing decision-making support for product operation.

[0197] Corresponding to the above method embodiments, this specification also provides embodiments of a multimodal data processing platform. Figure 5 FIG. shows a schematic structural diagram of a multimodal data processing platform provided by an embodiment of this specification. As Figure 5 shown, the multimodal data processing platform 500 includes a front-end interface 502 and a response unit 504;

[0198] The front-end interface 502 is configured to receive a task description text and original data of a chart generation task sent by a user front-end.

[0199] The response unit 504 is configured to execute the steps of the above chart generation method.

[0200] In the embodiments of this specification, it is ensured that even for uncommon chart types, target charts with high accuracy and high applicability can be generated, effectively improving the diversity and quality of chart generation.

[0201] The above is a schematic solution of a multimodal data processing platform according to this embodiment. It should be noted that the technical solution of this multimodal data processing platform and the technical solution of the above chart generation method belong to the same concept. For the details not described in the technical solution of the multimodal data processing platform, reference can be made to the description of the technical solution of the above chart generation method.

[0202] Corresponding to the above method embodiments, this specification also provides embodiments of a chart generation device. Figure 6 FIG. shows a schematic structural diagram of a chart generation device provided by an embodiment of this specification. As Figure 6 shown, the device includes:

[0203] An acquisition module 602 configured to acquire a task description text and original data of a chart generation task;

[0204] An analysis module 604 configured to analyze the chart generation task based on the task description text and original data to obtain a generation scheme for the chart generation task;

[0205] A retrieval module 606 configured to retrieve a reference sample chart from a sample chart database based on the generation scheme;

[0206] A generation module 608 configured to execute the chart generation task on the original data based on the reference sample chart and the generation scheme through a multimodal generation model to generate a target chart.

[0207] Optionally, the parsing module 604 is further configured to perform data analysis on the original data to obtain data element information of the original data; and perform joint parsing on the chart generation task based on the task description text and the data element information to obtain a generation scheme for the chart generation task.

[0208] Optionally, the parsing module 604 is further configured to perform data statistics on the original data to obtain statistical information of the original data, extract a preset number of data from the original data as sampling data; and combine the statistical information and the sampling data to obtain data element information of the original data.

[0209] Optionally, the apparatus further includes: a vision module configured to obtain a reference picture of the chart generation task; and perform picture style analysis on the reference picture to obtain visual description information of the reference picture.

[0210] Optionally, the parsing module 604 is further configured to perform multimodal joint parsing on the chart generation task based on the task description text, the data element information, and the visual description information to obtain a generation scheme for the chart generation task.

[0211] Optionally, the vision module is further configured to perform picture style analysis on the reference picture through a vision-language model to obtain visual description information of the reference picture.

[0212] Optionally, the parsing module 604 is further configured to, through a scheme generation model, based on the task description text and the original data, use a chain of thought to parse the task objective of the chart generation task to obtain a task objective analysis result, and based on the task description text and the original data, under the guidance of the task objective analysis result, use a chain of thought to parse the task implementation path of the chart generation task to obtain a generation scheme design result, and generate a generation scheme for the chart generation task based on the task objective analysis result and the generation scheme design result.

[0213] Optionally, the generation scheme includes at least one of a target chart type, a reference picture of the chart generation task, and visual description information of the reference picture of the chart generation task; the retrieval module 606 is further configured to perform at least one of the following: perform keyword matching retrieval from the classification label field of the sample database based on the target chart type to obtain a reference sample chart, where the classification label field is a field in the sample database used to identify the chart type; perform vector matching retrieval from the preset semantic vectors of the sample database based on the semantic vector of the visual description information to obtain a reference sample chart, where the preset semantic vectors are semantic encoding vectors in the sample database used to identify text description information; perform vector matching retrieval from the preset visual vectors of the sample database based on the visual vector of the reference picture to obtain a reference sample chart, where the preset visual vectors are visual encoding vectors in the sample database used to identify visual description information.

[0214] Optionally, the retrieval module 606 is further configured to: retrieve a plurality of recalled sample charts from the sample chart database based on the generation scheme; sort the plurality of recalled sample charts through a pre-trained sorting model to obtain a sorting result of the plurality of recalled sample charts; and screen out a reference sample chart based on the sorting result of the plurality of recalled sample charts.

[0215] Optionally, the device further includes: a prompt integration module configured to integrate the task description text, the original data, the generation scheme, and the reference sample chart through a preset prompt information template to obtain prompt information for the chart generation task;

[0216] Correspondingly, the generation module 608 is further configured to: input the prompt information into a multimodal generation model, perform a chart generation task on the original data, and obtain chart data.

[0217] Optionally, the generation module 608 is further configured to: perform a chart generation task on the original data based on the reference sample chart through a multimodal generation model using the generation scheme to obtain chart data; and render a target chart based on the chart data.

[0218] Optionally, the device further includes: an inspection module configured to, if the rendering fails, obtain an error message; perform a chart generation task on the original data again through a multimodal generation model based on the reference sample chart with the error message as correction feedback data using the generation scheme to obtain updated chart data; and re-render the target chart based on the updated chart data.

[0219] Optionally, the device further includes: a reflection module configured to, if the rendering is successful, analyze whether the target chart meets the task objective requirements of the image generation task and the scheme constraints of the generation scheme based on the task description text and the generation scheme through a vision-language model to obtain an analysis result; if the analysis result is in line, output the target chart to the user terminal; if the analysis result is not in line and includes adjustable parameters of the generation scheme, adjust the generation scheme based on the adjustable parameters through a multimodal generation model, and perform a chart generation task on the original data again based on the reference sample chart using the adjusted generation scheme to obtain updated chart data, and re-render the target chart based on the updated chart data.

[0220] In the embodiments of the present specification, it is ensured that even for uncommon chart types, chart data with high accuracy and high applicability can be generated, and the rendering module renders to obtain the target chart, effectively improving the diversity and quality of chart generation.

[0221] The above is a schematic solution of a chart generation device according to this embodiment. It should be noted that the technical solution of this chart generation device and the technical solution of the above chart generation method belong to the same concept. For the details not described in detail in the technical solution of the chart generation device, reference can be made to the description of the technical solution of the above chart generation method.

[0222] Figure 7 The block diagram of a computing device provided by an embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 through a bus 730, and a database 750 is used to store data.

[0223] The computing device 700 further includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interfaces (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0224] In an embodiment of this specification, the above components of the computing device 700 and Figure 7 other components not shown in the figure may also be connected to each other, for example, through a bus. It should be understood that Figure 7 the shown block diagram of the computing device structure is only for illustrative purposes and is not a limitation on the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0225] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0226] Wherein, the processor 720 is configured to execute the following computer program / instructions, and when the computer program / instructions are executed by the processor, the steps of the above chart generation method are implemented.

[0227] The above is a schematic solution of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above chart generation method belong to the same concept. For the details not described in detail in the technical solution of the computing device, reference can be made to the description of the technical solution of the above chart generation method.

[0228] An embodiment of this specification also provides a computer-readable storage medium, which stores computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above chart generation method are implemented.

[0229] The above is a schematic solution of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above chart generation method belong to the same concept. For the details not described in detail in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above chart generation method.

[0230] An embodiment of this specification also provides a computer program product, including computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above chart generation method are implemented.

[0231] The above is a schematic solution of a computer program product according to this embodiment. It should be noted that the technical solution of the computer program product and the technical solution of the above chart generation method belong to the same concept. For the details not described in detail in the technical solution of the computer program product, reference can be made to the description of the technical solution of the above chart generation method.

[0232] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0233] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, removable hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM for short), random access memory (RAM for short), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0234] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential for the embodiments of this specification.

[0235] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0236] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The alternative embodiments do not elaborate on all the details and do not limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can understand and pass through this specification well. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for generating a chart, comprising: Obtaining a task description text and original data of a chart generation task; Based on the task description text and the original data, parsing the chart generation task to obtain a generation scheme for the chart generation task, wherein the generation scheme for the chart generation task is an intermediate instruction generated by parsing the task description text and the original data, and the generation scheme includes at least one of a reference picture of the chart generation task and visual description information of the reference picture of the chart generation task; Based on the generation scheme, retrieving a reference sample chart from a sample chart database; Through a multi-modal generation model, based on the reference sample chart, using the generation scheme to execute the chart generation task on the original data to generate a target chart; Wherein, the retrieving a reference sample chart from a sample chart database based on the generation scheme includes at least one of the following: Based on the semantic vector of the visual description information, performing vector matching retrieval from preset semantic vectors in the sample database to obtain a reference sample chart, wherein the preset semantic vector is a semantic coding vector for identifying text description information in the sample database; Based on the visual vector of the reference picture, performing vector matching retrieval from preset visual vectors in the sample database to obtain a reference sample chart, wherein the preset visual vector is a visual coding vector for identifying visual description information in the sample database.

2. The method according to claim 1, wherein the parsing the chart generation task based on the task description text and the original data to obtain the generation scheme for the chart generation task includes: Performing data analysis on the original data to obtain data element information of the original data; Based on the task description text and the data element information, jointly parsing the chart generation task to obtain the generation scheme for the chart generation task.

3. The method according to claim 2, wherein the performing data analysis on the original data to obtain data element information of the original data includes: Performing data statistics on the original data to obtain statistical information of the original data, and extracting a preset number of data from the original data as sampling data; Combining the statistical information and the sampling data to obtain data element information of the original data.

4. The method according to claim 2, before the jointly parsing the chart generation task based on the task description text and the data element information to obtain the generation scheme for the chart generation task, further comprising: Obtaining a reference picture of the chart generation task; Performing picture style analysis on the reference picture to obtain visual description information of the reference picture; The jointly parsing the chart generation task based on the task description text and the data element information to obtain the generation scheme for the chart generation task includes: Based on the task description text, the data element information and the visual description information, performing multi-modal joint parsing on the chart generation task to obtain the generation scheme for the chart generation task.

5. The method according to claim 4, wherein the analyzing the picture style of the reference picture to obtain the visual description information of the reference picture includes: Analyzing the picture style of the reference picture through a visual language model to obtain the visual description information of the reference picture.

6. The method according to any one of claims 1-5, wherein the parsing the chart generation task based on the task description text and the original data to obtain the generation scheme of the chart generation task includes: Through a scheme generation model, based on the task description text and the original data, using a chain of thought to parse the task objective of the chart generation task to obtain a task objective analysis result, and based on the task description text and the original data, under the guidance of the task objective analysis result, using a chain of thought to parse the task implementation path of the chart generation task to obtain a generation scheme design result, and based on the task objective analysis result and the generation scheme design result, generating the generation scheme of the chart generation task.

7. The method according to claim 1, wherein the retrieving a reference example chart from the example chart database based on the generation scheme includes: Retrieving a plurality of recalled example charts from the example chart database based on the generation scheme; Sorting the plurality of recalled example charts through a pre-trained sorting model to obtain a sorting result of the plurality of recalled example charts; Filtering out a reference example chart based on the sorting result of the plurality of recalled example charts.

8. The method according to claim 1, before the performing the chart generation task on the original data by the multi-modal generation model based on the reference example chart and using the generation scheme to obtain chart data, further includes: Integrating the task description text, the original data, the generation scheme and the reference example chart through a preset prompt information template to obtain the prompt information of the chart generation task; The performing the chart generation task on the original data by the multi-modal generation model based on the reference example chart and using the generation scheme to obtain chart data includes: Inputting the prompt information into the multi-modal generation model to perform the chart generation task on the original data to obtain chart data.

9. The method according to claim 1, wherein the performing the chart generation task on the original data by the multi-modal generation model based on the reference example chart and using the generation scheme to generate a target chart includes: Performing the chart generation task on the original data by the multi-modal generation model based on the reference example chart and using the generation scheme to obtain chart data; Rendering a target chart based on the chart data.

10. The method according to claim 9, after the rendering the target chart based on the chart data, further includes: If the rendering fails, obtaining an error message; Performing the chart generation task on the original data again by the multi-modal generation model based on the reference example chart, using the error message as correction feedback data and using the generation scheme to obtain updated chart data; Re-render the target chart based on the updated chart data.

11. The method according to claim 9, after rendering the target chart based on the chart data, further comprising: If the rendering is successful, analyze, through a visual language model, whether the target chart meets the task objective requirements of the chart generation task and the scheme constraints of the generation scheme based on the task description text and the generation scheme, to obtain an analysis result; If the analysis result is that it meets the requirements, output the target chart to the user terminal; If the analysis result is that it does not meet the requirements and includes adjustable parameters of the generation scheme, adjust the generation scheme based on the adjustable parameters through the multimodal generation model, and based on a reference example chart, re-execute the chart generation task on the original data using the adjusted generation scheme to obtain updated chart data, and re-render the target chart based on the updated chart data.

12. A multimodal data processing platform, comprising a front-end interface and a response unit; The front-end interface is used to receive the task description text and the original data of the chart generation task sent by the user front-end; The response unit is used to execute the steps of the chart generation method according to any one of claims 1 to 11.

13. A computing device, comprising: A memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium, which stores computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images