Chart generation method, multi-modal data platform, computing equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a chart generation method, a multi-modal data platform, computing equipment and a computer readable storage medium. The chart generation method comprises the steps of obtaining a task description text and original data of a chart generation task; based on the task description text and the original data, analyzing the chart generation task to obtain a generation scheme of the chart generation task; based on the generation scheme, retrieving a reference sample chart from a sample chart database; and executing a chart generation task on the original data by adopting the generation scheme based on the reference sample chart through the multi-modal generation model to generate a target chart. The target chart with high accuracy and high applicability can be generated even for uncommon chart types, and the diversity and quality of chart generation are effectively improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of artificial intelligence, and in particular to a chart generation method, a multimodal data platform, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence technology, automatic chart generation can effectively improve the efficiency of chart drawing through chart generation models, such as large language models or multimodal models dedicated to chart drawing.

[0003] At present, through the chart generation model, under the guidance of the task description text of the input chart generation task, the input raw data is charted, the chart data is output, the chart data is rendered on the front end, and the target chart is obtained.

[0004] However, due to the limitations of model training data and capabilities, the supported chart types are relatively few, and the chart generation model relies entirely on the knowledge that has been learned. When the training set of the chart generation model does not cover all possible chart styles and formats, it is easy to make errors in the generation of some uncommon chart types, resulting in insufficient accuracy and applicability of the generated results. Therefore, a more flexible and scalable chart generation method is urgently needed. Summary of the invention

[0005] In view of this, an embodiment of the present specification provides a method for generating a chart. One or more embodiments of the present specification also relate to a multimodal data processing platform, a chart generating apparatus, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0006] One aspect of an embodiment of the present specification provides a chart generation method, including: obtaining a task description text and raw data of a chart generation task; parsing the chart generation task based on the task description text and the raw data to obtain a generation plan for the chart generation task; retrieving a reference sample chart from a sample chart database based on the generation plan; and using a multimodal generation model and the generation plan to perform the chart generation task on the raw data to generate a target chart based on the reference sample chart.

[0007] In one embodiment of the present specification, a specific generation scheme is generated by combining the analysis of the task description text and the original data, and a reference sample chart is retrieved from the sample chart database to supplement the knowledge limitations of the multimodal generation model, get rid of the dependence on pre-trained model knowledge, and also pass through the actual existing diversified chart examples, thereby enriching the style and format of chart generation, and solving the accuracy and applicability problems of the generation results caused by insufficient coverage of the model training set. The generation scheme guides the multimodal generation model to generate charts according to the reference samples, ensuring that even for uncommon chart types, high-accuracy and high-applicability target charts can be generated, effectively improving the diversity and quality of chart generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flow chart of a chart generation method provided by one embodiment of this specification; Figure 2 is a schematic diagram of a target chart in a chart generation method provided by an embodiment of this specification; Figure 3 It is a flowchart of a method for generating a chart provided by an embodiment of this specification; Figure 4 It is a process flow chart of a method for generating a chart applied to a product operation scenario provided by an embodiment of this specification; Figure 5 is a structural diagram of a multimodal data processing platform provided by an embodiment of this specification; Figure 6 It is a structural schematic diagram of a chart generating device provided by an embodiment of this specification; Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0009] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0010] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0011] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0012] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0013] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

[0014] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0015] First, the terms involved in one or more embodiments of this specification are explained.

[0016] Retrieval-augmented Generation (RAG): A method that combines information retrieval and generation techniques. Specifically, RAG first retrieves document fragments related to the input query from a large-scale knowledge database, and then inputs these contents into the generation model together with the query to assist in generating more accurate and information-rich content.

[0017] Large Language Model (LLM): A large language model refers to a language processing artificial intelligence model with a large number of parameters. It can understand and generate high-quality human language text. It achieves understanding of language structure and semantics by learning massive Internet text data.

[0018] Visual Language Model (VLM): A visual language model is a deep learning model used to process the relationship between images and text. It can understand and generate text describing the content of an image, or generate corresponding images based on the text description.

[0019] Chart Generation: Chart generation is the reasoning task of converting data into visual charts (such as bar charts, line charts, pie charts, etc.) to more intuitively analyze and understand data trends and relationships. It is an important part of data analysis and report production.

[0020] Chain of Thought (CoT): Chain of Thought is a reasoning method that gradually breaks down complex problems into a series of simple intermediate steps in order to solve problems or derive conclusions more systematically. This method helps to improve the transparency and accuracy of complex problem handling.

[0021] Prompt: Prompt refers to the prompt or input data used to guide the model to generate responses in artificial intelligence dialogues. Carefully designed prompts can effectively guide the model to output expected results.

[0022] JavaScript Object Notation (JSON): A lightweight data exchange format that uses human-readable text to represent structured data, consisting of key-value pairs and nested structures.

[0023] Vega-Lite: A high-level visualization specification language based on JSON syntax, focusing on quickly generating interactive charts through concise declarative syntax.

[0024] Comma-Separated Values ​​(CSV): A common file format that stores tabular data in plain text, widely used for data exchange and storage.

[0025] Portable Network Graphics (PNG): A bitmap image format that uses a lossless compression algorithm and supports transparency channels (Alpha channels) and multiple color depths (such as 24-bit true color).

[0026] Scalable Vector Graphics (SVG): A two-dimensional vector graphics format based on XML syntax. It defines geometric shapes such as points, lines, and curves through mathematical formulas and can be scaled without distortion.

[0027] HTML5 Canvas: An element in the HTML5 standard that dynamically draws bitmaps through JavaScript scripts, providing a pixel-level graphics operation interface. It is based on the immediate rendering mode and is suitable for generating high-performance interactive content such as games, dynamic charts, and image editing tools.

[0028] ECharts: A JavaScript-based data visualization library that provides a variety of chart types (such as line charts, heat maps, 3D globes) and interactive functions (such as dragging, zooming, and data filtering).

[0029] Transformer: A deep learning model architecture based on self-attention mechanism.

[0030] At present, chart generation can convert data into visual charts (such as bar charts, line charts, pie charts, etc.). These graphics can effectively show the relationship and trend between data, helping users to understand complex information more intuitively. In the information age, the importance of data visualization has become increasingly prominent. It not only improves the efficiency of data interpretation, but also provides an intuitive basis for decision-making. In the traditional chart drawing process, users generally have already thought about the corresponding drawing plan, and then use tools such as Excel and Tableau to specify the chart type, confirm which fields in the data are mapped to the x-axis, y-axis or other visual channels, and then manually adjust the chart layout and the style and color of each chart element (such as title, legend, background grid, other marks), etc., until the user's aesthetics and intentions are met. Drawing a picture requires a high time cost.

[0031] Through the chart generation model, under the guidance of the task description text of the input chart generation task, the input raw data is charted, the chart data is output, and the chart data is rendered on the front end to obtain the target chart. However, due to the limitations of model training data and capabilities, the supported chart types are relatively few, and it completely relies on the knowledge that the chart generation model has learned. When the training set of the chart generation model does not cover all possible chart styles and formats, it is easy to make mistakes when generating some uncommon chart types, resulting in insufficient accuracy and applicability of the generated results. Therefore, a more flexible and scalable chart generation method is urgently needed.

[0032] In order to solve the above problems, in one or more embodiments of this specification, an automatic chart generation method based on multimodal retrieval enhanced generation is proposed, which combines large models and multimodal retrieval enhanced generation technology to analyze task target requirements and data characteristics, thereby achieving automatic and high-quality chart generation. This specification also involves a multimodal data processing platform, a chart generation device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0033] See also Figure 1 , Figure 1 A flowchart of a chart generation method provided by an embodiment of the present specification is shown, comprising the following specific steps: Step 102: Obtain the task description text and original data of the chart generation task.

[0034] The embodiments of this specification are applied to applications, websites or system platforms with multimodal data generation capabilities, and are suitable for scenarios such as data analysis, work reports, and academic research.

[0035] The chart generation task is an inference task that converts raw data into visual charts according to the task goal requirements. For example, a chart generation task for the frequency of events in the past year applied to product operation and maintenance. The task description text of the chart generation task is the chart generation requirements described in natural language, including chart types, data mapping rules, style constraints, etc., which are used to describe the expectations for the generated target chart, for example, "Help me draw the relationship between xx variables and xx variables." The raw data of the chart generation task is structured or semi-structured input data, such as CSV, JSON, database tables, etc. The raw data can come from a data table file in a local database, or it can be obtained from a remote database, without limitation here.

[0036] For example, on a large language model platform with multimodal data generation capabilities, the user front-end inputs the task description text of the chart generation task: "Use a line chart to show the number of events each month", and imports the original data of the data table file CSV from the local database, as shown in Table 1:

[0037] Table 1 By obtaining the task description text and original data of the chart generation task, support for the task target requirement description and task object data is provided for the subsequent parsing of the chart generation task and generation of chart data.

[0038] Step 104: Based on the task description text and the original data, the chart generation task is parsed to obtain a generation plan for the chart generation task.

[0039] The generation scheme of the chart generation task is an intermediate instruction generated by parsing the task description and raw data, including chart type, data field mapping logic, style configuration, etc. The generation scheme details how to process raw data, select appropriate chart types, configure visual elements, etc. For example, use a line chart to represent time series data, and recommend color and title styles.

[0040] Based on the task description text and the original data, the chart generation task is parsed to obtain a generation plan for the chart generation task. One optional way is: based on the task description text and the original data, the chart generation task is analyzed for intent and requirements, and a generation plan for the chart generation task is obtained, wherein the intent analysis and requirements analysis include but are not limited to: at least one of data analysis, visualization requirements analysis and image style analysis. Another optional way is: based on the task description text and the original data, the chart generation task is analyzed for intent and a generation plan for the chart generation task is obtained. Another optional way is: based on the task description text and the original data, the requirements analysis of the chart generation task is performed to obtain a generation plan for the chart generation task, which is not limited here.

[0041] For example, based on the task description text "Use a line chart to show the number of events each month" and the original data in Table 1 above, a visualization requirement analysis and data analysis are performed on the chart generation task to obtain a generation plan for the chart generation task: Detailed design specifications: - Chart type and layout: Use a line chart, and display it as a single chart. - Color scheme and style: Use blue as the line color, white background, and gray border. - Coordinate axis configuration and scale: X-axis is month (1-12), Y-axis is the number of events, and the scale is automatically adjusted according to the data range. - Labels, titles, and annotations: The X-axis label is "Month", the Y-axis label is "Number of Events", and the chart title is "Monthly Event Number Trend". - Legend position and format: No legend is required. - Gridlines and other reference elements: Thin gray gridlines are enabled for both the X-axis and the Y-axis.

[0042] Based on the task description text and original data, the chart generation task is parsed to obtain the generation plan of the chart generation task, which provides a retrieval basis for subsequent retrieval and provides solution support for the subsequent generation of chart data.

[0043] Step 106: Based on the generation plan, retrieve a reference sample graph from the sample graph database.

[0044] The sample chart database is a database that stores sample chart templates, including chart configurations (such as Vega-Lite / JSON), visual styles, applicable scenario labels, etc. The reference sample chart is a chart template that matches the generation solution retrieved from the sample chart database, including chart configurations (such as Vega-Lite specifications), visual styles (color matching, layout), applicable scenario labels, etc.

[0045] Based on the generation scheme, a reference sample chart is retrieved from the sample chart database. One optional method is: based on the generation scheme, a reference sample chart is retrieved from the sample chart database multimodally. Another optional method is: based on the generation scheme, a reference sample chart is heuristically retrieved from the sample chart database. Another optional method is: based on the generation scheme, a reference sample chart is retrieved from the sample chart database cross-modally, which is not limited here.

[0046] Exemplarily, a hybrid retrieval scheme of multi-way recall is adopted, and based on the generation scheme, reference sample graphs that meet the preset retrieval conditions are retrieved from the sample graph database.

[0047] Based on the generation scheme, reference sample charts are retrieved from the sample chart database to supplement the knowledge limitations of the multimodal generation model, get rid of the dependence on pre-trained model knowledge, and use actual diverse chart instances to enrich the style and format of chart generation, solving the problems of accuracy and applicability of generation results caused by insufficient coverage of the model training set.

[0048] Step 108: Using the multimodal generation model, based on the reference sample chart, a generation scheme is used to perform a chart generation task on the original data to generate a target chart.

[0049] The multimodal generation model is a deep learning model with multimodal data generation capabilities. It can process text, structured data, and visual style information at the same time, including but not limited to large language models or deep learning models based on the Transformer architecture. For example, the multimodal generation model supports end-to-end generation of text-to-chart configurations (such as converting "line chart" instructions into Vega-Lite code), and can also parse style features in reference examples for transfer learning. The target chart is the visualization result of the rendered generated chart data. The target chart can be a static image (PNG / SVG) or an interactive component (HTML5 Canvas).

[0050] Through the multimodal generation model, based on the reference sample chart, a generation scheme is used to perform the chart generation task on the original data to generate a target chart. One optional way is: through the multimodal generation model, based on the reference sample chart, a generation scheme is used to perform the chart generation task on the original data to directly generate the target chart. Another optional way is: through the multimodal generation model, based on the reference sample chart, a generation scheme is used to perform the chart generation task on the original data to generate chart data, and a target chart is generated based on the chart data. There is no limitation here.

[0051] Exemplarily, through a large language model, the reference sample chart is used as the supplementary knowledge for retrieval enhancement generation, and the above generation scheme is used to perform the chart generation task on the original data in Table 1 to obtain chart data: import pandas as pd# Import the pandas library for data processing.

[0052] import seaborn as sns# Import the seaborn library for advanced data visualization.

[0053] import matplotlib.pyplot as plt# Import the matplotlib.pyplot module for drawing charts.

[0054] plt.rcParams['font.sans-serif'] = ['SimHei']# Use SimHei font to support Chinese display.

[0055] plt.rcParams['axes.unicode_minus'] = False# Solve the problem of negative sign display.

[0056] def preprocess(data): # Make sure the Month column is of string type so it can be displayed in the chart.

[0057] data['Month'] = data['Month'].astype(str) # Convert the data type of the 'Month' column to a string.

[0058] return data# Returns the processed data.

[0059] def plot(data): data = preprocess(data)# Call the preprocessing function to process the data.

[0060] sns.set_style("whitegrid")# Set the background style to white grid lines.

[0061] plt.figure(figsize=(10, 6))# Set the chart size to 10 inches wide and 6 inches high.

[0062] ax = sns.lineplot(data=data, x='Month', y='Event_Count', marker='o', markersize=8, linewidth=2, color='#1f77b4')# Use seaborn to draw a line chart, specify parameters such as data, X-axis, Y-axis, and markers.

[0063] plt.title('Monthly event number change trend', pad=20, fontsize=16)# Set the chart title, and set the spacing and font size.

[0064] plt.xlabel('Month', fontsize=14)# Set the X-axis label text and font size.

[0065] plt.ylabel('Number of events', fontsize=14)# Set the Y-axis label text and font size.

[0066] plt.xticks(rotation=45)# Rotate the X-axis label 45 degrees to prevent label overlap.

[0067] for x, y in zip(data['Month'], data['Event_Count']): plt.annotate(f'{int(y)}', # Add annotations to each data point to show the number of events.

[0068] (x, y), # The location of the annotation (data point location).

[0069] textcoords="offset points", #Specify the unit of the annotation position offset relative to the point.

[0070] xytext=(0, 10), # The offset of the annotation relative to the data point.

[0071] ha='center', # Set the horizontal alignment to center.

[0072] fontsize=12)# Set comment font size.

[0073] plt.tight_layout()# Automatically adjust the sub-image parameters to fill the entire image area.

[0074] plt.savefig('plot.png', dpi=300, bbox_inches='tight') # Save the chart as a PNG file with a resolution of 300 dpi, making sure all content is within the image.

[0075] plt.close()# Close the current graphic and release memory.

[0076] Convert the chart data into the ECharts standard configuration, and call the ECharts front-end rendering engine in the HTML page to dynamically render an interactive line chart. The line chart supports displaying values ​​when the mouse hovers over it, and triggering events by clicking on data points (which can extend the drill-down analysis logic). Render the visual features such as blue lines, gray grid lines, and rotated labels defined by the generation scheme. The chart container size automatically adjusts with the browser window, and automatically compresses the spacing to prevent element overlap when displayed on a mobile device. The above interactive line chart is suitable for dashboard scenarios that need to be displayed online. Users can explore data details by zooming and dragging. Specific target charts are as follows: Figure 2 As shown, Figure 2 A schematic diagram of a target chart in a chart generation method provided in one embodiment of this specification is shown: The horizontal axis is the month, and the vertical axis is the number of events. The various data points are connected by a line graph to represent the changing trend of the number of events per month.

[0077] In the embodiments of this specification, by combining the analysis of the task description text and the original data, a specific generation scheme is generated, and a reference sample chart is retrieved from the sample chart database, so as to supplement the knowledge limitations of the multimodal generation model, get rid of the dependence on pre-trained model knowledge, and also pass through the actual existing diversified chart examples, thereby enriching the style and format of chart generation, and solving the accuracy and applicability problems of the generation results caused by insufficient coverage of the model training set. The generation scheme guides the multimodal generation model to generate charts according to reference samples, ensuring that even for uncommon chart types, high-accuracy and high-applicability target charts can be generated, effectively improving the diversity and quality of chart generation.

[0078] In an optional embodiment of the present specification, step 104 includes the following specific steps: performing data analysis on the original data to obtain data metadata information of the original data; based on the task description text and the data metadata information, jointly parsing the chart generation task to obtain a generation plan for the chart generation task.

[0079] The data meta information of raw data is a set of statistical features and structural features extracted from the raw data. The data meta information of raw data includes but is not limited to: field name, data type (numeric, categorical, etc.), missing value situation, distribution characteristics (such as mean, median, mode, standard deviation, etc.) and correlation coefficients between fields (such as Pearson correlation coefficient).

[0080] An optional method for performing data analysis on the original data to obtain data meta information of the original data is to perform data analysis on the original data in terms of statistical feature dimensions and sampling feature dimensions to obtain data meta information of the original data. Another optional method is to perform data analysis on the original data in terms of statistical feature dimensions to obtain data meta information of the original data. Another optional method is to perform data analysis on the original data in terms of sampling feature dimensions to obtain data meta information of the original data, which is not limited here.

[0081] Exemplarily, the original data in Table 1 above is subjected to data analysis in terms of statistical feature dimensions and sampling feature dimensions to obtain data meta information of the original data: "Field type": {"Month": {"Type": "Discrete integer", "Range": [1, 12], "Number of enumerated values": 12}, "Number of events": {"Type": "Continuous numeric", "Range": [17, 40], "Standard deviation": 8.2}}; "Data distribution": {"Month": {"Periodicity": "Monthly cycle (12 months)", "Interval unit": 1}, "Number of events": {"Range": 23, "Mean": 28.1}}; "Data quality": {"Missing value ratio": 0%, "Outlier detection": "None"}}.

[0082] Through the large language model, the chart generation task is jointly parsed based on the task description text and data metadata: 1. Task analysis: - What is the industry background and target audience? - What insights need to be conveyed? - What are the common visualization methods in this field? 2. Data metadata analysis: - What are the key features of the provided data? - What relationships or patterns need to be highlighted? 3. Visualization generation plan, based on data characteristics and project background: - Which chart types are most effective? - What alternatives have been considered and why were they rejected? - If necessary, how should multiple elements be composed? Obtain the generation plan for the above chart generation task.

[0083] In the embodiments of this specification, by performing data analysis on the data metadata of the original data, the deep data features of the original data required by the task objectives can be more accurately captured, and then a more reasonable and effective chart generation plan can be formulated, which helps to improve the accuracy and applicability of the generated charts.

[0084] In an optional embodiment of the present specification, data analysis is performed on the original data to obtain data metadata of the original data, including the following specific steps: performing data statistics on the original data to obtain statistical information of the original data, and extracting a preset number of data from the original data as sampling data; merging the statistical information and the sampling data to obtain data metadata of the original data.

[0085] The statistical information of the original data is a set of quantitative features extracted from the original data, including but not limited to: the number, mean, variance, quantile of each numerical column, the number of non-numerical columns after deduplication, high-frequency values, etc. Sampled data is a typical data entry extracted from the original data according to preset rules, which is used to assist the model in understanding the data structure (such as the mapping relationship between fields). The extraction methods include but are not limited to random sampling, stratified sampling, and head-tail sampling. For example, N data entries are extracted from a data table as sampled data.

[0086] Exemplarily, data statistics are performed on the original data in Table 1 to obtain statistical information of the original data, and N sampled data are extracted from the original data. The statistical information and the sampled data are combined to obtain data metadata of the original data: Month: Total number: 12; Mean: 6.5; Standard deviation: 3.60555; Minimum: 1; First quartile: 3.75; Median: 6.5; Third quartile: 9.25; Maximum: 12. Number of events: Total number: 12; Mean: 27.1667; Standard deviation: 7.45695; Minimum: 17; First quartile: 21.5; Median: 26.5; Third quartile: 32.75; Maximum: 40.

[0087] In the embodiments of this specification, data metadata is merged through data statistics and sampling, which can not only comprehensively capture the distribution characteristics and potential patterns of the data, but also effectively assist the model in understanding the data structure and the mapping relationship between fields, which not only improves the accuracy and efficiency of the chart generation task, but also makes the generated visualization results closer to the task goal requirements.

[0088] In an optional embodiment of the present specification, before jointly parsing the chart generation task based on the task description text and data meta information to obtain a generation solution for the chart generation task, the following specific steps are also included: obtaining a reference image for the chart generation task; performing image style analysis on the reference image to obtain visual description information of the reference image; jointly parsing the chart generation task based on the task description text and data meta information to obtain a generation solution for the chart generation task, including the following specific steps: performing multimodal joint parsing of the chart generation task based on the task description text, data meta information and visual description information to obtain a generation solution for the chart generation task.

[0089] The reference images for the chart generation task are visual images used for style transfer in the image generation task, such as example images actively uploaded by users (such as historical charts of similar project scenarios provided by users), or candidate template images recommended by the system (highly relevant samples automatically matched according to the task description). The visual description information of the reference image is the description information of the visual features extracted from the reference image, including but not limited to: chart type, color scheme and style, axis configuration, labels, titles and annotations, legend position, etc.

[0090] Perform image style analysis on the reference image to obtain visual description information of the reference image. One optional method is: perform image style analysis on the reference image through an image parsing tool to obtain visual description information of the reference image. Another optional method is: perform image style analysis on the reference image through a deep learning model to obtain visual description information of the reference image, which is not limited here.

[0091] Based on the task description text, data meta information and visual description information, a multimodal joint analysis is performed on the chart generation task to obtain a generation plan for the chart generation task. An optional method is: based on the task description text, data meta information and visual description information, an intent analysis and a requirement analysis are performed on the chart generation task to obtain a generation plan for the chart generation task, wherein the intent analysis and the requirement analysis include but are not limited to: at least one of data analysis, visualization requirement analysis and image style analysis.

[0092] For example, the user uploads a picture of a reference chart for a chart generation task on the front end, performs a picture style analysis on the reference picture, and obtains visual description information of the reference picture: Chart type: line chart. Primary color system (such as HEX color value #2E75B6), background transparency, and aspect ratio of the chart container. Coordinate axis: scale line direction (inward / outward), axis thickness, and label rotation angle. Data marking: line width (2px), mark point shape (circular / square), data label font size (12pt). Text layout: title font (Microsoft YaHei), legend position (upper right corner), annotation alignment (center alignment). Hover highlight effect (such as enlarging the data point by 1.2 times), animation transition duration (300ms).

[0093] Based on the task description text "Use a line chart to show the number of events each month", the original data and visual description information in Table 1 above, visualization requirements analysis, data analysis and image style analysis are performed on the chart generation task respectively to obtain a generation plan for the above chart generation task.

[0094] In the embodiments of this specification, by introducing reference images of visual modalities as references for image generation tasks and combining task description text and data metadata for multimodal joint analysis, not only the demand understanding dimension of chart generation is enriched, but also the visual accuracy and style consistency of the generated charts are significantly improved. The generated chart data can not only accurately reflect data characteristics and user intentions, but also match specific visual style requirements, thereby enhancing the professionalism and aesthetics of the target chart and better meeting the diverse needs of users.

[0095] In an optional embodiment of the present specification, performing image style analysis on a reference image to obtain visual description information of the reference image includes the following specific steps: performing image style analysis on the reference image through a visual language model to obtain visual description information of the reference image.

[0096] The visual language model is a deep learning model used to process the relationship between images and text. The visual language model can understand and generate text that describes the content of an image, or generate a corresponding image based on the text description.

[0097] In the embodiments of this specification, by analyzing the reference image through the visual language model, its visual description information can be accurately extracted, which not only improves the accuracy of style transfer, but also further enhances the professionalism and aesthetics of the target chart, and better meets the diverse needs of users.

[0098] In an optional embodiment of the present specification, step 104 includes the following specific steps: through a solution generation model, based on the task description text and the original data, a thinking chain is used to parse the task objective of the chart generation task to obtain a task objective analysis result, and based on the task description text and the original data, under the guidance of the task objective analysis result, a thinking chain is used to parse the task implementation path of the chart generation task to obtain a generation solution design result, and based on the task objective analysis result and the generation solution design result, a generation solution for the chart generation task is generated.

[0099] The solution generation model is a deep learning model with solution generation capabilities that can process text data, including but not limited to large language models or deep learning models based on the Transformer architecture. For example, the solution generation model supports end-to-end generation of text to text. The thinking chain is a logical framework for the multimodal generation model to perform complex reasoning tasks. By explicitly decomposing the correlation between task goals and implementation paths, a step-by-step reasoning chain is formed. The task goal analysis result is a natural language description of the essence of the task goal requirements. The task implementation path of the chart generation task is a set of execution strategies based on the task goal. The result of the generation solution design is a structured output after multimodal reasoning.

[0100] For example, through a large language model, based on the task description text and raw data, a chain of thought (task analysis: - What is the industry background and target audience? - What insights need to be conveyed? - What are the common visualization methods in this field? Data meta-information analysis: - What are the key features of the provided data? - What relationships or patterns need to be highlighted?) is used to analyze the task objectives of the chart generation task, and the task objective analysis results are obtained: 1. Industry background positioning: The identification scenario label is the operation and maintenance monitoring field, and the target audience is the IT operation and maintenance team and department managers; the core demand needs to reflect the cyclical fluctuation law of the number of events and assist in resource scheduling decisions. 2. Key features of the data: The number of events peaked in September (40 times) and fell back to a low point in December (22 times). 3. Which relationships or patterns are highlighted: Frequent events in the third quarter (may be related to increased server load); the number of events at the end of the year dropped significantly (or related to the project off-season). 4. Adaptation of the visualization generation solution: Applicable chart type priority: line chart (emphasis on time trend) > bar chart (suitable for discrete comparison) > heat map (requires multi-dimensional data).

[0101] Based on the task description text and raw data, under the guidance of the task goal analysis results, the thinking chain (visual generation scheme, according to data characteristics and project background: - Which chart types are most effective? - What alternatives have been considered and why were they rejected? - If necessary, how should multiple elements be composed?) is used to parse the task implementation path of the chart generation task and obtain the design results of the generation scheme: 1. Data mapping rules: X-axis: bind the "month" field and set it as a discrete classification axis (to avoid scale errors caused by automatic completion of the year); Y-axis: bind the "number of events" field and use a linear scale (original value display, no aggregation calculation required). 2. Visual enhancement strategy: Peak annotation: add a red frame (HEX #FF4500) to the September data point and mark it as "annual peak"; trend line overlay: add a gray dotted trend line (quadratic polynomial fitting) to show the long-term change direction. 3. Interactive logic design: Hover feedback: display the event classification proportion of the month (need to associate a sub-dataset); drill-down function: double-click the peak month to jump to the event distribution chart of the day.

[0102] Based on the task objective analysis results and the generation plan design results, a generation plan for the above-mentioned chart generation task is generated.

[0103] In the embodiments of this specification, by adopting the thinking chain to conduct detailed analysis of the task objectives and implementation paths, not only the understanding of the requirements of the chart generation task is refined, but also the accuracy and practicality of the generation plan are significantly improved, so that the generated charts can accurately reflect the key characteristics of the data and the project background, and enhance the effectiveness of information communication and the professionalism of visual expression.

[0104] In an optional embodiment of the present specification, the generation scheme includes at least one of a target chart type, a reference picture of the chart generation task, and visual description information of the reference picture of the chart generation task; step 106 includes at least one of the following: Based on the target chart type, a keyword matching search is performed from the classification label field of the sample database to obtain a reference sample chart, wherein the classification label field is a field in the sample database used to identify the chart type; based on the semantic vector of the visual description information, a vector matching search is performed from the preset semantic vector of the sample database to obtain a reference sample chart, wherein the preset semantic vector is a semantic coding vector in the sample database used to identify text description information; based on the visual vector of the reference image, a vector matching search is performed from the preset visual vector of the sample database to obtain a reference sample chart, wherein the preset visual vector is a visual coding vector in the sample database used to identify the visual description information.

[0105] The sample database contains data tables of several high-quality chart generation solutions, which are stored in the sample database, a multimodal database. Each data in the data table mainly contains information such as chart categories, subcategories, visualization purposes, visual descriptions, applicable scenarios, data features, sample data and codes, sample image vectors, etc. The sample data in the sample database can be generated using LLM and manually selected and verified. Optionally, the sample database is open and extensible. To meet the diverse chart drawing needs of different industries, you only need to add the corresponding sample chart information to the sample database to generate charts that meet the target chart style, so as to support more types and styles of chart generation tasks.

[0106] The classification label field is a field used to identify the chart type in the sample database. It can be an exact match keyword or a fuzzy extended label set. The preset semantic vector is a semantic encoding vector used to identify text description information in the sample database. Different text descriptions can be compared by calculating the similarity between vectors. The preset visual vector is a visual encoding vector used to identify visual description information in the sample database. Different visual descriptions can be compared by calculating the similarity between vectors.

[0107] The target chart type is a specific chart type determined according to the task objective requirements, such as a bar chart, a pie chart, a line chart, etc. The semantic vector of the visual description information is the semantic encoding vector of the visual description information. The visual vector of the reference image is the visual encoding vector of the reference image.

[0108] Exemplarily, the chart_type field in the generation scheme is parsed: enhanced_line_chart, the keyword "line chart - enhanced version" of the standardized classification label is extracted, the keyword is accurately matched and fuzzy expanded in the classification label field of the sample database, and the reference sample chart is recalled. Exemplarily, the visual description text "blue line, gray grid line, peak red frame annotation" in the generation scheme is encoded as a semantic vector, the cosine similarity between the vector and the semantic vector of the visual description field in the sample database is calculated, the Top-K similar results are retained as the recall results, and the reference sample chart is recalled. Exemplarily, the user uploads a "dark blue gradient line chart with hover prompt" as a reference image, encodes the reference image as a visual vector, calculates the cosine similarity between the vector and the preset visual vector of the reference sample chart in the sample database, retains the Top-K similar results as the recall results, and recalls the reference sample chart.

[0109] In the embodiments of this specification, a multimodal retrieval method is used to accurately match or fuzzily expand the target chart type, semantic vector and visual vector, thereby effectively improving the recall accuracy and diversity of the reference sample charts. This not only enhances the flexibility and customization level of chart generation, but also ensures that the generated target chart meets both data display requirements and the user's visual expectations.

[0110] In an optional embodiment of the present specification, based on the generation scheme, a reference sample chart is retrieved from a sample chart database, including the following specific steps: based on the generation scheme, multiple recall sample charts are retrieved from the sample chart database to obtain sorting results of the multiple recall sample charts; the multiple recall sample charts are sorted by a pre-trained sorting model; and based on the sorting results of the multiple recall sample charts, a reference sample chart is screened out.

[0111] The recalled sample charts are a set of chart templates that are preliminarily screened from the sample chart database in the process of multimodal retrieval based on the generation scheme and have a high degree of match with the task objective requirements. The recalled sample charts are a candidate set retrieved from a large number of sample charts by methods such as keyword matching, semantic vector matching or visual vector matching. They are preliminary result sets that are close to the task objective requirements of the chart generation task, but may not have been accurately sorted and optimized. The pre-trained ranking model is a machine learning model that is pre-trained with a large number of samples and can sort a given set of items (such as recalled sample charts) according to specific criteria or goals. The model is usually based on a deep learning architecture such as Transformer or RNN. The ranking results of multiple recalled sample charts are an ordered list obtained by sorting multiple recalled sample charts by the pre-trained ranking model, which contains the results of all recalled sample charts arranged from high to low according to factors such as their relevance, quality and applicability to the chart generation task.

[0112] For example, a pre-trained sorting model (such as a deep neural network based on multimodal contrastive learning) is used to finely sort hundreds of sample charts for multi-way recall. The sorting model evaluates the matching degree through the following dimensions: 1. Functional adaptation: compatibility between chart type and data features (such as whether time series data is suitable for line charts). 2. Style consistency: similarity of color scheme (such as the color difference between the main color and the reference image) and label layout (such as the rotation angle of the axis label). 3. Interaction logic fit: whether the interaction requirements defined in the generation plan are supported (such as hover prompts and drill-down analysis).

[0113] After the sorting model outputs the matching score of each recalled sample chart, it is sorted in descending order by score and the Top-K ones are taken as reference sample charts.

[0114] In the embodiments of this specification, the search results are carefully sorted to improve the relevance, quality and user satisfaction of the reference sample charts, ensure the matching degree and visual consistency of the generated solutions, optimize the accuracy and professionalism of chart generation, and effectively improve the level of chart customization and user experience.

[0115] In an optional embodiment of the present specification, before step 108, the following specific steps are also included: integrating the task description text, original data, generation scheme and reference sample chart through a preset prompt information template to obtain prompt information of the chart generation task; Correspondingly, step 108 includes the following specific steps: inputting the prompt information into the multimodal generation model, performing the chart generation task on the original data, and obtaining chart data.

[0116] The preset prompt information template is a pre-set structured text framework, which is used to integrate the input information of the chart generation task (task description text, generation plan, reference sample chart, etc.) into a unified format of prompt information that can be parsed by the multimodal generation model. The prompt information of the chart generation task is the multimodal input data integrated by the preset template, which contains contextual information such as task goal requirements, generation plan and reference examples. As the input of the multimodal generation model, it guides the multimodal generation model on how to convert the raw data into visual chart data that meets the user's expectations.

[0117] Through the preset prompt information template, the task description text, original data, generation plan and reference sample chart are integrated to obtain the prompt information of the chart generation task. Some of the contents have been given in the above examples and will not be repeated here: For example, the prompt message is: "You are a data visualization expert. Now there is a Python visualization code generation task. You need to first read the reference sample chart code, and then combine the original data and the task description text, and adopt the recommended visualization generation solution to implement the visualization code for the original data. ## Reference sample chart starts; target chart type: basic line chart; sample data format: ...; reference sample chart code: ...; ## Reference sample chart ends. Below, the characteristics of user data and task target requirements are given. ## User data starts: Task description text: Use a line chart to show the number of events each month; recommended visualization generation solution: ...; original data: ...; metadata information of the original data: ...; # User data ends. Now, please refer to the sample and combine the actual situation of the user data to generate visualization chart code data that meets the requirements." The above prompt information is input into the large language model, and based on the reference sample chart, a generation scheme is used to perform the chart generation task on the original data to obtain the above chart data.

[0118] In the embodiments of this specification, through the preset prompt information templates and the dynamically generated prompt information, the system can deeply integrate user needs, analysis solutions and reference examples in the external knowledge base to form high-quality input for the multimodal generation model, which significantly improves the accuracy, style richness and cross-scenario adaptability of chart generation, especially when dealing with complex interaction needs or niche chart types.

[0119] In an optional embodiment of the present specification, step 108 includes the following specific steps: using a multimodal generation model, based on a reference sample chart, using a generation scheme to perform a chart generation task on the original data to obtain chart data; and rendering a target chart based on the chart data.

[0120] The chart data is the chart code data generated based on the generation scheme and the reference sample template, including complete visualization parameters, such as: 1. Structural configuration: define the chart type (such as line chart), data mapping relationship (such as x-axis binding month field), coordinate axis range and scale in Vega-Lite specification or JSON format. 2. Visual parameters: including color scheme (such as HEX color value #1f77b4), line width, marker style, font size and other details. 3. Interaction logic: If the reference sample contains dynamic interaction (such as hover prompt), its event response configuration is inherited.

[0121] Through the multimodal generation model, based on the reference sample graph, the generation scheme is used to perform the graph generation task on the original data to obtain the graph data. One optional way is: through the multimodal generation model, the reference sample graph is used as the supplementary knowledge for retrieval enhancement generation, the generation scheme is used to perform the graph generation task on the original data to obtain the graph data. Another optional way is: through the multimodal generation model, the reference sample graph is used as the iterative starting point for iterative optimization generation, the generation scheme is used to perform the graph generation task on the original data to obtain the graph data, which is not limited here.

[0122] One optional way to render the target chart based on the chart data is to use a front-end rendering engine to render the target chart based on the chart data. Another optional way is to use a front-end rendering template to render the target chart based on the chart data. This is not limited here.

[0123] Exemplarily, through a large language model, the above-mentioned reference sample chart is used as supplementary knowledge for retrieval enhancement generation, the above-mentioned generation scheme is used to perform the chart generation task on the original data in Table 1, obtain the chart data, and call the ECharts front-end rendering engine in the HTML page to dynamically render an interactive line chart.

[0124] By guiding the multimodal generation model to generate charts based on reference examples through the generation scheme, it is ensured that highly accurate and highly applicable chart data can be generated even for uncommon chart types. Rendering the target chart on this basis can improve the rendering effect.

[0125] In an optional embodiment of the present specification, after rendering the target chart based on the chart data, the following specific steps are also included: if the rendering fails, obtaining error information; through a multimodal generation model, based on a reference sample chart, using the error information as correction feedback data, adopting a generation scheme to re-execute the chart generation task on the original data, and obtaining updated chart data; re-rendering the target chart based on the updated chart data.

[0126] The error information is the error diagnostic data returned during the chart rendering process, which contains key information such as error type, location description, context data, etc., and is used to locate logical conflicts or technical defects in the generation plan or chart configuration.

[0127] Through a multimodal generation model, based on a reference sample chart, with error information as correction feedback data, a generation scheme is adopted to re-execute the chart generation task on the original data to obtain updated chart data. An optional method is: through a multimodal generation model, with reference sample charts as supplementary knowledge for retrieval enhancement generation, with error information as correction feedback data, a generation scheme is adopted to re-execute the chart generation task on the original data to obtain updated chart data.

[0128] An optional way to re-render the target chart based on the updated chart data is: re-render the target chart based on the updated chart data through the front-end rendering engine. Another optional way is: render the target chart based on the updated chart data through the front-end rendering template, which is not limited here.

[0129] For example, if an error occurs when rendering the generated chart data, the large model will be called for inspection and rewriting. The corresponding prompt information is as follows: You are a data visualization expert. Now you have a Python visualization code modification task. You need to read the source code first, and then refer to the user's modification suggestions (or error messages), modify the source code, and implement chart drawing: Value Error: Cannot convert nontype column to str.

[0130] We need to ensure that the `Month` column does not cause a `Value Error` when converted to a string type. We can use the `astype(str)` method to ensure the correctness of the conversion. Get the modified chart data and re-render it based on the above updated chart data through the front-end rendering engine. Figure 2 Target chart shown.

[0131] In the embodiments of this specification, in the event of a rendering failure, adjustment generation and re-rendering are completed based on the error information. The stability and accuracy of chart generation are significantly improved through the intelligent error diagnosis and correction feedback mechanism. Not only can the logical conflicts or technical defects in the generation plan be quickly located and resolved, but the chart configuration can also be automatically optimized to ensure that the final rendering result meets expectations, thereby enhancing user experience and satisfaction.

[0132] In an optional embodiment of the present specification, after rendering the target chart based on the chart data, the following specific steps are also included: if the rendering is successful, analyzing whether the target chart meets the task objective requirements of the image generation task and the scheme constraints of the generation scheme through a visual language model based on the task description text and the generation scheme, and obtaining an analysis result; if the analysis result is compliant, outputting the target chart to the user terminal; if the analysis result is not compliant and includes adjustable parameters of the generation scheme, adjusting the generation scheme based on the adjustable parameters through a multimodal generation model, and based on a reference sample chart, using the adjusted generation scheme to re-execute the chart generation task on the original data, obtaining updated chart data, and re-rendering the target chart based on the updated chart data.

[0133] The solution constraints of the generated solution are a set of unchangeable parameters defined in the generated solution, for example, chart types (such as line charts), key data mapping rules (such as X-axis binding month fields), and style requirements explicitly specified by the user (such as prohibiting the use of red).

[0134] The analysis results include a binary classification conclusion (compliant / noncompliant) output by the visual language model, with a detailed description of the non-compliant items (such as "the Y-axis is not labeled with units").

[0135] The adjustable parameters of the generated scheme are parameters that allow dynamic changes, such as color saturation, grid line thickness, font size, and other style parameters that do not affect the expression of core data.

[0136] An optional way to re-render the target chart based on the updated chart data is: re-render the target chart based on the updated chart data through the front-end rendering engine. Another optional way is: render the target chart based on the updated chart data through the front-end rendering template, which is not limited here.

[0137] For example, if the rendering is successful, the visual language model is used to analyze whether the target graph meets the task target requirements of the image generation task and the solution constraints of the generation solution based on the task description text and the generation solution, and the analysis result is obtained: {"Function Verification": {"Chart Type Matching": "Line Chart (Compliant)", "Data Mapping Correctness": "X-axis is bound to month (Compliant), Y-axis is bound to number of events (Compliant)"},"Style Verification": {"Main Color": "A blue line is detected (HEX #1f77b4, Compliant)", "Title Text": "The title detected is 'Monthly Event Quantity Trend' (Compliant)", "Anomaly": "The Y-axis label does not show the unit (does not comply with the 'Units Need to be Labeled' constraint in the generated solution)"}} The output analysis result is "non-compliant" and the abnormal items that need to be corrected are marked.

[0138] Through the multimodal generative model, the generation scheme is adjusted based on adjustable parameters: # Y-axis configuration in the original generation scheme: "yAxis": {"label": "Number of events", "unit": ""#undefined unit} # Adjusted generation plan: "yAxis": {"label": "Number of events (times)", "unit": "times", "fontSize": 14# Synchronously enlarge the font size to improve readability}.

[0139] Based on the reference sample chart, the adjusted generation scheme is used to re-execute the chart generation task on the original data to obtain updated chart data, and then re-render based on the updated chart data. Figure 2 The target chart is shown. After re-rendering, the visual language model detects that the Y-axis has been added with units, the style fully complies with the generated solution, and the output analysis result is "Compliant". The target chart is displayed through the user terminal.

[0140] In the embodiments of this specification, post-generation self-reflection logic is introduced, and through multiple rounds of feedback iteration methods, the functions of automatic error repair and automatic optimization of drawing results are realized, thereby further improving the quality of generating target charts, ensuring that the final rendering results meet expectations, and enhancing user experience and satisfaction.

[0141] Corresponding to the above-mentioned embodiments, Figure 3 A flowchart of a method for generating a chart provided by an embodiment of the present specification is shown. Figure 3 As shown: Start: The process starts from the "Start" node. User input stage: The user inputs the original data, task description text and reference pictures. Intention analysis and demand analysis stage: Perform data analysis on the original data to obtain metadata information, and perform image style analysis on the reference pictures to obtain visual description information. Based on the task description text, metadata information and visual description information, perform multimodal joint analysis of the visual chart generation task to obtain the generation plan. Multimodal retrieval stage of chart categories: Based on the generation plan, perform multimodal hybrid retrieval, sort the retrieval results, and screen out high-quality reference sample charts. Chart generation stage: Based on metadata information, generation plan and reference sample charts, integrate prompt information, perform chart generation tasks, render the generated chart data, and return error information if rendering fails, and re-adjust and regenerate. If rendering is successful, check whether the rendered chart meets the task target requirements of the image generation task and the scheme constraints of the generation plan. If not, return the adjustable parameters, and re-adjust and regenerate. When the chart meets the requirements, the process is completed at the "End" node.

[0142] The following combination Figure 4 , taking the application of the chart generation method provided in this specification in the product operation scenario as an example, the chart generation method is further explained. Figure 4 A flowchart of a processing process of a chart generation method applied to a product operation scenario provided by an embodiment of this specification is shown, including the following specific steps: Step 402: Obtain the task description text, original data and reference pictures of the task of generating the product operation chart.

[0143] For example, obtaining the task description text, raw data (such as sales data, user growth data, etc.), and reference images (such as previous successful marketing campaign charts) of a task to generate a product operation chart about product performance, market trends, or user behavior analysis.

[0144] Step 404: Perform data statistics on the original data to obtain statistical information of the original data, extract a preset number of sampled data from the original data, merge the statistical information and the sampled data, and obtain data metadata of the original data.

[0145] For example, calculate the quarterly repurchase rate and extract sample data to generate data metadata.

[0146] Step 406: Perform image style analysis on the reference image using a visual language model to obtain visual description information of the reference image.

[0147] For example, through the visual language model, the image style analysis of the reference image is performed to obtain the visual description information of the chart type, color scheme and label style of the promotional image.

[0148] Step 408: Through the large language model, based on the task description text and the original data, the thinking chain is used to parse the task objective of the task of generating the product operation chart to obtain the task objective analysis result, and based on the task description text, the original data and the visual description information, under the guidance of the task objective analysis result, the thinking chain is used to parse the task implementation path of the task of generating the product operation chart to obtain the generation plan design result, and based on the task objective analysis result and the generation plan design result, a generation plan for the task of generating the product operation chart is generated.

[0149] For example, for the task of "analyzing the differences in retention rates of different user groups from Q1 to Q3 of 2024", a grouped bar chart solution is designed to compare the quarterly retention rates of young users (18-24 years old) and senior users (35+ years old), and mark the significant difference values.

[0150] Step 410: Based on the target chart type, perform keyword matching retrieval from the classification label field of the sample database to obtain a recalled sample chart; based on the semantic vector of the visual description information, perform vector matching retrieval from the preset semantic vector of the sample database to obtain a recalled sample chart; based on the visual vector of the reference image, perform vector matching retrieval from the preset visual vector of the sample database to obtain a recalled sample chart; sort multiple recalled sample charts using a pre-trained sorting model; and screen out a reference sample chart based on the sorting results of the multiple recalled sample charts.

[0151] For example, by matching the keyword "quarterly comparison", semantic vector and visual vector, the 2023 promotional season bar chart is recalled from the sample library, and the quarterly sales comparison chart with uniform color matching and clear labels is selected as a reference sample.

[0152] Step 412: By using a preset prompt information template, the task description text, the generation plan and the reference sample chart are integrated to obtain prompt information for the task of generating the product operation chart.

[0153] For example, the task description, grouped bar chart scheme, and reference examples are integrated into a prompt: "Generate a two-color grouped bar chart with quarters on the X-axis and retention rate on the Y-axis. Refer to the color scheme of the promotional chart (red / blue) and mark the difference value." Step 414: Input the prompt information into the large language model, and based on the reference sample chart, use the generation scheme to perform the product operation chart generation task on the original data to obtain chart data.

[0154] For example, the prompt "Generate a two-color grouped bar chart with quarters on the X-axis and retention rate on the Y-axis. Refer to the color scheme of the promotional chart (red / blue) and mark the difference values." is input into the large language model. Based on the style and structure of the reference sample chart and combined with the original data, the task of generating a product operation chart is performed to obtain a data chart containing a comparison of quarterly retention rates.

[0155] Step 416: Rendering a target chart based on the chart data.

[0156] For example, using the front-end rendering engine, based on the generated chart data, a two-color grouped bar chart is rendered that clearly shows the comparison of the quarterly retention rates of young users and veteran users, and ensures that the significant difference values ​​are correctly marked on the chart.

[0157] Step 418: If rendering fails, obtain error information, use the large language model, based on the reference sample chart, use the error information as correction feedback data, adopt the generation solution to re-execute the product operation chart generation task on the original data, obtain updated chart data, re-render the target chart based on the updated chart data, and output the target chart to the user terminal.

[0158] For example, if an error "color code cannot be recognized" occurs during rendering, the system captures this error message and feeds it back to the large language model. The model adjusts the color settings based on this feedback, re-executes the task of generating the product operation chart, ensures that the color is correctly applied, and renders the chart again until it succeeds.

[0159] Step 420: If the rendering is successful, the target diagram is analyzed to determine whether it meets the task objective requirements of the image generation task and the solution constraints of the generation solution through the visual language model based on the task description text and the generation solution, and an analysis result is obtained.

[0160] For example, after rendering is completed, the generated chart is checked through the visual language model to see whether it accurately reflects the differences in quarterly retention rates of different user groups, including whether elements such as labels, colors, and significant marks are in line with the expected design, to confirm that the chart meets the product operation analysis needs.

[0161] Step 422: If the analysis result is in compliance, output the target chart to the user terminal.

[0162] For example, after inspection and confirmation, the chart not only shows the comparison of retention rates in each quarter, but also provides in-depth insights through the annotation of significant difference values, which fully meets the requirements of the original task objectives. Therefore, it is directly output to the user terminal to provide decision support.

[0163] Step 424: If the analysis result is non-compliant and includes adjustable parameters of the generation scheme, the generation scheme is adjusted based on the adjustable parameters through the large language model, and based on the reference sample chart, the adjusted generation scheme is used to re-execute the product operation chart generation task on the original data to obtain updated chart data, re-render the target chart based on the updated chart data, and output the target chart to the user terminal.

[0164] For example, if it is found that the chart fails to highlight key retention rate differences, the system will adjust the color contrast in the plan or add more annotations as adjustable parameters, and then re-execute the task of generating the product operation chart until a visual chart that meets the requirements and is easy to understand is generated.

[0165] In the embodiment of this specification, the user demand analysis method based on the thinking chain enables the system to deeply understand customer needs and accurately convert them into specific drawing visual language solutions. This method guides the large language model to think step by step, ensuring that each step from analysis to generation scheme design is more accurate, thereby improving the accuracy of chart generation in subsequent steps. Secondly, the introduction of high-quality chart sample database and multimodal retrieval enhancement generation link effectively solves the problem that the existing large language model is prone to errors when dealing with uncommon or complex chart elements. By supporting multiple ways to match sample charts in the base library, sample code is provided to the large language model for reference, which not only improves the success rate of chart generation, but also enhances the beauty of generated charts. The introduction of post-generation self-reflection logic realizes the automatic error correction and optimization function in the chart generation process. Through a multi-round feedback iteration mechanism, the system can automatically correct errors and re-execute the chart generation task after the initial rendering fails until a chart that meets the requirements is successfully output. This self-optimization capability further improves the quality of the final generated chart, ensuring that it can meet the specific needs of customers and reach a higher professional standard, and is suitable for various product operation analysis scenarios. The entire process not only improves work efficiency, but also ensures the professionalism and aesthetics of data visualization results, providing decision-making support for product operations.

[0166] Corresponding to the above method embodiment, this specification also provides a multimodal data processing platform embodiment, Figure 5 FIG. 1 shows a schematic diagram of a multimodal data processing platform provided by an embodiment of the present specification. Figure 5 As shown, the multimodal data processing platform 500 includes a front-end interface 502 and a response unit 504; The front-end interface 502 is used to receive the task description text and original data of the chart generation task sent by the user front end; The response unit 504 is used to execute the steps of the above-mentioned chart generation method.

[0167] In the embodiments of the present specification, it is ensured that even for uncommon chart types, target charts with high accuracy and high applicability can be generated, thereby effectively improving the diversity and quality of chart generation.

[0168] The above is a schematic scheme of a multimodal data processing platform of this embodiment. It should be noted that the technical scheme of the multimodal data processing platform and the technical scheme of the above-mentioned chart generation method belong to the same concept, and the details of the technical scheme of the multimodal data processing platform that are not described in detail can be found in the description of the technical scheme of the above-mentioned chart generation method.

[0169] Corresponding to the above method embodiment, this specification also provides a diagram generating device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a chart generating device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises: An acquisition module 602 is configured to acquire a task description text and raw data of a chart generation task; The parsing module 604 is configured to parse the chart generation task based on the task description text and the original data to obtain a generation solution for the chart generation task; A retrieval module 606 is configured to retrieve a reference sample diagram from a sample diagram database based on the generation scheme; The generation module 608 is configured to generate a target chart by performing a chart generation task on the original data using a generation scheme based on a reference sample chart through a multimodal generation model.

[0170] Optionally, the parsing module 604 is further configured to: perform data analysis on the original data to obtain data metadata of the original data; and perform joint parsing on the chart generation task based on the task description text and the data metadata to obtain a generation plan for the chart generation task.

[0171] Optionally, the parsing module 604 is further configured to: perform data statistics on the original data to obtain statistical information of the original data, and extract a preset number of data from the original data as sampled data; and combine the statistical information and the sampled data to obtain data metadata of the original data.

[0172] Optionally, the device further comprises: a visual module configured to obtain a reference image for a chart generation task; perform image style analysis on the reference image to obtain visual description information of the reference image; Optionally, the parsing module 604 is further configured to: perform multimodal joint parsing on the chart generation task based on the task description text, data meta information and visual description information to obtain a generation solution for the chart generation task.

[0173] Optionally, the visual module is further configured to: perform image style analysis on the reference image through a visual language model to obtain visual description information of the reference image.

[0174] Optionally, the parsing module 604 is further configured to: through the solution generation model, based on the task description text and the original data, use the thinking chain to parse the task objective of the chart generation task to obtain the task objective analysis result, and based on the task description text and the original data, under the guidance of the task objective analysis result, use the thinking chain to parse the task implementation path of the chart generation task to obtain the generation solution design result, and generate the generation solution of the chart generation task based on the task objective analysis result and the generation solution design result.

[0175] Optionally, the generation scheme includes at least one of a target chart type, a reference image for the chart generation task, and visual description information of the reference image for the chart generation task; the retrieval module 606 is further configured as at least one of the following: based on the target chart type, performing a keyword matching retrieval from a classification label field in a sample database to obtain a reference sample chart, wherein the classification label field is a field in the sample database used to identify the chart type; based on a semantic vector of the visual description information, performing a vector matching retrieval from a preset semantic vector in the sample database to obtain a reference sample chart, wherein the preset semantic vector is a semantic coding vector in the sample database used to identify text description information; based on a visual vector of the reference image, performing a vector matching retrieval from a preset visual vector in the sample database to obtain a reference sample chart, wherein the preset visual vector is a visual coding vector in the sample database used to identify the visual description information.

[0176] Optionally, the retrieval module 606 is further configured to: retrieve multiple recall sample charts from the sample chart database based on the generation scheme; sort the multiple recall sample charts through a pre-trained sorting model to obtain sorting results of the multiple recall sample charts; and screen out reference sample charts based on the sorting results of the multiple recall sample charts.

[0177] Optionally, the device further comprises: a prompt integration module configured to integrate the task description text, the original data, the generation scheme and the reference sample chart through a preset prompt information template to obtain prompt information of the chart generation task; Correspondingly, the generation module 608 is further configured to: input the prompt information into the multimodal generation model, perform the chart generation task on the original data, and obtain the chart data.

[0178] Optionally, the generation module 608 is further configured to: perform a chart generation task on the original data based on a reference sample chart using a generation scheme through a multimodal generation model to obtain chart data; and render a target chart based on the chart data.

[0179] Optionally, the device also includes: a checking module, configured to obtain error information if rendering fails; through a multimodal generation model, based on a reference sample chart, using the error information as correction feedback data, adopting a generation scheme to re-execute the chart generation task on the original data, and obtaining updated chart data; and re-rendering the target chart based on the updated chart data.

[0180] Optionally, the device also includes: a reflection module, which is configured to analyze whether the target chart meets the task objective requirements of the image generation task and the scheme constraints of the generation scheme through a visual language model based on the task description text and the generation scheme if the rendering is successful, and obtain an analysis result; if the analysis result is compliant, output the target chart to the user terminal; if the analysis result is not compliant and includes adjustable parameters of the generation scheme, adjust the generation scheme based on the adjustable parameters through a multimodal generation model, and based on a reference sample chart, use the adjusted generation scheme to re-execute the chart generation task on the original data to obtain updated chart data, and re-render the target chart based on the updated chart data.

[0181] In the embodiments of the present specification, it is ensured that highly accurate and highly applicable chart data can be generated even for uncommon chart types, and the rendering module renders the target chart, which effectively improves the diversity and quality of chart generation.

[0182] The above is a schematic scheme of a chart generating device of this embodiment. It should be noted that the technical scheme of the chart generating device and the technical scheme of the above chart generating method belong to the same concept, and the details not described in detail in the technical scheme of the chart generating device can be referred to the description of the technical scheme of the above chart generating method.

[0183] Figure 7 The block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0184] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)) of wired or wireless type, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).

[0185] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0186] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0187] The processor 720 is used to execute the following computer program / instructions, which implement the steps of the above-mentioned chart generation method when executed by the processor.

[0188] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned chart generation method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned chart generation method.

[0189] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned chart generation method when executed by a processor.

[0190] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned chart generation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned chart generation method.

[0191] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned chart generation method when executed by a processor.

[0192] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned chart generation method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned chart generation method.

[0193] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0194] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents contained in the computer readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0195] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0196] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and pass this specification well. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for generating a chart, comprising: Obtain the task description text and raw data of the chart generation task; Based on the task description text and the original data, the chart generation task is parsed to obtain a generation plan for the chart generation task; Based on the generation scheme, retrieving a reference sample chart from a sample chart database; Through a multimodal generation model, based on the reference sample chart, the chart generation task is performed on the original data using the generation scheme to generate a target chart.

2. The method according to claim 1, wherein the step of parsing the chart generation task based on the task description text and the original data to obtain a generation solution for the chart generation task comprises: Performing data analysis on the original data to obtain data metadata of the original data; Based on the task description text and the data meta information, the chart generation task is jointly parsed to obtain a generation solution for the chart generation task.

3. According to the method of claim 2, the step of performing data analysis on the original data to obtain data metadata of the original data comprises: Performing data statistics on the original data to obtain statistical information of the original data, and extracting a preset number of data from the original data as sampled data; The statistical information and the sampled data are combined to obtain data metadata information of the original data.

4. The method according to claim 2, before jointly parsing the chart generation task based on the task description text and the data meta information to obtain a generation solution for the chart generation task, further comprises: Obtaining a reference image for the graph generation task; Performing picture style analysis on the reference picture to obtain visual description information of the reference picture; The step of jointly parsing the chart generation task based on the task description text and the data meta information to obtain a generation solution for the chart generation task includes: Based on the task description text, the data meta information and the visual description information, a multimodal joint analysis is performed on the chart generation task to obtain a generation solution for the chart generation task.

5. The method according to claim 4, wherein the performing picture style analysis on the reference picture to obtain visual description information of the reference picture comprises: The reference image is analyzed for image style using a visual language model to obtain visual description information of the reference image.

6. The method according to any one of claims 1 to 5, wherein the step of parsing the chart generation task based on the task description text and the original data to obtain a generation solution for the chart generation task comprises: Through the solution generation model, based on the task description text and the original data, the thinking chain is used to analyze the task objectives of the chart generation task to obtain task objective analysis results, and based on the task description text and the original data, under the guidance of the task objective analysis results, the thinking chain is used to analyze the task implementation path of the chart generation task to obtain a generation solution design result, and based on the task objective analysis results and the generation solution design results, a generation solution for the chart generation task is generated.

7. The method according to claim 1, wherein the generation scheme comprises at least one of a target chart type, a reference picture of the chart generation task, and visual description information of the reference picture of the chart generation task; The retrieving a reference sample chart from a sample chart database based on the generation scheme includes at least one of the following: Based on the target chart type, a keyword matching search is performed from the classification label field of the sample database to obtain a reference sample chart, wherein: The classification label field is a field in the sample database used to identify the chart type; Based on the semantic vector of the visual description information, a vector matching search is performed from a preset semantic vector in a sample database to obtain a reference sample chart, wherein the preset semantic vector is a semantic coding vector in the sample database for identifying text description information; Based on the visual vector of the reference image, a vector matching search is performed from preset visual vectors in a sample database to obtain a reference sample chart, wherein the preset visual vector is a visual coding vector in the sample database for identifying visual description information.

8. The method according to claim 1 or 7, wherein the step of retrieving a reference sample chart from a sample chart database based on the generation scheme comprises: Based on the generation scheme, a plurality of recalled sample graphs are retrieved from a sample graph database; Sorting the multiple recall sample charts by using a pre-trained sorting model to obtain sorting results of the multiple recall sample charts; Based on the sorting results of the multiple recalled sample charts, a reference sample chart is screened out.

9. The method according to claim 1, before performing the chart generation task on the original data using the multimodal generation model based on the reference sample chart and adopting the generation scheme to obtain the chart data, further comprises: By integrating the task description text, the original data, the generation scheme and the reference sample chart through a preset prompt information template, prompt information of the chart generation task is obtained; The method of performing the chart generation task on the original data by using the multimodal generation model based on the reference sample chart and adopting the generation scheme to obtain chart data includes: The prompt information is input into a multimodal generation model, and the chart generation task is performed on the original data to obtain chart data.

10. The method according to claim 1, wherein the step of generating a target chart by using a multimodal generation model and the generation scheme to perform the chart generation task on the original data based on the reference sample chart comprises: By using a multimodal generation model, based on the reference sample chart, the generation scheme is adopted to perform the chart generation task on the original data to obtain chart data; A target chart is rendered based on the chart data.

11. The method according to claim 10, after rendering the target chart based on the chart data, further comprising: If rendering fails, get the error information; By means of the multimodal generation model, based on the reference sample chart, using the error information as correction feedback data, and adopting the generation scheme to re-execute the chart generation task on the original data, the updated chart data is obtained; The target chart is re-rendered based on the updated chart data.

12. The method according to claim 10, after rendering the target chart based on the chart data, further comprising: If the rendering is successful, the target graph is analyzed to determine whether it meets the task target requirements of the image generation task and the solution constraints of the generation solution through a visual language model based on the task description text and the generation solution, and an analysis result is obtained; If the analysis result is in compliance, outputting the target chart to the user terminal; If the analysis result is inconsistent with and includes adjustable parameters of the generation scheme, the generation scheme is adjusted based on the adjustable parameters through the multimodal generation model, and based on the reference sample chart, the adjusted generation scheme is used to re-execute the chart generation task on the original data to obtain updated chart data, and the target chart is re-rendered based on the updated chart data.

13. A multimodal data processing platform, comprising a front-end interface and a response unit; The front-end interface is used to receive the task description text and original data of the chart generation task sent by the user front-end; The response unit is used to execute the steps of the chart generation method according to any one of claims 1 to 12.

14. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images