Chart semantic embedding visualization method and system based on improved diffusion model

Abstract

The invention discloses a chart semantic embedding visualization method and a chart semantic embedding visualization system based on an improved diffusion model. The method comprises the steps of performing targeted preprocessing on chart data; extracting the preprocessed graph semantic context and generating related subject terms; a cross-modal context containing interaction and arrangement between a text condition and a visual sample is incorporated into a forward flow and a reverse flow by improving a framework of difusion; propagating this context to all time steps of the two processes to adjust their trajectories, thereby facilitating cross-modal conditional modeling; and a chart semantic embedding visualization mode is innovated by utilizing a diffusion model output result. The system comprises a data preprocessing module, an improved diffusion model generation module, a result evaluation module and the like. According to the method, the semantic accuracy and richness of chart visualization are effectively improved, and the method is widely applied to various data analysis, display and decision scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of data visualization, and in particular to a chart semantic embedding visualization method and system based on an improved diffusion model. Background Art

[0002] With the explosive growth of data volume, data visualization has become crucial for information communication and analysis. As a common means of data visualization, charts are widely used in various fields, such as business analysis, scientific research, financial data analysis, etc. However, traditional chart visualization methods have many limitations when processing complex chart data. On the one hand, for the various types of data contained in the chart (such as structural data, time series data, etc.), existing methods often cannot fully explore their internal relationships, resulting in poor visualization effects and inability to accurately convey the key information of the data. On the other hand, when faced with large-scale or high-dimensional chart data, the processing efficiency of traditional methods is low and cannot meet the user's needs for quick information acquisition. At the same time, the existing visualization system lacks sufficient interactivity, making it difficult for users to deeply explore the information behind the chart data. Summary of the invention

[0003] In view of the deficiencies in the prior art, the object of the present invention is to provide a diagram semantic embedding visualization method and system based on an improved diffusion model.

[0004] The object of the present invention is achieved through the following technical solution: a diagram semantic embedding visualization method based on an improved diffusion model, comprising the following steps:

[0005] Select different types of charts to preprocess the original data to obtain preprocessed chart data; when the type of the chart is a bar chart, a line chart, a pie chart, or a scatter chart, convert the chart data into a basic chart corresponding to the type of the chart;

[0006] Extract the semantic context of preprocessed chart data and generate relevant subject terms;

[0007] The input layer and loss function of the diffusion model are improved, and the preprocessed chart data are processed using the improved diffusion model; the improved diffusion model has a branch in the input layer that specifically processes chart structure information, and its loss function includes a penalty term that considers the characteristics of the chart data. If the chart data has time series characteristics, a time series consistency constraint is added to the loss function;

[0008] Combining the improved diffusion model with an auxiliary model, the auxiliary model includes a convolutional neural network (CNN) for extracting chart image features and a recurrent neural network (RNN) for processing chart data sequence characteristics, the auxiliary model processes the chart data at different stages and inputs the results into the improved diffusion model;

[0009] A visual interactive mode is established. When a user operates on a chart, the diffusion model generates relevant explanation or prediction information in real time and displays it in a visual form next to the chart.

[0010] Furthermore, the extraction of the semantic context of the chart and the generation of relevant keywords are specifically as follows:

[0011] A pre-trained model with sentence transformer is used to extract keywords from the titles provided by the raw data;

[0012] According to the keywords, Word2Vec is used to estimate word similarity, generate a batch of keywords, convert the keywords into vectors, and use the retrieval corpus to sort the keyword retrieval results. The retrieval corpus adopts the English Wikipedia Dump corpus of 2021.11.

[0013] Furthermore, the improved diffusion model has a branch in the input layer that specifically processes chart structure information, including: a feature extraction sublayer and a fusion sublayer; the feature extraction sublayer uses a specially designed convolution kernel to capture local features of the chart structure; the fusion sublayer is used to fuse the extracted local features with other conventionally preprocessed chart data features (such as numerical features, etc.).

[0014] Furthermore, the penalty items include: structural similarity penalty item, data range penalty item, and time series characteristic penalty item (for time series chart data).

[0015] Furthermore, the use of the improved diffusion model to process the preprocessed chart data includes: at the beginning of the forward diffusion process, for text conditions, using advanced natural language processing technology to obtain its feature representation; for example, using a pre-trained language model (such as BERT), inputting the input text conditions into the model, and obtaining the vector representation of each word through the last layer output of the model, wherein the vector not only contains the semantic information of the word, but also captures the context-related semantic relationship; performing a pooling operation (such as average pooling or maximum pooling) on ​​the vector to obtain a global vector representation of the entire text condition; for visual samples, utilizing the feature extraction capability of a convolutional neural network (CNN); and pooling the visual samples. The sample (such as a chart image) is input into a pre-trained CNN model (such as VGGNet, ResNet, etc.); feature maps are extracted from the middle layer of the network, and the feature maps contain multi-level information of the visual sample, from low-level edge and texture features to high-level semantic features; the appropriate middle layer output is selected as the feature representation of the visual sample, while retaining enough detail information to obtain a high-level understanding of the image content; the generated image methods are divided into unconditional and conditional, and the generated image targets are divided into foreground and background. The unconditional method does not consider the data structure information of the original chart, and the conditional method considers the data structure information of the original chart; the unconditional background generation is generated through the original text2img pipeline, and the unconditional foreground generation considers P ojbect And generate graph I g The cross attention locates the target area, calculates the threshold to distinguish the target from the background, and obtains a mask; g , and obtain the rough target area I obj , using the segmentation neural network ISNet to eliminate redundant information and obtain the foreground image with the background removed; conditional foreground generation integrates the semantic context into the visual tags in the chart while adhering to the data represented in the chart. c Calculate its mask image I mask , fused with the attention map in the generation process, and the main color of the attention map is additionally extracted for fusion in the background generation.

[0016] Furthermore, the improved diffusion model is combined with the auxiliary model to include:

[0017] The output of the CNN is passed through a fully connected layer for feature compression and dimensionality reduction, converting the high-dimensional image features into vectors that match the input dimension of the improved diffusion model;

[0018] concatenating the vector with other preprocessed chart data features (such as numerical features extracted directly from the original data) as part of the input of the improved diffusion model;

[0019] At each forward time step, a cross-modal interaction layer based on the attention mechanism is introduced to calculate the correlation weight between the text condition and the visual sample features;

[0020] Furthermore, the calculation of the correlation weight between the text condition and the visual sample feature is specifically:

[0021] Perform linear transformations on the text vector and visual feature map to obtain the query, key, and value vectors;

[0022] For text vectors, a linear transformation is used to obtain the query vector; for visual feature maps, two different linear transformations are used to obtain the key vector and value vector respectively;

[0023] Calculate the attention weight matrix, which is calculated by the dot product of the query vector and the key vector and normalized by a softmax function; the attention weight matrix represents the degree of relevance of each word in the text to each position in the visual sample;

[0024] By multiplying the attention weight matrix with the value vector and performing a sum operation, a visual feature representation that integrates text information is obtained; the visual feature representation is the result of cross-modal interaction, which integrates the semantic information in the text into the features of the visual sample, forming an important component of the cross-modal context.

[0025] The present invention also provides a chart semantic embedding visualization system based on an improved diffusion model, comprising:

[0026] The feature extraction module includes a data information extraction unit and a semantic context extraction unit; the data information extraction unit is used to perform in-depth analysis on the input chart data, and adopts different analysis strategies for different types of charts; the semantic context extraction unit is used to generate subject words related to the chart data, extract keywords using the pre-trained MPNet model, estimate word similarity through Word2Vec, and use a word cloud generator to generate a word cloud diagram of the subject words;

[0027] The generation module includes a generation unit and a modification unit; the generation unit is used to improve the diffusion model, using stable-diffusion-v1-5 as the base model; the modification unit is used to handle the replication and refinement of the generated elements. For the bar chart, the grid is re-joined by calculating SSIM and the img2img pipeline is used for post-generation optimization;

[0028] The evaluation module is used to consider the original data structure according to different condition generation methods and use edge detection technology to effectively identify the prominent boundaries and edges in the image; for bar charts, the height is used as an important indicator; for line charts, the depiction of trends is used as an important indicator; for pie charts, the sector angle is used as an important indicator; for scatter plots, the size of the points is used as an important indicator.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] Through innovative data preprocessing methods, the present invention can fully mine various information in chart data, provide rich materials for subsequent visualization processing, thereby improving the accuracy and completeness of visualization and enabling users to obtain more comprehensive information.

[0031] The improved diffusion model effectively overcomes the shortcomings of traditional models in processing chart data. Through a dedicated structural information processing branch and a perfect loss function, it ensures a high degree of consistency between the generated chart and the original data. At the same time, the collaborative work with the auxiliary model further improves the processing ability of complex chart data.

[0032] The innovative design of the visualization output module greatly enhances the visualization effect and interactivity of the charts, allowing users to understand the chart data more intuitively and deeply explore the information behind the data through interactive operations, improving user experience and data analysis efficiency.

[0033] The existence of the evaluation module enables the system to be continuously optimized and improved, ensuring that the system always maintains good performance during long-term use and adapts to the needs of different users and different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A system architecture diagram of a chart semantic embedding visualization system based on an improved diffusion model of the present invention;

[0035] Figure 2 A workflow diagram of a chart semantic embedding visualization system based on an improved diffusion model of the present invention;

[0036] Figure 3 A schematic diagram of unconditional foreground generation in a diagram semantic embedding visualization method based on an improved diffusion model of the present invention;

[0037] Figure 4 The present invention provides a schematic diagram of conditional background generation in the diagram semantic embedding visualization method based on the improved diffusion model. DETAILED DESCRIPTION

[0038] In order to make the features and advantages of the present invention more clear, the technical scheme and system flow of the present invention are further described in detail below in conjunction with the drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and are only a part of the embodiments, not all of the embodiments, and do not have any limiting effect on the present invention.

[0039] like Figure 1 As shown, an embodiment of the present invention provides a chart semantic embedding visualization system based on an improved diffusion model, including:

[0040] The feature extraction module includes a data information extraction unit and a semantic context extraction unit; the data information extraction unit is used to perform in-depth analysis on the input chart data, and adopts different analysis strategies for different types of charts; the semantic context extraction unit is used to generate subject words related to the chart data, extract keywords using the pre-trained MPNet model, estimate word similarity through Word2Vec, and use a word cloud generator to generate a word cloud diagram of the subject words;

[0041] The generation module includes a generation unit and a modification unit; the generation unit is used to improve the diffusion model, using stable-diffusion-v1-5 as the base model; the modification unit is used to handle the replication and refinement of the generated elements. For the bar chart, the grid is re-joined by calculating SSIM and the img2img pipeline is used for post-generation optimization;

[0042] The evaluation module is used to consider the original data structure according to different condition generation methods and use edge detection technology to effectively identify the prominent boundaries and edges in the image; for bar charts, the height is used as an important indicator; for line charts, the depiction of trends is used as an important indicator; for pie charts, the sector angle is used as an important indicator; for scatter plots, the size of the points is used as an important indicator.

[0043] like Figure 2 As shown, an embodiment of the present invention provides a method for visualizing the semantic embedding of a chart based on an improved diffusion model, comprising the following steps: first, the original data passes through a feature extraction module, and the original data can be a file in various formats (such as CSV, JSON, etc.) or a related data structure extracted from a database. The original data first enters a data information extraction unit, and a parser in the unit starts a corresponding parsing process according to the chart type (determined by the file header information or the identifier in the data structure).

[0044] For line chart data, the parser scans the data line by line, identifies and extracts the coordinate value of each data point, and uses a specialized image analysis algorithm to detect the color code of the line, the thickness of the line, and the type of mark on the line (such as circular or square marks), and accurately obtains the scale range and label content of the coordinate axis. This information is temporarily stored in a specific data structure in memory.

[0045] For bar graphs, image recognition technology is used to analyze the bar graph image, determine the pixel height and width of each column, calculate the actual numerical height and width through the preset scale or relative scale relationship, and obtain the color information of the column. The spacing pattern between the columns is analyzed to determine whether there is grouping information, and determine the category label to which each column belongs.

[0046] For pie charts, calculate the central angle of each sector area, obtain the sector color through color recognition, and extract the corresponding label and proportion information.

[0047] For scatter plots, collect the two-dimensional or multi-dimensional coordinates, size information, and overall distribution pattern of each scatter point.

[0048] After the raw data is parsed, the feature extraction module extracts the semantic context and first performs keyword extraction: a pre-trained model with a sentence converter is used to extract keywords from the title provided in the raw data. Then perform related word retrieval: Word2Vec is used to estimate word similarity and convert words into vectors. The English Wikipedia Dump corpus of November 2021 is used as the search corpus to sort the word search results. After the topic word cloud is generated, the user can refer to it, then enter Prompt, select conditional / non-conditional, foreground / background to generate the image.

[0049] The preprocessed data enters the improved diffusion model for processing. After the input layer of the model receives the data, the structural information perception layer and attention mechanism layer based on the graph convolutional network (GCN) process the graph structure information according to the designed algorithms and parameters. At the same time, the loss function monitors the difference between the generated data and the original data in real time to ensure that the model is trained in the optimized direction. In the interaction between the model and the auxiliary model, CNN and RNN extract features and process sequences of the graph data according to their respective functions and algorithms, and then fuse the results with other data features and input them into the improved diffusion model. This interaction is a dynamic and iterative adjustment process. During the training process, the models continuously transmit information and optimize their parameters to improve their processing capabilities for graph data.

[0050] The generated image methods are divided into unconditional and conditional. The generated image targets are divided into foreground and background. The unconditional method does not consider the data structure information of the original chart, while the conditional method considers the data structure information of the original chart. The unconditional background is generated through the original text2img pipeline, and the unconditional foreground is generated by considering P ojbect And generate graph I g The cross attention locates the target area, calculates the threshold to distinguish the target from the background, and obtains a mask. g , and obtain the rough target area I obj Finally, the segmentation neural network ISNet is used to eliminate redundant information and obtain the foreground image with the background removed. Conditional foreground generation integrates semantic context into the visual tags in the graph while adhering to the data represented in the graph. First, the input image I c Calculate its mask image I mask , and fused with the attention map in the generation process. The main color of the attention map is additionally extracted in the background generation for fusion. The generated image will pass through the modification unit and then displayed on the canvas. The user can make further modifications based on the results of the evaluation module. The evaluation module takes into account the original data structure for the conditional generation method. For bar charts, the main focus is on height as an indicator. For line charts, the main focus is on depicting trends. For pie charts, the angle of each sector is measured. For scatter plots, the size of each point is estimated. Edge detection technology is used to effectively identify prominent boundaries and edges in images.

[0051] Figure 3 This is a schematic diagram of unconditional foreground generation in the diagram semantic embedding visualization method based on the improved diffusion model of the present invention. The prompt input by the user is processed by word embedding to obtain the query Q, and K, V are obtained from the visual feature map. The attention map is obtained, where d represents the dimension of the potential projection dimension of Q and K, and the attention score is normalized using the Softmax function. The threshold is calculated by comparing the pixel difference to extract the object of interest from the cluttered background to distinguish the target from the background and obtain a mask. g , and obtain the rough target area I obj ,Finally, the segmentation neural network ISNet is used to eliminate redundant information and obtain the foreground image with the background removed.

[0052] Figure 4 The schematic diagram of conditional background generation in the diagram semantic embedding visualization method based on the improved diffusion model of the present invention is to extract the main color based on the attention map, and then inject the original diagram I c Synthesize a color mask and connect the obtained color mask with the semantic information to obtain the fusion image I fuse. Then inject I during the generation process fuse Reconstruction is achieved to obtain the generated graph I g .

[0053] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for visualizing chart semantic embedding based on an improved diffusion model, characterized in that: The following steps are involved: Select different types of charts to preprocess the original data to obtain preprocessed chart data; when the type of the chart is a bar chart, a line chart, a pie chart, or a scatter chart, convert the chart data into a basic chart corresponding to the type of the chart; Extract the semantic context of preprocessed chart data and generate relevant subject terms; The input layer and loss function of the diffusion model are improved, and the preprocessed chart data are processed using the improved diffusion model; the improved diffusion model has a branch in the input layer that specifically processes chart structure information, and its loss function includes a penalty term that considers the characteristics of the chart data. If the chart data has time series characteristics, a time series consistency constraint is added to the loss function; Combining the improved diffusion model with an auxiliary model, the auxiliary model includes a convolutional neural network (CNN) for extracting chart image features and a recurrent neural network (RNN) for processing chart data sequence characteristics, the auxiliary model processes the chart data at different stages and inputs the results into the improved diffusion model; A visual interactive mode is established. When a user operates on a chart, the diffusion model generates relevant explanation or prediction information in real time and displays it in a visual form next to the chart.

2. The method for visualizing the semantic embedding of a chart based on an improved diffusion model according to claim 1, characterized in that: The method of extracting the semantic context of the preprocessed chart data and generating relevant keywords is specifically as follows: A pre-trained model with sentence transformer is used to extract keywords from the titles provided by the raw data; According to the keywords, Word2Vec is used to estimate word similarity, generate a batch of subject words, convert the subject words into vectors, and use the search corpus to sort the subject word search results.

3. The method for visualizing the semantic embedding of a chart based on an improved diffusion model according to claim 1, characterized in that: The improved diffusion model has a branch in the input layer that specifically processes chart structure information, including: a feature extraction sublayer and a fusion sublayer; the feature extraction sublayer uses a convolution kernel to capture local features of the chart structure; the fusion sublayer is used to fuse the extracted local features with other conventionally preprocessed chart data features.

4. The method for visualizing the semantic embedding of a chart based on an improved diffusion model according to claim 1, characterized in that: The penalty items include: structural similarity penalty item, data range penalty item, and time series characteristic penalty item.

5. The method for visualizing the semantic embedding of a chart based on an improved diffusion model according to claim 1, characterized in that: The use of the improved diffusion model to process the pre-processed chart data includes: at the beginning stage of the forward diffusion process, for text conditions, using advanced natural language processing technology to obtain its feature representation; for visual samples, using the feature extraction capability of the convolutional neural network (CNN) to input the visual samples into a pre-trained CNN model; extracting feature maps from the middle layer of the network, the feature maps contain multi-level information of the visual samples, the multi-level information including low-level edge and texture features and high-level semantic features; selecting a suitable middle layer output as the feature representation of the visual sample, while retaining sufficient detail information, obtaining a high-level understanding of the image content; the generated image mode is divided into unconditional and conditional, the generated image target is divided into foreground and background, the unconditional mode does not consider the data structure information of the original chart, and the conditional mode considers the data structure information of the original chart; the unconditional background generation is generated through the original text2img pipeline, and the unconditional foreground generation considers P ojbect And generate graph I g The cross attention locates the target area, calculates the threshold to distinguish the target from the background, and obtains a mask; g , and obtain the rough target area I obj , using the segmentation neural network ISNet to eliminate redundant information and obtain the foreground image with the background removed; conditional foreground generation integrates the semantic context into the visual tags in the chart while adhering to the data represented in the chart. c Calculate its mask image I mask , fused with the attention map in the generation process, and the main color of the attention map is additionally extracted for fusion in the background generation.

6. The method for visualizing the semantic embedding of a chart based on an improved diffusion model according to claim 1, characterized in that: The improved diffusion model combined with the auxiliary model includes: The output of the CNN is passed through a fully connected layer for feature compression and dimensionality reduction, converting the high-dimensional image features into vectors that match the input dimension of the improved diffusion model; concatenating the vector with other preprocessed chart data features as part of the input of the improved diffusion model; At each forward time step, a cross-modal interaction layer based on the attention mechanism is introduced to calculate the correlation weights between text conditions and visual sample features.

7. The method for visualizing the semantic embedding of a chart based on an improved diffusion model according to claim 6, characterized in that: The calculation of the correlation weight between the text condition and the visual sample feature is specifically: For text vectors, a linear transformation is used to obtain the query vector; for visual feature maps, two different linear transformations are used to obtain the key vector and value vector respectively; Calculate the attention weight matrix, which is calculated as the dot product of the query vector and the key vector, and normalize it through a softmax function; The attention weight matrix represents the degree of relevance between each word in the text and each position in the visual sample; By multiplying the attention weight matrix with the value vector and performing a sum operation, a visual feature representation that integrates text information is obtained; the visual feature representation is the result of cross-modal interaction, which integrates the semantic information in the text into the features of the visual sample, forming an important component of the cross-modal context.

8. A chart semantic embedding visualization system based on an improved diffusion model, characterized in that: include: The feature extraction module includes a data information extraction unit and a semantic context extraction unit; the data information extraction unit is used to perform in-depth analysis on the input chart data, and adopts different analysis strategies for different types of charts; the semantic context extraction unit is used to generate subject words related to the chart data, extract keywords using the pre-trained MPNet model, estimate word similarity through Word2Vec, and use a word cloud generator to generate a word cloud diagram of the subject words; The generation module includes a generation unit and a modification unit; the generation unit is used to improve the diffusion model, using stable-diffusion-v1-5 as the base model; the modification unit is used to handle the replication and refinement of the generated elements. For the bar chart, the grid is re-joined by calculating SSIM and the img2img pipeline is used for post-generation optimization; The evaluation module is used to consider the original data structure according to different condition generation methods and use edge detection technology to effectively identify the prominent boundaries and edges in the image; for bar charts, the height is used as an important indicator; for line charts, the depiction of trends is used as an important indicator; for pie charts, the sector angle is used as an important indicator; for scatter plots, the size of the points is used as an important indicator.

Images