A graph semantic embedding visualization method and system based on an improved diffusion model

Abstract

The present invention discloses a method and system for visualizing the semantic embedding of charts based on an improved diffusion model. The method comprises performing targeted preprocessing on chart data; extracting the semantic context of the preprocessed chart and generating relevant keywords; incorporating a cross-modal context containing the interaction and arrangement between textual conditions and visual samples into the forward and reverse processes by improving the diffusion architecture; propagating this context to all time steps of the two processes to adjust their trajectories, thereby promoting cross-modal conditional modeling; and utilizing the diffusion model output results to innovate the visualization method of the semantic embedding of charts. The system comprises a data preprocessing module, an improved diffusion model generation module, and a result evaluation module. The present invention effectively improves the semantic accuracy and richness of chart visualization and is widely applicable to various data analysis, display, and decision-making scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data visualization, and particularly relates to a chart semantic embedding visualization method and system based on an improved diffusion model. BACKGROUND

[0002] With the explosive growth of data, the visualization of data has become crucial for information communication and analysis. Charts, as a common means of data visualization, are widely used in various fields such as business analysis, scientific research, financial data analysis, etc. However, traditional chart visualization methods have many limitations in dealing with complex chart data. On the one hand, for the various types of data (such as structured data, time series data, etc.) contained in the chart, existing methods often cannot fully mine the internal relationship, resulting in poor visualization effect and inability to accurately convey the key information of the data. On the other hand, when facing large-scale or high-dimensional chart data, the processing efficiency of traditional methods is low, which cannot meet the user's demand for quickly obtaining information. At the same time, the existing visualization system lacks sufficient interactivity, and users cannot deeply explore the information behind the chart data. SUMMARY

[0003] In view of the deficiencies of the prior art, the purpose of the present application is to provide a chart semantic embedding visualization method and system based on an improved diffusion model.

[0004] The purpose of the present application is achieved by the following technical solution: a chart semantic embedding visualization method based on an improved diffusion model, comprising the following steps:

[0005] Selecting different types of charts to preprocess the original data to obtain preprocessed chart data; when the type of the chart is a bar chart, a line chart, a pie chart or a scatter chart, converting the chart data into a basic chart corresponding to the type of the chart;

[0006] Extracting the semantic context of the preprocessed chart data and generating relevant keywords;

[0007] Improving the input layer and the loss function of the diffusion model, and using the improved diffusion model to process the preprocessed chart data; the improved diffusion model has a branch specially processing chart structure information in the input layer, and the loss function contains a penalty term considering the characteristics of chart data, and if the chart data has time series characteristics, a time series consistency constraint is added to the loss function;

[0008] The improved diffusion model is combined with an auxiliary model, the auxiliary model including a convolutional neural network (CNN) for extracting features of chart images and a recurrent neural network (RNN) for processing characteristics of chart data sequences, the auxiliary model processing chart data at different stages and inputting results into the improved diffusion model;

[0009] A visual interaction mode is established, and when a user performs an operation on a chart, the diffusion model generates relevant interpretation or prediction information in real time and displays the information in a visual form beside the chart.

[0010] Further, the semantic context of the chart is extracted and relevant keywords are generated, specifically as follows:

[0011] A pre-trained model with a sentence converter is used to extract keywords from titles provided by original data;

[0012] According to the keywords, word similarity is estimated using Word2Vec to generate a batch of keywords, the keywords are converted into vectors, and the retrieval corpus is used to sort the keyword retrieval results, and the retrieval corpus uses the English Wikipedia Dump corpus of November 2021.

[0013] Further, the improved diffusion model is provided with a branch at the input layer for processing chart structure information, including a feature extraction sub-layer and a fusion sub-layer; the feature extraction sub-layer uses a specially designed convolution kernel to capture local features of the chart structure; and the fusion sub-layer is used to fuse the extracted local features with other chart data features (such as numerical features, etc.) that have been subjected to conventional preprocessing.

[0014] Further, the penalty term includes a structure similarity penalty term, a data range penalty term, and a time series characteristic penalty term (for time series chart data).

[0015] Further, the improved diffusion model is used to process the preprocessed chart data, which includes: in the initial stage of the forward diffusion process, for the text condition, advanced natural language processing techniques are used to obtain its feature representation; for example, using a pre-trained language model (such as BERT), input the input text condition into the model, and obtain the vector representation of each word through the output of the last layer of the model. The vector not only contains the semantic information of the word, but also captures the context-related semantic relationship; perform pooling operation (such as average pooling or maximum pooling) on the vector to obtain the global vector representation of the entire text condition; for the visual sample, the feature extraction capability of the convolutional neural network CNN is used; the visual sample (such as a chart image) is input into a pre-trained CNN model (such as VGGNet, ResNet, etc.); the feature map is extracted from the middle layer of the network, which contains multi-level information of the visual sample, from low-level edge and texture features to high-level semantic features; select the output of a suitable intermediate layer as the feature representation of the visual sample, which retains enough detailed information while obtaining a high-level understanding of the image content; the generated image mode is divided into unconditional and conditional, and the generated image target is divided into foreground and background. The unconditional mode does not consider the data structure information of the original chart, and the conditional mode considers the data structure information of the original chart; the unconditional background generation is generated through the original text2img pipeline, and the unconditional foreground generation considers the P ojbect and the generated image I g The cross-attention positioning target region is calculated to distinguish the target and the background, and a mask is obtained; the I g is compared pixel by pixel to obtain the rough target region I obj , and the segmentation neural network ISNet is used to eliminate redundant information to obtain the foreground image with the background removed; the conditional foreground generation integrates the semantic context into the visual mark in the chart, while adhering to the data represented in the chart. First, calculate the mask image I c from the input image I mask , and fuse the attention map in the generation process. The main color of the attention map is additionally extracted and fused in the background generation.

[0016] Further, the improved diffusion model is combined with an auxiliary model, which includes:

[0017] The output of the CNN is subjected to a fully connected layer for feature compression and dimension reduction, converting high-dimensional image features into vectors matching the input dimension of the improved diffusion model;

[0018] The vector is spliced with other preprocessed chart data features (such as numerical features directly extracted from the original data) as part of the input of the improved diffusion model.

[0019] At each forward time step, an attention mechanism-based cross-modal interaction layer is introduced to calculate the correlation weight between the text condition and the visual sample feature;

[0020] Further, the correlation weight between the text condition and the visual sample feature is specifically:

[0021] The text vector and the visual feature map are respectively linearly transformed to obtain query Query, key Key and value Value vectors;

[0022] For the text vector, a linear transformation is performed to obtain a query vector; for the visual feature map, two different linear transformations are performed to obtain a key vector and a value vector, respectively;

[0023] An attention weight matrix is calculated, and the calculation method is the dot product of the query vector and the key vector, and the normalization is performed through a softmax function; the attention weight matrix represents the correlation degree of each word in the text and each position in the visual sample;

[0024] The visual feature representation fused with the text information is obtained by multiplying the attention weight matrix and the value vector and performing summation operation; the visual feature representation is the result of cross-modal interaction, and the semantic information in the text is integrated into the feature of the visual sample, forming an important part of the cross-modal context.

[0025] The application also provides a graph semantic embedding visualization system based on an improved diffusion model, comprising:

[0026] The feature extraction module comprises a data information extraction unit and a semantic context extraction unit; the data information extraction unit is used for deep analysis of input graph data, and different analysis strategies are adopted for different types of graphs; the semantic context extraction unit is used for generating topic words related to the graph data, extracting keywords using a pre-trained MPNet model, estimating word similarity through Word2Vec, and generating a word cloud map of the topic words using a word cloud generator;

[0027] The generation module comprises a generation unit and a modification unit; the generation unit is used for improving the diffusion model, and stable-diffusion-v1-5 is used as a base model; the modification unit is used for processing the replication and refinement of generated elements, and for bar charts, the SSIM calculation method is used to reassemble the grid and the img2img pipeline is used for optimization after generation;

[0028] The evaluation module considers the original data structure and effectively identifies the prominent boundaries and edges in the image by using edge detection technology according to the difference of the condition generation mode; for a bar chart, the height is taken as an important index; for a line chart, the depiction of the trend is taken as an important index; for a pie chart, the sector angle is taken as an important index; and for a scatter chart, the size of the point is taken as an important index.

[0029] Compared with the prior art, the beneficial effects of the present application are:

[0030] Through the innovative data preprocessing method, various information in the chart data can be fully mined, rich materials are provided for subsequent visualization processing, thereby improving the accuracy and integrity of visualization, and users can obtain more comprehensive information.

[0031] The improved diffusion model effectively overcomes the shortcomings of the traditional model in processing chart data, and through a special structure information processing branch and a perfect loss function, the high consistency of the generated chart and the original data is ensured, and the processing capacity for complex chart data is further improved through the cooperative work of the auxiliary model.

[0032] The innovative design of the visualization output module greatly enhances the visualization effect and interactivity of the chart, so that users can more intuitively understand the chart data, and through interactive operation, the information behind the data can be deeply explored, and the user experience and data analysis efficiency are improved.

[0033] The existence of the evaluation module enables the system to be continuously optimized and improved, ensuring that the system always maintains good performance in the long-term use process, and adapts to the needs of different users and different application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The system architecture diagram of the chart semantic embedding visualization system based on the improved diffusion model of the present application;

[0035] Figure 2 The working flow diagram of the chart semantic embedding visualization system based on the improved diffusion model of the present application;

[0036] Figure 3 The unconditional foreground generation schematic diagram in the chart semantic embedding visualization method based on the improved diffusion model of the present application;

[0037] Figure 4 The conditional background generation schematic diagram in the chart semantic embedding visualization method based on the improved diffusion model of the present application. DETAILED DESCRIPTION

[0038] In order to make the features and advantages of the present application clearer, the technical solutions and system processes of the present application are described in further detail below in combination with the drawings and examples. It should be noted that the following examples are intended to facilitate the understanding of the present application and are only a part of the examples, and do not limit the present application in any way.

[0039] As shown in Figure 1 The embodiment of the present application provides a chart semantic embedding visualization system based on an improved diffusion model, which comprises:

[0040] The feature extraction module comprises a data information extraction unit and a semantic context extraction unit; the data information extraction unit is used for deep analysis of input chart data, and different analysis strategies are adopted for different types of charts; the semantic context extraction unit is used for generating subject words related to chart data, extracting keywords by using a pre-trained MPNet model, estimating word similarity by using Word2Vec, and generating a word cloud diagram of subject words by using a word cloud generator;

[0041] The generation module comprises a generation unit and a modification unit; the generation unit is used for improving the diffusion model, and stable-diffusion-v1-5 is used as a base model; the modification unit is used for processing replication and refinement of generated elements, and for a bar chart, a method of calculating SSIM is used to re-splice the grid and the img2img pipeline is used for optimization after generation;

[0042] The evaluation module is used for considering the original data structure according to the difference of the conditional generation mode, effectively identifying the prominent boundaries and edges in the image by using edge detection technology; for a bar chart, height is used as an important index; for a line chart, the depiction of trend is used as an important index; for a pie chart, the sector angle is used as an important index; for a scatter chart, the size of the point is used as an important index.

[0043] As shown in Figure 2 The embodiment of the present application provides a chart semantic embedding visualization method based on an improved diffusion model, which comprises the following steps: first, the original data is subjected to a feature extraction module, and the original data can be various formats of files (such as CSV, JSON, etc.) or relevant data structures extracted from a database. The original data first enters a data information extraction unit, and a parser in the unit starts a corresponding analysis process according to the chart type (judged by file header information or identification in the data structure).

[0044] For line chart data, the parser scans the data line by line, identifies and extracts the coordinate values of each data point, and detects the color code, line thickness value, and type of markers (such as circular or square markers) on the line through specialized image analysis algorithms, and accurately obtains the scale range and label content of the coordinate axis. These information are temporarily stored in a specific data structure in memory.

[0045] For bar charts, image recognition techniques are used to analyze the bar chart image, determine the pixel height and width of each bar, and calculate the actual numerical height and width through a pre-set scale or relative proportion, while obtaining the color information of the bars. Analyze the spacing pattern between the bars to determine whether there is grouping information, and determine the category label to which each bar belongs.

[0046] For pie charts, calculate the central angle of each sector, obtain the sector color through color recognition, and extract the corresponding label and proportion information.

[0047] For scatter plots, collect two-dimensional or multi-dimensional coordinates, size information, and overall distribution pattern of each scatter point.

[0048] After the raw data is parsed, the feature extraction module extracts the semantic context. First, keyword extraction is performed: a pre-trained model with a sentence converter is used to extract keywords from the title provided in the raw data. Then, related word retrieval is performed: Word2Vec is used to estimate word similarity and convert words into vectors, and the 2021.11 English Wikipedia Dump corpus is used as the retrieval corpus to sort the word retrieval results. After generating the topic word cloud, the user can refer to it and then input the Prompt, select the conditions / conditions, foreground / background to generate the image.

[0049] The preprocessed data enters the improved diffusion model for processing. The input layer of the model receives the data, and the structure information perception layer based on the graph convolution network (GCN) and the attention mechanism layer process the graph structure information according to the designed algorithm and parameters, while the loss function monitors the difference between the generated data and the original data in real time to ensure that the model is trained in the optimized direction. In the interaction process between the model and the auxiliary model, CNN and RNN perform feature extraction and sequence processing on the graph data according to their respective functions and algorithms, and then fuse the results with other data features and input them into the improved diffusion model. This interaction is a dynamic and iterative adjustment process, in which the models continuously exchange information and optimize their parameters to improve their processing capabilities for graph data.

[0050] The generated image is divided into unconditional and conditional, and the generated image target is divided into foreground and background, the unconditional method does not consider the data structure information of the original chart, and the conditional method considers the data structure information of the original chart. ojbect And the generated image I g The cross attention positioning target region is located, the threshold is calculated to distinguish the target and the background, and a mask is obtained. g In the image I obj Pixel-by-pixel comparison is performed to obtain a rough target region I c Finally, the segmentation neural network ISNet is used to eliminate redundant information to obtain a foreground image removed from the background. The conditional foreground generation integrates the semantic context into the visual mark in the chart, while adhering to the data represented in the chart. First, the input image I mask The mask image I g is calculated, and the attention map in the generation process is fused, and the main color of the attention map extracted in the background generation is fused. The generated image is subjected to a modification unit and then displayed on a canvas, and the user can modify the next step according to the result of the evaluation module. The evaluation module considers the original data structure for the conditional generation method. For bar charts, the height is the main concern. For line charts, the trend is mainly concerned. For pie charts, the angle of each sector is measured. For scatter plots, the size of each point is estimated. The edge detection technique effectively identifies the prominent boundaries and edges in the image.

[0051] Figure 3 The unconditional foreground generation schematic diagram in the chart semantic embedding visualization method based on the improved diffusion model of the application is shown in the figure. The prompt input by the user is processed by word embedding to obtain a query Q, K and V are obtained from the visual feature map, and the attention map is obtained through Where d represents the dimension of the latent projection dimension of Q and K, and the attention score is normalized by using the Softmax function. The object of interest is extracted from the cluttered background by comparing the pixel difference, the threshold is calculated to distinguish the target and the background, and a mask is obtained. In the image I g Pixel-by-pixel comparison is performed to obtain a rough target region I obj Finally, the segmentation neural network ISNet is used to eliminate redundant information to obtain a foreground image removed from the background.

[0052] Figure 4 The conditional background generation schematic diagram in the chart semantic embedding visualization method based on the improved diffusion model of the application is shown in the figure. Based on the attention map, the main color is extracted, and then a color mask is synthesized by injecting the original chart I c The obtained color mask is connected with the semantic information to obtain a fusion image I fuse. Then inject I fuse Reconfiguration is implemented to obtain a generated graph I g .

[0053] The above-described embodiments detail the technical solutions and beneficial effects of the present application. It should be understood that the above-described embodiments are only specific embodiments of the present application and are not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A chart semantic embedding visualization method based on an improved diffusion model, characterized by: The following steps are involved: Select different types of charts to perform data preprocessing on the original data to obtain preprocessed chart data; when the type of the chart is a bar chart, a line chart, a pie chart, or a scatter chart, convert the chart data into a basic chart corresponding to the type of the chart; Extract the semantic context of preprocessed chart data and generate relevant keywords; Improve the input layer and loss function of the diffusion model, and use the improved diffusion model to process the preprocessed chart data, including: the generated image methods are divided into unconditional and conditional, the generated image targets are divided into foreground and background, the unconditional method does not consider the data structure information of the original chart, and the conditional method considers the data structure information of the original chart; the unconditional background is generated through the original text2img pipeline, and the unconditional foreground is generated considering P object And generate graph I g The cross attention locates the target area, calculates the threshold to distinguish the target and background, and obtains a mask; g Perform pixel-by-pixel comparison to obtain a rough target area I obj , using the segmentation neural network ISNet to eliminate redundant information and obtain the foreground image with the background removed; conditional foreground generation integrates the semantic context into the visual markup in the graph while adhering to the data represented in the graph, first using the input image I c Calculate its mask image I mask , and fused with the attention map in the generation process, and the main color of the attention map is additionally extracted for fusion in the background generation; the improved diffusion model has a branch in the input layer that specifically processes the chart structure information, and its loss function includes a penalty term that considers the characteristics of the chart data. If the chart data has time series characteristics, the time series consistency constraint is added to the loss function; Combining the improved diffusion model with an auxiliary model, wherein the auxiliary model includes a convolutional neural network (CNN) for extracting chart image features and a recurrent neural network (RNN) for processing chart data sequence characteristics, wherein the auxiliary model processes the chart data at different stages and inputs the results into the improved diffusion model; A visual interactive mode is established. When a user operates on a chart, the diffusion model generates relevant explanation or prediction information in real time and displays it in a visual form next to the chart.

2. The method for visualizing chart semantic embedding based on the improved diffusion model according to claim 1, characterized in that: The method of extracting the semantic context of the pre-processed chart data and generating relevant keywords is specifically as follows: Use a pre-trained model with sentence transformer to extract keywords from the titles provided by the original data; According to the keywords, word similarity is estimated using Word2Vec to generate a batch of subject words, which are converted into vectors, and the subject word search results are sorted using the search corpus.

3. The method for visualizing chart semantic embedding based on the improved diffusion model according to claim 1, characterized in that: The improved diffusion model has a branch in the input layer that specifically processes chart structure information, including: a feature extraction sublayer and a fusion sublayer; the feature extraction sublayer uses a convolution kernel to capture local features of the chart structure; the fusion sublayer is used to fuse the extracted local features with other conventionally preprocessed chart data features.

4. The method for visualizing chart semantic embedding based on the improved diffusion model according to claim 1, characterized in that: The penalty items include: structural similarity penalty item, data range penalty item, and time series characteristic penalty item.

5. The method for visualizing chart semantic embedding based on the improved diffusion model according to claim 1, characterized in that: The use of the improved diffusion model to process the pre-processed chart data includes: at the beginning of the forward diffusion process, for text conditions, using advanced natural language processing technology to obtain its feature representation; for visual samples, using the feature extraction capability of the convolutional neural network (CNN) to input the visual samples into a pre-trained CNN model; extracting feature maps from the middle layer of the network, the feature maps containing multi-level information of the visual samples, the multi-level information including low-level edge and texture features and high-level semantic features; selecting appropriate middle layer outputs as feature representations of the visual samples, while retaining sufficient detail information while obtaining a high-level understanding of the image content; the generated image modes are divided into unconditional and conditional, and the generated image targets are divided into foreground and background, the unconditional mode does not consider the data structure information of the original chart, and the conditional mode considers the data structure information of the original chart; the unconditional background is generated through the original text2img pipeline, and the unconditional foreground is generated by considering P object And generate graph I g The cross attention locates the target area, calculates the threshold to distinguish the target and background, and obtains a mask; g Perform pixel-by-pixel comparison to obtain a rough target area I obj , using the segmentation neural network ISNet to eliminate redundant information and obtain the foreground image with the background removed; conditional foreground generation integrates the semantic context into the visual markup in the graph while adhering to the data represented in the graph, first using the input image I c Calculate its mask image I mask , fused with the attention map in the generation process, and the main color of the attention map is additionally extracted for fusion in background generation.

6. The method for visualizing chart semantic embedding based on the improved diffusion model according to claim 1, characterized in that: The improved diffusion model is combined with the auxiliary model to include: The output of the CNN is passed through a fully connected layer for feature compression and dimensionality reduction, converting the high-dimensional image features into vectors that match the input dimension of the improved diffusion model; The vector is concatenated with other pre-processed chart data features as part of the input of the improved diffusion model; At each forward time step, a cross-modal interaction layer based on the attention mechanism is introduced to calculate the correlation weight between the text condition and the visual sample features.

7. The method for visualizing chart semantic embedding based on the improved diffusion model according to claim 6, characterized in that: The calculation of the correlation weight between the text condition and the visual sample feature is specifically: For text vectors, a linear transformation is performed to obtain the query vector; for visual feature maps, two different linear transformations are performed to obtain the key vector and value vector respectively; Calculate the attention weight matrix, which is calculated as the dot product of the query vector and the key vector, and normalize it through a softmax function; The attention weight matrix represents the degree of relevance between each word in the text and each position in the visual sample; By multiplying the attention weight matrix with the value vector and performing a sum operation, a visual feature representation that integrates text information is obtained; the visual feature representation is the result of cross-modal interaction, which integrates the semantic information in the text into the features of the visual sample, forming an important component of the cross-modal context.

8. A chart semantic embedding visualization system based on an improved diffusion model, characterized by: include: The feature extraction module includes a data information extraction unit and a semantic context extraction unit. The data information extraction unit is used to perform in-depth analysis on the input chart data, using different analysis strategies for different types of charts. The semantic context extraction unit is used to generate keywords related to the chart data, using a pre-trained MPNet model to extract keywords, and using Word2Vec to estimate word similarity, and using a word cloud generator to generate a word cloud diagram of the keyword. The generation module includes a generation unit and a modification unit; the generation unit is used to improve the diffusion model and use the improved diffusion model to process the pre-processed chart data, including: the generated image mode is divided into unconditional and conditional, the generated image target is divided into foreground and background, the unconditional mode does not consider the data structure information of the original chart, and the conditional mode considers the data structure information of the original chart; the unconditional background is generated through the original text2img pipeline, and the unconditional foreground is generated by considering P object And generate graph I g The cross attention locates the target area, calculates the threshold to distinguish the target and background, and obtains a mask; g Perform pixel-by-pixel comparison to obtain a rough target area I obj , using the segmentation neural network ISNet to eliminate redundant information and obtain the foreground image with the background removed; conditional foreground generation integrates the semantic context into the visual markup in the graph while adhering to the data represented in the graph, first using the input image I c Calculate its mask image I mask , fused with the attention map during the generation process, and the main color of the attention map is additionally extracted for fusion in the background generation; stable-diffusion-v1-5 is used as the base model; the modification unit is used to handle the replication and refinement of the generated elements. For the bar chart, the grid is re-stitched by calculating SSIM and the img2img pipeline is used for post-generation optimization; The evaluation module is used to consider the original data structure according to different condition generation methods and use edge detection technology to effectively identify prominent boundaries and edges in the image; for bar charts, the height is used as an important indicator; for line charts, the depiction of trends is used as an important indicator; for pie charts, the sector angle is used as an important indicator; for scatter plots, the size of the points is used as an important indicator.

Images