A method for interactive visual exploratory analysis using Vega-Lite

Abstract

The present invention relates to a method for realizing interactive visual exploratory analysis using Vega-Lite, comprising: step 1, constructing a knowledge graph of a problem type model, an analysis data set model, an EDA component model, and a Vega-Lite interactive grammar model in the form of a neo4j graph database, and establishing a connection relationship between the models; step 2, using EDA to perform data analysis, selecting an analysis data set based on the analysis problem type, recommending the type of visualization component according to the needs of each step of analysis, and continuing to perform EDA data analysis on the results obtained from the interactive behavior of the visualization component until an accurate conclusion is obtained; step 3, connecting the graph database of four models based on the py2neo package in python, reading the configuration information of entity nodes and attribute nodes and the relationship between nodes, starting from a problem type to recommending visualization components, and finally automatically generating Vega-Lite visualization grammar. The method realizes the separation of data and visualization presentation, and improves the credibility of EDA data analysis results.

Description

Technical Field

[0001] The present invention relates to a visualization implementation method, and in particular to a method for implementing interactive visualization exploratory analysis using Vega-Lite Background Art

[0002] Traditional data analysis, Exploratory Data Analysis (EDA) and Bayesian data analysis are three popular data analysis methods. They all aim at specific responses and use data to combine the factors that affect the response, ultimately achieving a controllable fit from the factors to the final response. However, the analysis steps and focuses among the three are different.

[0003] For traditional data analysis, data collection is followed by the establishment of a model (normality, linearity, etc.), and subsequent analysis, estimation, and testing focus on the parameters of the model.

[0004] For EDA, data collection is followed by analysis of the data's characteristics with the goal of inferring which model is most appropriate.

[0005] For Bayesian data analysis, analysts attempt to incorporate scientific / engineering knowledge / expertise into the analysis by imposing data-independent distribution on the parameters of the selected model, and infer the model parameters by combining the prior distribution of the parameters with the collected data. Compared with the other two data analysis methods that are based on partial available data and rely on basic assumptions, EDA finds the most appropriate model through the characteristics of all available data and very few basic assumptions, reducing the tedious process of model parameter estimation and model analysis, and the resulting graphical model can intuitively and accurately show the characteristics of all data.

[0006] Visualization after data analysis is a process of expressing abstract concepts in a figurative way and visualizing abstract language in a concrete and graphic way. Through elements such as charts and graphs, people can quickly understand the meaning, trends and correlations in the data. Interactive visualization is an advanced feature in graphic visualization. It must have a way to interact with humans, such as clicking buttons, moving sliders, and a fast enough response time to show the true relationship between input and output. Interactive visualization can be achieved through interactive tools (such as Tableau, PowerBI, Gephi, etc.), configuration tools (using json data objects such as Echarts, Plotly, etc.) and programmatic tools (Processin, D3.js, OpenGL, WebGL, etc.).

[0007] Among them, EDA refers to a data analysis method that explores the structure and regularity of data by drawing, tabulating, fitting equations, calculating characteristic quantities, etc., with as few assumptions as possible. Compared with pure mathematical data analysis, EDA focuses on drawing. The main goal is to maximize the data analyst's insight into data characteristics, discover the potential structure of data, extract important variables, detect outliers and anomalies, test whether basic assumptions are valid, develop simple models, determine the best combination of influencing factors, etc. Each EDA analysis starts from a specific problem. According to the type of problem and the description of the research object and indicators, the analysis data set is selected and the analysis data type is converted. According to the type of problem, the data set description, feature analysis, correlation analysis, trend analysis and other data set exploratory analysis are obtained. According to the calculation and feature analysis of each feature of the data set, the type of visualization component that meets the requirements of the problem is selected. This process is relatively cumbersome and based on experience.

[0008] The visualization implementation method Vega-Lite is a set of high-level syntaxes proposed in 2017 that can quickly build interactive visualizations from data to graphics. It is based on the underlying visualization syntax Vega for upper-level encapsulation. Compared with other relatively low-level visualization syntaxes, Vega-Lite can complete the creation of some general charts, combined charts and their interactive functions through JSON configuration code. Its basic part is mainly composed of three parts: data, mark type, encoding, and visualization interaction is achieved by configuring the JSON syntax of the params module. This part requires technicians to master the configuration module parameter information.

[0009] The EDA process in the prior art only accepts relevant information from the problem and the data set, and the analysis process is relatively cumbersome, which places high demands on analysts. In addition, the existing interactive visualization implementation methods have certain requirements on the operation of the software on the one hand, and programming on the other hand. On the other hand, users need to understand the complete meaning and information of the parameters in order to realize the corresponding interactive visualization function by configuring the parameters. The implementation process is closely linked to the data. Summary of the invention

[0010] The purpose of the present invention is to analyze the type of problem in exploratory analysis, select the visualization format of the EDA model through the selection of data sets and exploratory analysis of data set features, and create knowledge graphs of problem type model, analysis data set model, EDA component model and Vega-Lite interactive grammar model based on the neo4j graph database. Then, by reading the relevant configuration information of visualization in the knowledge graph, the automatic generation from problem type to interactive visualization grammar is realized, and finally the separation of data and presentation and interactive visualization are achieved.

[0011] To achieve the above object, the specific technical solution of the present invention is to provide a method for implementing interactive visual exploratory analysis using Vega-Lite, which is characterized by:

[0012] Step 1: Build the knowledge graphs of the problem type model, analysis data set model, EDA component model, and Vega-Lite interactive grammar model in the form of a neo4j graph database, save the relevant information of each model in the form of entity nodes, attribute nodes, and relationships to the knowledge graph, and establish the connection relationship between the models;

[0013] Step 2: Use EDA to perform data analysis. Select the analysis data set based on the analysis problem type. Perform EDA data analysis based on the structural and quality characteristics of the analysis data set. Recommend the type of visualization component based on the needs of each step of analysis. Continue to perform EDA data analysis on the results of the interaction behavior of the visualization components until an accurate conclusion is obtained.

[0014] Step 3: Exploratory analysis of the implementation of the interactive visualization grammar automatic generation algorithm. Based on the py2neo package in Python, the graph database of the four models is connected, the configuration information of the entity nodes and attribute nodes and the relationship between the nodes are read, starting from a question type to recommend visualization components, and finally automatically generating the Vega-Lite visualization grammar.

[0015] Furthermore, in step 1, the problem types are divided into four types: descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, among which: descriptive analysis includes two categories: state description and state evaluation; diagnostic analysis includes four categories: time series correlation, influencing factor comparison and screening problems, health diagnosis, and multivariate correlation; predictive analysis includes two categories: regression problems and alarm prediction; decision analysis includes optimal solution.

[0016] Furthermore, in step 1, the analysis data set model includes four parts: analysis data set definition (dataset_aq), analysis data set column (AnaColumn), data structure feature (disn_feature), and data quality feature (qulity_feature).

[0017] Furthermore, the analysis data set definition is a large table used to analyze subject-oriented data hierarchies; the analysis data set columns are descriptions of each column of data in the analysis data set table; the data set structure characteristics define the analysis data set including location, trend characteristics, and correlation information; the data set quality characteristics define the quality evaluation system of the analysis data set.

[0018] Furthermore, in step 1, the EDA component model consists of four parts: EDA related configuration definition (edasetting), EDA related configuration instantiation (such as lag_settings), EDA visualization definition (edapt) and EDA visualization instantiation (such as lag_pt).

[0019] Furthermore, in step 1, the Vega-Lite interactive grammar model includes four parts: data, mark type, encoding, and interactive parameters.

[0020] Furthermore, in step 2, the EDA data analysis starts from a specific problem under the four major categories of descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, and selects an analysis data set according to the type of problem and the research object and indicator description. The types of analysis data sets are divided into three categories: single variable, multivariate, and time series. The data types of the analysis data set columns that constitute the analysis data set are divided into three types: discrete data, continuous data, and time series data.

[0021] Furthermore, in step 3, the exploratory analysis interactive visualization grammar automatic generation algorithm is implemented in Python language. By calling the py2neo package to connect to the model's graph database, starting from a question type, the descriptive information on the entity nodes and attribute nodes in the four model graphs and the relationship information between the nodes are read respectively to obtain the recommended visualization components. Finally, based on the format requirements of the Vega-Lite grammar, a json format grammar is constructed.

[0022] The beneficial effects of the present invention are:

[0023] (1) The present invention proposes a method for realizing interactive visual exploratory analysis using Vega-Lite. The EDA model is used to realize the function of displaying the structural characteristics and quality characteristics of the analysis data set more intuitively, comprehensively and effectively based on the problem type only. At the same time, the Vega-Lite syntax is used to simply realize the complex interactive visualization function of the EDA model, completing the separation of data and visual presentation. In addition, an algorithm for automatically generating Vega-Lite syntax from the knowledge graph of the model is realized based on Python programming, which simplifies the data analysis process and improves the efficiency and intelligence level of data analysis.

[0024] (2) The present invention is based on EDA and stores all the information of the model in the knowledge graph. Through programming, it realizes the process of automatically generating an interactive visualization grammar based on a given question type, then reading the model information in the graph. This solves the cumbersome problems of the original exploratory analysis and visualization process, realizes the separation of data and visualization presentation, and the results of the interactive behavior of the visualization components can be used as input information for human-computer interaction to continue EDA data analysis, thereby improving the credibility of the EDA data analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A knowledge graph of the problem type model of the present invention;

[0026] Figure 2 The knowledge graph of the analysis data set model of the present invention;

[0027] Figure 3 A knowledge graph of the EDA component model of the present invention;

[0028] Figure 4 The knowledge graph of the Vega-Lite interactive grammar model of the present invention;

[0029] Figure 5 It is the EDA data analysis flow chart of the present invention;

[0030] Figure 6 A flowchart for automatically generating an algorithm for the exploratory analysis interactive visual grammar of the present invention;

[0031] Figure 7 (a) is a syntax diagram of the JSON format of the delay graph instance automatically generated by the program;

[0032] Figure 7 (b) is the visualization effect of the delay graph. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solution of the present application, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] This example provides a method for using Vega-Lite to implement interactive visual exploratory analysis, which includes the following steps:

[0035] Step 1: Build the knowledge graphs of the problem type model, analysis data set model, EDA component model, and Vega-Lite interactive grammar model in the form of a neo4j graph database, save the relevant information of each model in the form of entity nodes, attribute nodes, and relationships to the knowledge graph, and establish the connection relationship between the models;

[0036] In step 1, the problem types are divided into four categories: descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, among which:

[0037] Descriptive analysis includes two categories: status description and status evaluation;

[0038] Diagnostic analysis includes four categories: time series correlation, comparison of influencing factors and screening issues, health diagnosis, and multivariate correlation;

[0039] Predictive analysis includes two categories: regression problems and alarm prediction;

[0040] Decision analysis involves optimal solution.

[0041] like Figure 1 As shown in the figure, the four major types of problems and their sub-problems are instantiated as the problem type (buiq_type), and the four attributes of problem type name (issueTypeName), problem type classification (IssueTypeCal), solution type (solveType), and adaptive data set type (dsType) are added to the problem type node. The corresponding information is improved in the instantiation node, and all node information of the model is imported into the knowledge graph through the statement of the neo4j graph database.

[0042] like Figure 2 As shown, in step 1, the analysis data set model mainly consists of four parts: analysis data set definition (dataset_aq), analysis data set column (AnaColumn), data structure feature (disn_feature), and data quality feature (qulity_feature).

[0043] The analysis dataset definition is a large table used to analyze the subject-oriented data hierarchy, and adds the dataset type (datasettype) attribute and the relationship attribute related to the analysis dataset column (AnaColumn) - data columns (columns).

[0044] The analysis data set column is a description of each column of data in the analysis data set table. It adds three attributes: data column role (Character), analysis data column type (Datatype), basic physical quantity type (phy_type), and a relationship attribute related to the analysis data set (dataset_aq) - data table (table).

[0045] The structural characteristics of the data set define the information of the analyzed data set including location, trend characteristics, correlation, randomness, shape, dispersion, symmetry, outliers, frequency, etc.

[0046] The quality characteristics of the dataset define the quality evaluation system of the analytical dataset, including the analysis and processing of missing values, consistency, outliers, etc. of the dataset.

[0047] At the structural feature nodes and quality feature nodes, the relationship between the analysis dataset and the analysis dataset is established by adding the attribute of the adaptation dataset type (dsType), and then all the information of the model is imported into the knowledge graph through the statements of the neo4j graph database.

[0048] like Figure 3 As shown, in step 1, the EDA component model mainly consists of four parts: EDA related configuration definition (edasettings), EDA related configuration instantiation (such as lag_settings), EDA visualization definition (edapt) and EDA visualization instantiation (such as lag_pt);

[0049] The EDA-related configuration definition mainly includes the configuration parameters of the EDA interactive visualization image, including the number of encoding channels (channel_num), encoding channel type (edachannels), visualization type (vega_type) and other configurations related to Vega-Lite syntax;

[0050] EDA-related configuration instantiation assigns values ​​to configuration parameters related to Vega-Lite syntax in each type of interactive visualization;

[0051] EDA visualization defines the overall process based on exploratory data analysis, adding configuration parameters such as problem analysis steps, usage steps, and input data set list;

[0052] EDA visualization instantiation assigns values ​​to configuration parameters related to the data analysis process in different types of visualization diagrams.

[0053] In this embodiment, EDA-related configuration definition, EDA-related configuration instantiation, EDA visualization definition and EDA visualization instantiation are taken as four entity nodes, configuration parameters are taken as attribute nodes, and the relationship between the two nodes is set to one of "attribute relationship", "inheritance relationship", "relationship attribute relationship" and "dependency relationship". The model is imported into the knowledge graph through the statement of the neo4j graph database, and each node stores the descriptive information of the related nodes and relationships including name (name), type (typeName), description (description), etc.

[0054] like Figure 4As shown, in step 1, the Vega-Lite interactive grammar model mainly includes four modules: data, mark type, encoding, and interaction parameters. Each module contains corresponding module definition and module instantiation. The four modules are set as entity nodes respectively. The Vega-Lite grammar parameters that need to be configured for the implementation of each module are set as attribute nodes. The relationship between the two nodes is set to one of "attribute relationship", "inheritance relationship", "relationship attribute relationship", and "dependency relationship". The model is imported into the knowledge graph through the statement of the neo4j graph database.

[0055] Step 2: Use EDA to perform data analysis. Select the analysis data set based on the analysis problem type. Perform EDA data analysis based on the structural and quality characteristics of the analysis data set. Recommend the type of visualization component based on the needs of each step of the analysis. Continue to perform EDA data analysis on the results of the interaction behavior of the visualization components until an accurate conclusion is reached.

[0056] like Figure 5 As shown, the EDA data analysis described in step 2 starts from a specific problem under the four major categories of descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems. The analysis data set is selected according to the type of problem and the research object and indicator description. The types of analysis data sets can be mainly divided into three categories: single variable, multivariate, and time series. The data types of the analysis data set columns that constitute the analysis data set can be divided into three types: discrete data, continuous data, and time series data.

[0057] The characteristics of the analysis data set, including structural characteristics and quality characteristics, are analyzed separately. In the EDA component model, visualization component types are recommended according to the different structural characteristics and quality characteristics of the analysis data set. The results obtained from the interactive behavior of the visualization components can continue to perform EDA data analysis until an accurate conclusion is obtained.

[0058] Step 3: Exploratory analysis of the implementation of the interactive visualization grammar automatic generation algorithm. Based on the py2neo package in Python, the graph database of the four models is connected, the configuration information of the entity nodes and attribute nodes and the relationship between the nodes are read, starting from a question type to recommend visualization components, and finally automatically generating the Vega-Lite visualization grammar.

[0059] In step 3, the exploratory analysis interactive visualization grammar automatic generation algorithm described in step 3 is implemented by Python language, and the algorithm flow chart is as follows: Figure 6, by calling the py2neo package to connect to the graph database of the model, starting from a question type, read the descriptive information on the entity nodes and attribute nodes in the four model graphs and the relationship information between the nodes to obtain the recommended visualization components, and finally construct a json format syntax based on the format requirements of the Vega-Lite syntax according to the four parts of data (data), mark type (mark), encoding (encoding), and interaction parameters (params).

[0060] like Figure 7 (a) and Figure 7 As shown in (b), Figure 7 (a) is the syntax of the json format of the delay graph instance automatically generated by the program. Figure 7 (b) is the visualization effect of the delay graph. Figure 7 In the syntax of (a), the data module is the selected analysis data set. This data set corresponds to the time series data set in the analysis data set model in step 1. It solves the time series correlation problem under the diagnostic analysis in the problem type model in step 1. In order to analyze the structural characteristics of the analysis data set such as randomness, correlation, and outliers, a delay graph instantiation object (lag_settings) is constructed in the EDA-related configuration instantiation of the EDA component model in step 1, and the configurations are respectively related to Figure 7 (a) Parameter information related to the mark module and encoding module in the syntax, such as the number of encoding channels (channel_num) of the delay graph is 2, the encoding channel types (edachannels) are x and y respectively, the visualization type (vega_type) is point, and the interaction mode corresponds to Figure 7 (a) The "mouseover" operation under the select object in the params module of the syntax can zoom in and out the data points by moving and hovering the mouse.

[0061] Starting from the time series correlation problem, the EDA data analysis of step 2 is carried out. By calling the program in step 3, the visualization component - delay graph is recommended based on the problem type information and data set structure feature information read from the model map in step 1. Then, the descriptive information on the entity nodes and attribute nodes of the delay graph and the relationship information between nodes in the Vega-Lite interactive syntax model and the EDA component model are read. According to the format requirements of the Vega-Lite syntax, the read information is constructed into four parts: data, mark type, encoding, and interactive parameters. Figure 7 The syntax of the delay graph json format in (a) is imported into Vega Editor to achieve Figure 7Interactive visualization graph in (b) - Generation of latency graph.

[0062] Although the principles of the present invention are described in detail above in conjunction with the preferred embodiments of the present invention, those skilled in the art should understand that the above embodiments are merely explanations of the exemplary implementations of the present invention, and are not intended to limit the scope of the present invention. The details in the embodiments do not constitute limitations on the scope of the present invention, and any obvious changes such as equivalent transformations, simple replacements, etc. based on the technical solution of the present invention, without departing from the spirit and scope of the present invention, fall within the protection scope of the present invention.

Claims

1. A method for implementing interactive visual exploratory analysis using Vega-Lite, characterized in that: Step 1: Build the knowledge graphs of the problem type model, analysis data set model, EDA component model, and Vega-Lite interactive grammar model in the form of a neo4j graph database, save the relevant information of each model in the form of entity nodes, attribute nodes, and relationships to the knowledge graph, and establish the connection relationship between the models; Step 2: Use EDA to perform data analysis. Select the analysis data set based on the analysis problem type. Perform EDA data analysis based on the structural and quality characteristics of the analysis data set. Recommend the type of visualization component based on the needs of each step of analysis. Continue to perform EDA data analysis on the results of the interaction behavior of the visualization components until an accurate conclusion is obtained. Step 3: Exploratory analysis of the implementation of the interactive visualization grammar automatic generation algorithm. Based on the py2neo package in Python, the graph database of the four models is connected, the configuration information of the entity nodes and attribute nodes and the relationship between the nodes are read, starting from a question type to recommend visualization components, and finally automatically generating the Vega-Lite visualization grammar.

2. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 1, characterized in that: In step 1, the problem types are divided into four types: descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems. Among them: descriptive analysis includes two categories: state description and state evaluation; diagnostic analysis includes four categories: time series correlation, influencing factor comparison and screening problems, health diagnosis, and multivariate correlation; predictive analysis includes two categories: regression problems and alarm prediction; decision analysis includes optimal solution.

3. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 1, characterized in that: In step 1, the analysis data set model includes four parts: analysis data set definition, analysis data set columns, data structure characteristics, and data quality characteristics.

4. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 3, characterized in that: The analysis data set definition is a large table used to analyze the data hierarchy that is subject-oriented; the analysis data set column is a description of each column of data in the analysis data set table; The structural characteristics of the data set define the information about the location, trend characteristics, and correlation of the analyzed data set; Dataset quality characteristics define the quality evaluation system of the analysis dataset.

5. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 1, characterized in that: In step 1, the EDA component model consists of four parts: EDA-related configuration definition, EDA-related configuration instantiation, EDA visualization definition and EDA visualization instantiation.

6. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 1, characterized in that: In step 1, the Vega-Lite interactive grammar model includes four parts: data, tag type, encoding, and interactive parameters.

7. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 1, characterized in that: In step 2, the EDA data analysis starts from a specific problem under the four major categories of descriptive analysis problems, diagnostic analysis problems, predictive analysis problems, and decision analysis problems, and selects an analysis data set based on the type of problem and the research object and indicator description. The types of analysis data sets are divided into three categories: single variable, multivariate, and time series. The data types of the analysis data set columns that constitute the analysis data set are divided into three types: discrete data, continuous data, and time series data.

8. The method for implementing interactive visual exploratory analysis using Vega-Lite according to claim 1, characterized in that: In step 3, the exploratory analysis interactive visualization grammar automatic generation algorithm is implemented in Python language. By calling the py2neo package to connect to the model's graph database, starting from a question type, the descriptive information on the entity nodes and attribute nodes in the four model graphs and the relationship information between the nodes are read respectively to obtain the recommended visualization components. Finally, based on the format requirements of the Vega-Lite grammar, a json format grammar is constructed.

Images