Visual data cleaning script cleaning process analysis system

Abstract

The invention discloses a visual data cleaning script cleaning process analysis system. The system comprises a program executor used for detecting and storing intermediate table information and column state information in each intermediate table; the code analyzer is used for extracting input table information, output table information and functions of table information data for performing data conversion operation on the intermediate table; the data conversion inference device is used for determining data conversion operation types and parameters of the table information data and finally generating data conversion operation description information; the cleaning layout constructor is used for matching intermediate table information to each node in the directed acyclic graph framework to obtain an intermediate table directed acyclic graph; and the graph icon drawing device is used for distributing the data conversion operation graph to the nodes to form edges of the directed acyclic graph so as to obtain a data cleaning process visualization graph. The system can display the change of multiple tables in the data cleaning script and visualize complex data conversion operation.

Description

technical field [0001] The invention belongs to the field of program visualization, and relates to a visual data cleaning script cleaning process analysis system. Background technique [0002] Data Wrangling is a process of organizing complex and messy data into an ideal data format through cleaning and transformation operations. It is an important pre-step for tasks such as data access, data modeling, and data visual analysis. Two-dimensional data table is an effective means of organizing data. People widely use various tables in communication, scientific research and data analysis activities. Because the original table often contains "dirty" data, or the data format, content, etc. do not meet the expected goals, data workers must clean the table. [0003] Data cleaning involves various data transformation operations, such as removing duplicate rows, filling missing values, splitting composite columns, etc. Using programming languages ​​such as R and Python to write speci...

AI Technical Summary

Problems solved by technology

There are many works aimed at visualizing data lineage. For example, TACO is a visual comparison tool that can show the difference of tabular data over time, but this tool cannot visualize the cleaning process of data cleaning scripts completed in programming languages; Datamations can use animated Formally explain the steps in the data cleaning process, but the tool can only reveal the data changes of one table, and cannot display the changes of multiple tables at the same time

In addition, the above tools can only visualize a small part of data transformation operations, but cannot support complex operations such as transformations involving table structures (such as Pivot, Unpivot, etc.)

Images