Constraint-based data arrangement script input and output space visualization method

Abstract

The invention discloses a constraint-based data arrangement script input and output space visualization method. The method comprises the following steps of generating a constraint space of an input and output table of a data arrangement script by utilizing a data table constraint generator; the constraint space of the input and output table is visualized; the constraint space of the input and output table is supported to be modified so as to align the requirements of original data or downstream tasks, and after the constraint space is modified, the data table constraint generator is called again to generate a new constraint space of the input and output table; according to the method, the conflict between the constraints and the conflict between the data and the constraints are detected, and conflict positioning is carried out, so that the efficiency of understanding and debugging the script by the user can be improved, the user can be helped to discover hidden and difficult-to-discover errors through the constraints, and the robustness of the script is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data visualization, program visualization and human-computer interaction, and specifically relates to a constraint-based data sorting script input and output space visualization method. Background Art

[0002] In the era of big data, massive amounts of data are generated every day. These raw data are usually in a disorganized format and have chaotic data content. They need to be converted and cleaned to meet the needs of downstream tasks such as visual analysis, machine learning, and data mining. Data wrangling is the process of converting data into the expected format and content of downstream tasks. Users often need to perform data wrangling tasks on raw data according to task requirements. Data workers usually complete data wrangling tasks by writing scripts. Commonly used scripting languages ​​include Python and R, with corresponding function libraries such as Pandas and Tidyverse.

[0003] In order to improve the efficiency of completing data sorting tasks, data workers usually reuse existing data sorting scripts. These data sorting scripts may come from scripts written by themselves in the past, or scripts written by others obtained from code repositories or technical blogs. In addition, many existing data workers will choose to provide their own requirements to the large language model and let it generate the corresponding data sorting scripts. Before reusing the script, the data worker first needs to understand the selected script, determine whether it can process the data that needs to be sorted, and whether the output of the script can meet the needs of downstream tasks.

[0004] Although some data wrangling scripts come with a small amount of executable data in code repositories or technical blogs, even for scripts in the same field and similar tasks, it is difficult to guarantee that the current scripts can be applied to new data. This is because each piece of data may have unique problems, and data workers usually only consider specific problems in a single data or sample data when writing scripts, making it difficult to fully cover all possible problems. For scripts without executable data, such as those that cannot be uploaded and shared on the Internet due to privacy issues, or scripts generated by large language models, users need to read the script line by line, understand its data conversion operations, and construct the corresponding input and output. This process requires users to have an in-depth understanding of the logic and conversion operations of the script to ensure the accuracy and effectiveness of data wrangling.

[0005] Existing methods to help users understand data wrangling scripts can be divided into two categories: one is to help data workers debug scripts by providing statistical information on the input and output of data wrangling scripts, or the execution status; the other is to help data workers understand the execution results of each step of the operation by visualizing the changes caused by each step of data conversion operation. However, both methods require executable data, and the script can only be understood with the help of the content of this data. In addition, these methods cannot show the complete input and output space of the script, that is, they cannot clearly define the boundary conditions of the input data that the script can process, and whether the generated data can fully meet the needs of downstream tasks. For data workers who have clear requirements for the input and output of the script, they are more concerned about the input and output of the data wrangling script, rather than the specific data conversion operations performed in the script. Therefore, data workers often need to spend a lot of time to build the input and output space of the script. How to efficiently help data workers understand the input and output space of a script has become a challenging task. Summary of the invention

[0006] To solve the above technical challenges, the purpose of the present invention is to provide a constraint-based data sorting script input and output space visualization method. The method constructs the constraints of the input and output tables from the data sorting script through a rule-based model, and combines visualization and human-computer interaction technology to help users understand the input and output space of the script, support users to adjust the input and output space according to needs, and guide users to find conflicts between needs and scripts and debug the scripts, thereby enhancing the robustness of the script.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a constraint-based data sorting script input and output space visualization method, comprising the following steps:

[0008] Use the data table constraint generator to generate the constraint space of the input and output tables of the data sorting script;

[0009] Visualize the constraint space of input and output tables;

[0010] Supports modification of the constraint space of the input and output tables to align with the original data or the requirements of downstream tasks. After modifying the constraint space, the data table constraint generator will be called again to generate a new constraint space for the input and output tables.

[0011] Detect conflicts between constraints, conflicts between data and constraints, and locate conflicts.

[0012] In the method of the present invention, the data table constraint generator includes an input table space constraint construction module, an output table space constraint construction module, a constraint back propagation module, and a constraint back construction module, and generates the constraint space of the input and output tables based on these four modules;

[0013] The input table space constraint building module determines the constraints on the input table caused by these data conversion operations by analyzing each data conversion operation in the data sorting script;

[0014] The output table space constraint construction module transfers the constraints generated by the input table to the output table through the content of the data conversion operation;

[0015] When the input table space constraint construction module updates the constraint of an original column in a non-input table and needs to update the constraint space of the generated column, the constraint back-propagation module locates the back-propagation path from the generated column to the original column, and back-propagates along the constraint propagation path until it stops at the original column, and then starts forward propagation from the end point;

[0016] After locating the reverse propagation path, the constraint reverse construction module needs to propagate the constraints in the output table to the input table. Specifically, according to the content of the data conversion operation, the constraint content is transferred to the input table to make the space of the input table clearer.

[0017] In the method of the present invention, the constraint space of the input and output tables is visualized, including visualized constraint icons, constraint labels, and data range multi-layer bar charts, wherein the constraint icons are used to represent data types, data sorting, missing values ​​in the data, and duplicate values ​​in the data; the data labels represent all constraint types; and the data range multi-layer bar charts are used to represent the data range.

[0018] In the method of the present invention, the constraint types include data type, data range, data format, data sorting, special values ​​within the data, repeated values ​​within the data, missing values ​​within the data, comparative relationships between data, derived relationships between data, and joint relationships between data.

[0019] In the method of the present invention, the constraint labels and the data range multi-layer bar chart use a unified color coding to indicate the data range of the constraint, wherein color one indicates that all data must satisfy the constraint, color two indicates that at least one data needs to satisfy the constraint; color three indicates that optional data satisfies the constraint; and color four indicates that no data satisfies the constraint.

[0020] In the method of the present invention, the conflict between the constraints occurs after the user modifies the constraint space of the input and output tables, which conflicts with the constraints generated by the script. The conflict between the data and the constraints occurs after the user uploads a data table that requires the execution of a data sorting script, and the conflict between the data table and the constraints is detected.

[0021] The method of the present invention also includes: generating sample data that meets the constraints and is semantically rich according to the constraint space of the input and output tables.

[0022] The method of the present invention also includes: constructing the derivation of the data columns in the script according to the data conversion operations in the data sorting script and the constraint space of the input and output tables, and generating a data column derivation visualization view.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention uses a constraint-based method to represent the space of the input and output tables of the data sorting script, which can help users understand whether a data sorting script can process the current data and whether the original data can be converted into expected data that meets any downstream requirements. It supports users to adjust the constraint space to align with task requirements and helps users modify and debug scripts. Based on experimental findings, this method can improve the efficiency of user understanding and debugging scripts compared to existing script debugging tools. And it can help users find hidden and difficult-to-find errors through constraints, thereby improving the robustness of the script. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a constraint-based data sorting script input and output space visualization method provided in an embodiment;

[0027] Figure 2 It is a visualization diagram of the input and output space of the data sorting script provided in the embodiment;

[0028] Figure 3 It is a structural diagram of a constraint generator of a data arrangement script provided in an embodiment;

[0029] Figure 4 is a schematic diagram of constraint visualization elements provided by an embodiment;

[0030] Figure 5 It is a visualization flow chart of the input and output space of the data sorting script provided in the embodiment;

[0031] Figure 6 is a constraint modification flow chart provided by an embodiment;

[0032] Figure 7 It is a schematic diagram of conflict detection and positioning provided by an embodiment. DETAILED DESCRIPTION

[0033] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0034] The inventive concept of the present invention is: a constraint-based data sorting script input and output space visualization method is proposed. The method uses a rule-based model to construct the constraints of the input and output tables from the data sorting script, and expresses the input and output space through these constraints. Through visualization and human-computer interaction technology, the user is helped to understand the boundary conditions of the input and output of the data sorting script. In addition, the method allows the user to modify the input and output space through interaction to align the user's requirements for the script input and output. The method can also detect whether the script can process the current data through constraints, and find conflicts between user requirements and the script, and guide the user to modify and debug these conflicts, thereby enhancing the robustness of the script.

[0035] like Figure 1 As shown, the constraint-based data sorting script input and output space visualization method provided in the embodiment includes:

[0036] Step 1: Upload the data sorting script. Users can choose to click Figure 2 Upload the script file by clicking the Upload Script button in the upper right corner of the view shown in A, or directly copy the script to the script editor in the view, as shown in Figure 2 As shown in A2.

[0037] Step 2: Data cleaning script input and output constraint construction. Users can click Figure 2 By clicking the execute script button in the A view, the present invention will call the data table constraint generator to generate the constraint space of the current data sorting script.

[0038] The data table constraint generator can generate the constraint space of the script's input and output tables by parsing the functions and parameters in the data sorting script. There are ten types of constraints, namely: data type, data range, data format, data sorting, special values ​​in the data, duplicate values ​​in the data, missing values ​​in the data, comparison relationships between data, derived relationships between data, and joint relationships between data. Constraints are divided into four scopes according to the amount of data they act on, namely: all data must satisfy (All), at least one piece of data must satisfy (Exist), optional data must satisfy (Optional), and no data must satisfy (Not). Figure 3 As shown, the data table constraint generator includes four modules, namely:

[0039] The input table space constraint building module analyzes each data conversion operation in the data sorting script to determine the constraints that these data conversion operations impose on the input table, such as Figure 3 As shown in A, if the current operation is to replace the data in the Page column that matches the regular expression with empty, the constraints of the input table may require that all data in the Page column be character type and at least one piece of data can match the regular expression.

[0040] Output tablespace constraint building blocks, such as Figure 3 As shown in B, the constraints generated by the input table are transferred to the output table through the content of the data conversion operation, such as Figure 3 As shown in B1, this step of data conversion operation requires that all data types of the article_id column in the output table are character type, and the column does not contain data matching the regular expression, etc. At the same time, the constraints of the Page column will also be passed to the output table.

[0041] Constrained back-propagation modules, such as Figure 3 As shown in Figure C, in the process of constructing the constraint space of the input and output tables, the constraint space of the input and output tables will become clearer. When the input table space constraint construction module updates the constraints of a column that is not an original column in the input table (for example, the article_id column is generated by the Page column in the input table) and needs to update the constraint space of the generated column, the constraint backpropagation module locates the backpropagation path from the generated column to the original column, and backpropagates along this constraint propagation path until it stops at the original column, and then starts forward propagation from the end point, as shown in Figure 1. Figure 3 Serial numbers 1 to 4.

[0042] After locating the reverse propagation path, the constraint reverse construction module needs to propagate the constraints in the output table to the input table. Specifically, according to the content of the data conversion operation, the constraint content is passed to the input table to make the space of the input table more clear. Figure 3 As shown in D, if a new data format constraint is added to the article_id column, it is necessary to determine whether it can be passed to the Page column of the input table.

[0043] Step 3: Data column derivation visualization. Based on the relevant constraint information generated in step 2, the current step will build the derivation of the data column in the script and generate a data column derivation visualization view, such as Figure 3 A1 and Figure 5 As shown in the figure, this visualization helps users understand which data columns in the output table are derived from the data columns in the data table after the data conversion operation.

[0044] Step 4: Constraint visualization. That is, visualize the constraint space of the input and output tables, including visualization such as Figure 3 Middle A and Figure 2 The constraint icon shown in E, such as Figure 3 Medium B and Figure 2 The constraint label shown in D, and Figure 3 The data range multi-layer bar chart shown in C, where the constraint icons are used to represent data types, data sorting, missing values ​​in the data, and duplicate values ​​in the data; the data labels represent all constraint types; and the data range multi-layer bar chart is used to represent the data range.

[0045] like Figure 3 As shown, the constraint labels and data range multi-layer bar charts use uniform color coding to indicate the data range of the constraints, where color one (e.g., green) indicates that all data must satisfy the constraint, color two (e.g., blue) indicates that at least one piece of data needs to satisfy the constraint; color three (e.g., purple) indicates that optional data satisfies the constraint; color four (e.g., orange) indicates that no data satisfies the constraint.

[0046] Specifically, based on the constraint space information in step 2, the constraint view will be displayed according to the type of constraint, such as Figure 2 As shown in B, the corresponding visualization elements are generated, such as Figure 2 B1 and Figure 5 In the constraint view, by clicking on the column name of each column, the detailed view will be displayed, such as Figure 2 As shown in C, all constraint information in the column is displayed. In the detailed view, all constraint information is expressed by constraint labels. The constraint labels in the detailed view support click interaction by users. After clicking, detailed constraint information will be expanded, including five parts: constraint description, constraint example, conflict data (such as the uploaded data conflicts with the constraint), constraint modification and constraint operation.

[0047] Step 5: Sample data generation. Based on the constraint space of step 2, the current step will generate sample data that meets the constraints and is semantically rich, such as Figure 2 B2 and Figure 5 This helps users gain a preliminary understanding of the constraint space of the current column and conduct further exploration.

[0048] Step 6: Constraint modification. In the detailed view, users can modify the constraints in the constraint panel to align with task requirements. Figure 6 As shown, after the user modifies the constraint content, he clicks the submit button in the constraint operation panel in the detailed view, and the present invention will call the constraint generator to recalculate the constraint space of the data sorting script input and output according to the modified content.

[0049] Step 7, data upload. Users can click the Upload Data button to upload the data that needs to be executed by the script, such as Figure 3Click the button in the upper left corner of the B view. This will determine whether the current data can be converted by the current script into data that meets the requirements of downstream tasks.

[0050] Step 8, conflict detection. After step 7 is completed, the present invention will detect conflicts between data and constraint space, such as Figure 7 As shown, the conflicting constraints and conflicting data will be displayed.

[0051] It should also be noted that it also includes conflict detection between constraints. Conflicts between constraints usually occur when users modify the input and output table constraint space, which conflicts with the constraints generated by the script. Prompting conflicts can help data workers find mismatches between data and requirements. At the same time, this method can help users understand the causes of two types of conflicts, that is, what constraints have caused the conflict, such as Figure 7 as shown in .

[0052] Step 9, conflict location: After the detection in step 8 is completed, the location where the conflict occurs in the script, that is, the code fragment that causes the conflict, can help the script to be modified according to the conflict problem.

[0053] Specifically, the user can click on any conflicting constraint, and the present invention will highlight the specific script content that causes the conflict, such as Figure 7 As shown in the figure, it helps users to modify and debug the corresponding script content.

[0054] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A constraint-based data sorting script input and output space visualization method, characterized in that: The following steps are involved: Use the data table constraint generator to generate the constraint space of the input and output tables of the data sorting script; Visualize the constraint space of input and output tables; Supports modification of the constraint space of the input and output tables to align with the original data or the requirements of downstream tasks. After modifying the constraint space, the data table constraint generator will be called again to generate a new constraint space for the input and output tables. Detect conflicts between constraints, conflicts between data and constraints, and locate conflicts.

2. The constraint-based data sorting script input-output space visualization method according to claim 1 is characterized in that: The data table constraint generator includes an input table space constraint construction module, an output table space constraint construction module, a constraint back propagation module, and a constraint back construction module, and generates the constraint space of the input and output tables based on these four modules; The input table space constraint building module determines the constraints on the input table caused by these data conversion operations by analyzing each data conversion operation in the data sorting script; The output table space constraint construction module transfers the constraints generated by the input table to the output table through the content of the data conversion operation; When the input table space constraint construction module updates the constraint of an original column in a non-input table and needs to update the constraint space of the generated column, the constraint back-propagation module locates the back-propagation path from the generated column to the original column, and back-propagates along the constraint propagation path until it stops at the original column, and then starts forward propagation from the end point; After locating the reverse propagation path, the constraint reverse construction module needs to propagate the constraints in the output table to the input table. Specifically, according to the content of the data conversion operation, the constraint content is transferred to the input table to make the space of the input table clearer.

3. The constraint-based data sorting script input-output space visualization method according to claim 1, characterized in that: The constraint space of the input and output tables is visualized, including visual constraint icons, constraint labels, and data range multi-layer bar charts. The constraint icons are used to indicate data types, data sorting, missing values ​​in the data, and duplicate values ​​in the data; the data labels indicate all constraint types; and the data range multi-layer bar charts are used to indicate data ranges.

4. The constraint-based data sorting script input-output space visualization method according to claim 3 is characterized in that: The constraint types include data type, data range, data format, data sorting, special values ​​within the data, repeated values ​​within the data, missing values ​​within the data, comparison relationships between data, derived relationships between data, and joint relationships between data.

5. The constraint-based data sorting script input-output space visualization method according to claim 3 is characterized in that: Constraint labels and data ranges Multi-layer bar charts use uniform color coding to indicate the data range of constraints. Color one indicates that all data must satisfy the constraint, color two indicates that at least one data needs to satisfy the constraint; color three indicates that optional data satisfies the constraint; and color four indicates that no data satisfies the constraint.

6. The constraint-based data sorting script input-output space visualization method according to claim 1, characterized in that: The conflict between constraints occurs when the user modifies the constraint space of the input and output tables, which conflicts with the constraints generated by the script. The conflict between data and constraints occurs when the user uploads a data table that requires the execution of a data sorting script, and detects the conflict between the data table and the constraints.

7. The constraint-based data sorting script input-output space visualization method according to claim 1, characterized in that: Also includes: Generate sample data that meets the constraints and is semantically rich according to the constraint space of the input and output tables.

8. The constraint-based data sorting script input-output space visualization method according to claim 1, characterized in that: Also includes: According to the data conversion operations in the data sorting script and the constraint space of the input and output tables, the derivation of the data columns in the script is constructed, and a data column derivation visualization view is generated.

Images