A constraint-based method for visualizing the input and output space of data organization scripts

Abstract

The present invention discloses a constraint-based data organization script input and output space visualization method, comprising the following steps: using a data table constraint generator to generate the constraint space of the input and output table of the data organization script; visualizing the constraint space of the input and output table; supporting modification of the constraint space of the input and output table to align with the original data or the requirements of downstream tasks, and after modifying the constraint space, calling the data table constraint generator again to generate a new constraint space of the input and output table; detecting conflicts between constraints and conflicts between data and constraints, and locating conflicts, thereby improving user understanding and script debugging efficiency, and helping users discover hidden and difficult-to-find errors through constraints, thereby improving the robustness of the script.

Description

Technical Field

[0001] The present invention belongs to the technical field of data visualization, program visualization and human-computer interaction, and particularly relates to a constraint-based data organization script input and output space visualization method. Background Art

[0002] In the era of big data, massive amounts of data are generated daily. This raw data is often disorganized and chaotic in content, requiring conversion and cleansing to meet the needs of downstream tasks such as visualization analysis, machine learning, and data mining. Data wrangling is the process of converting data into the format and content expected by downstream tasks. Users often need to wrangle raw data based on task requirements. Data workers typically accomplish this task by writing scripts. Commonly used scripting languages ​​include Python and R, along with corresponding libraries such as Pandas and Tidyverse.

[0003] To improve the efficiency of data wrangling tasks, data workers often reuse existing data wrangling scripts. These scripts may be written by themselves or obtained from others in code repositories or technical blogs. Furthermore, many data workers choose to provide their requirements to a large language model, which then generates the corresponding data wrangling scripts. Before reusing a script, data workers must first understand the selected script and determine whether it can handle the data being wrangled and whether the script's output meets the requirements of downstream tasks.

[0004] Although some data wrangling scripts come with a small amount of executable data in code repositories or technical blogs, even scripts in the same field and performing similar tasks cannot guarantee that the current scripts will be applicable to new data. This is because each piece of data may have unique problems, and data workers usually only consider specific problems in a single data set or sample data when writing scripts, making it difficult to fully cover all possible problems. For scripts without executable data, such as those that cannot be uploaded and shared online due to privacy issues, or scripts generated by large language models, users need to read the script line by line, understand its data conversion operations, and construct the corresponding input and output. This process requires users to have an in-depth understanding of the script's logic and conversion operations to ensure the accuracy and effectiveness of data wrangling.

[0005] Existing methods for helping users understand data wrangling scripts fall into two main categories: one is to assist data workers in debugging the script by providing statistical information about the script's input and output, or by providing execution status; the other is to help data workers understand the results of each step by visualizing the changes to the data caused by each data transformation operation. However, both methods require executable data, which is essential for understanding the script. Furthermore, these methods fail to display the complete input and output space of the script—that is, they fail to clearly define the boundary conditions under which the script can process the input data, nor can they clearly define whether the generated data fully meets the requirements of downstream tasks. Data workers who have specific requirements for the script's input and output are more interested in the input and output of the data wrangling script, rather than the specific data transformation operations performed in the script. As a result, data workers often spend a considerable amount of time constructing the script's input and output space. Effectively helping data workers understand the input and output space of a script has become a challenging task. Summary of the Invention

[0006] To address these technical challenges, the present invention provides a constraint-based method for visualizing the input and output space of data wrangling scripts. This method constructs input and output table constraints from data wrangling scripts using a rule-based model. Combining visualization and human-computer interaction techniques, this method helps users understand the script's input and output space, supports user adjustments based on requirements, and guides users in identifying conflicts between requirements and the script, allowing them to debug the script, thereby enhancing the script's robustness.

[0007] To achieve the above-mentioned object of the invention, an embodiment provides a constraint-based data sorting script input and output space visualization method, comprising the following steps:

[0008] Use the data table constraint generator to generate the constraint space of the input and output tables of the data sorting script;

[0009] Visualize the constraint space of input and output tables;

[0010] Supports modifying the constraint space of the input and output tables to align with the original data or the requirements of downstream tasks. After modifying the constraint space, the data table constraint generator will be called again to generate a new constraint space for the input and output tables.

[0011] Detect conflicts between constraints, conflicts between data and constraints, and locate conflicts.

[0012] In the method of the present invention, the data table constraint generator includes an input table space constraint construction module, an output table space constraint construction module, a constraint back propagation module, and a constraint back construction module, and generates the constraint space of the input and output tables based on these four modules;

[0013] The input table space constraint building module determines the constraints imposed on the input table by analyzing each data conversion operation in the data arrangement script;

[0014] The output table space constraint construction module transfers the constraints generated by the input table to the output table through the content of the data conversion operation;

[0015] When the input table space constraint construction module updates the constraints of a column of an original column in a non-input table and needs to update the constraint space of the generated column, the constraint backpropagation module locates the backpropagation path from the generated column to the original column, and backpropagates along the constraint propagation path until it stops at the original column, and then starts forward propagation from the end point;

[0016] After locating the reverse propagation path, the constraint reverse construction module needs to propagate the constraints in the output table to the input table. Specifically, according to the content of the data conversion operation, the constraint content is transferred to the input table to make the space of the input table more clear.

[0017] In the method of the present invention, the constraint space of the input and output tables is visualized, including visualization constraint icons, constraint labels, and data range multi-layer bar charts, wherein the constraint icons are used to represent data types, data sorting, missing values ​​in the data, and duplicate values ​​in the data; the data labels represent all constraint types; and the data range multi-layer bar chart is used to represent the data range.

[0018] In the method of the present invention, the constraint types include data type, data range, data format, data sorting, special values ​​within the data, repeated values ​​within the data, missing values ​​within the data, comparative relationships between data, derived relationships between data, and joint relationships between data.

[0019] In the method of the present invention, the constraint labels and the data range multi-layer bar chart use a unified color coding to indicate the data range of the constraint, wherein color one indicates that all data must satisfy the constraint, color two indicates that at least one data needs to satisfy the constraint; color three indicates that optional data satisfies the constraint; and color four indicates that no data satisfies the constraint.

[0020] In the method of the present invention, the conflict between the constraints occurs after the user modifies the constraint space of the input and output table, and conflicts with the constraints generated by the script. The conflict between the data and the constraints occurs after the user uploads a data table that requires the execution of the data sorting script, and detects the conflict between the data table and the constraints.

[0021] The method of the present invention further comprises: generating sample data that meets the constraints and is semantically rich according to the constraint space of the input and output tables.

[0022] The method of the present invention further includes: constructing the derivation of the data columns in the script according to the data conversion operations in the data sorting script and the constraint space of the input and output tables, and generating a data column derivation visualization view.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] This paper uses a constraint-based approach to represent the input and output table space of a data wrangling script. This helps users understand whether a data wrangling script can process the current data and whether it can convert the original data into the expected data that meets downstream requirements. It also supports users in adjusting the constraint space to align with task requirements, helping them modify and debug scripts. Experimental findings show that this method improves user understanding and script debugging efficiency compared to existing script debugging tools. Furthermore, constraints can help users discover hidden, difficult-to-find errors, improving the robustness of scripts. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a constraint-based data organization script input and output space visualization method provided by an embodiment;

[0027] Figure 2 This is a visualization diagram of the input and output space of the data sorting script provided in the embodiment;

[0028] Figure 3 Schematic diagram of the structure of the constraint generator of the data sorting script provided in the embodiment;

[0029] Figure 4 is a schematic diagram of constraint visualization elements provided by an embodiment;

[0030] Figure 5 It is a visualization flow chart of the input and output space of the data sorting script provided in the embodiment;

[0031] Figure 6 is a constraint modification flow chart provided by an embodiment;

[0032] Figure 7 This is a schematic diagram of conflict detection and positioning provided by an embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0034] The inventive concept of the present invention is to propose a constraint-based method for visualizing the input and output space of a data organization script. This method utilizes a rule-based model to construct constraints for the input and output tables from the data organization script and express the input and output space through these constraints. Through visualization and human-computer interaction techniques, the method helps users understand the boundary conditions of the input and output of the data organization script. In addition, the method allows users to interactively modify the input and output space to align the user's requirements for the script input and output. The method can also detect whether the script can process the current data through constraints, identify conflicts between user requirements and the script, and guide the user to modify and debug these conflicts, thereby enhancing the robustness of the script.

[0035] like Figure 1 As shown, the constraint-based data sorting script input and output space visualization method provided by the embodiment includes:

[0036] Step 1: Upload the data sorting script. Users can choose to click Figure 2 Upload the script file by clicking the Upload Script button in the upper right corner of the view shown in Figure A, or directly copy the script into the script editor in the view, as shown in Figure A. Figure 2 As shown in A2.

[0037] Step 2: Data sorting script input and output constraint construction. Users can click Figure 2 Click the execute script button in view A in the present invention to call the data table constraint generator to generate the constraint space of the current data sorting script.

[0038] The data table constraint generator can generate the constraint space of the script's input and output tables by parsing the functions and parameters in the data sorting script. There are ten types of constraints, namely: data type, data range, data format, data sorting, special values ​​in the data, repeated values ​​in the data, missing values ​​in the data, comparison relationships between data, derived relationships between data, and joint relationships between data. Constraints are divided into four scopes according to the amount of data they act on, namely: all data must be satisfied (All), at least one piece of data must be satisfied (Exist), optional data must be satisfied (Optional), and no data must be satisfied (Not). Figure 3 As shown, the data table constraint generator includes four modules:

[0039] The input table space constraint building module analyzes each data conversion operation in the data sorting script to determine the constraints that these data conversion operations impose on the input table, such as Figure 3 As shown in A, if the current operation is to replace the data in the Page column that matches the regular expression with empty data, the constraints of the input table may require that all data in the Page column be character type and at least one data item can match the regular expression.

[0040] Output tablespace constraint building blocks, such as Figure 3 As shown in B, the constraints generated by the input table are transferred to the output table through the content of the data conversion operation, such as Figure 3 As shown in B1, this data conversion step requires that all data types in the article_id column in the output table be character, and that the column contain no data matching the regular expression. The constraints of the Page column are also passed to the output table.

[0041] Constrained back-propagation modules, such as Figure 3 As shown in Figure C, in the process of constructing the constraint space of the input and output tables, the constraint space of the input and output tables will become clearer. When the input table space constraint construction module updates the constraint of a column that is not an original column in the input table (for example, the article_id column is generated from the Page column in the input table) and needs to update the constraint space of the generated column, the constraint backpropagation module locates the backpropagation path from the generated column to the original column, and backpropagates along this constraint propagation path until it stops at the original column, and then starts forward propagation from the end point, as shown in Figure 4. Figure 3 The serial numbers are 1 to 4.

[0042] After locating the reverse propagation path, the constraint reverse construction module needs to propagate the constraints in the output table to the input table. Specifically, according to the content of the data conversion operation, the constraint content is passed to the input table to make the space of the input table more clear. Figure 3 As shown in Figure D, if a data format constraint is added to the article_id column, it is necessary to determine whether it can be passed to the Page column of the input table.

[0043] Step 3: Data column derivation visualization. Based on the relevant constraint information generated in step 2, the current step will construct the derivation of the data column in the script and generate a data column derivation visualization view, such as Figure 3 A1 and Figure 5 As shown in the figure, this visualization helps users understand which data columns in the output table are derived from the data columns in the data table after the data conversion operation.

[0044] Step 4: Constraint visualization. This involves visualizing the constraint space of the input and output tables, including visualization of Figure 3 China A and Figure 2 The constraint icon shown in E, such as Figure 3 Middle B and Figure 2 The constraint label shown in D, and Figure 3 The data range multi-layer bar chart shown in Figure C, where the constraint icons are used to represent data types, data sorting, missing values ​​within the data, and duplicate values ​​within the data; the data labels represent all constraint types; and the data range multi-layer bar chart is used to represent the data range.

[0045] like Figure 3 As shown, the constraint labels and data range multi-layer bar chart uses a unified color coding to indicate the data range of the constraint, where color one (e.g., green) indicates that all data must satisfy the constraint, color two (e.g., blue) indicates that at least one data needs to satisfy the constraint; color three (e.g., purple) indicates that optional data satisfies the constraint; color four (e.g., orange) indicates that no data satisfies the constraint.

[0046] Specifically, based on the constraint space information in step 2, the constraint view will be displayed according to the type of constraint, such as Figure 2 As shown in B, the corresponding visualization elements are generated, such as Figure 2 B1 and Figure 5 In the constraint view, by clicking on the column name of each column, the detailed view will be displayed, as shown in the following example: Figure 2 As shown in C, all constraint information in the column is displayed. In the detailed view, all constraint information is expressed by constraint labels. The constraint labels in the detailed view support click interaction by users. After clicking, detailed constraint information will be expanded, including five parts: constraint description, constraint example, conflict data (such as uploaded data conflicts with the constraint), constraint modification and constraint operation.

[0047] Step 5: Sample data generation. Based on the constraint space of step 2, the current step will generate sample data that meets the constraints and is semantically rich, such as Figure 2 B2 and Figure 5 This helps users gain a preliminary understanding of the constraint space of the current column and facilitate further exploration.

[0048] Step 6: Constraint modification. Users can adjust the constraints in the detailed view by modifying the constraint panel to align with the task requirements. Figure 6 As shown, after the user modifies the constraint content, he clicks the submit button in the constraint operation panel in the detailed view, and the present invention will call the constraint generator to recalculate the constraint space of the data sorting script input and output according to the modified content.

[0049] Step 7, data upload. Users can click the Upload Data button to upload the data that needs to be executed by the script, such as Figure 3Click the button in the upper left corner of the B view to determine whether the current data can be converted by the current script into data that meets the requirements of downstream tasks.

[0050] Step 8, conflict detection. After step 7 is completed, the present invention will detect the conflict between the data and the constraint space, such as Figure 7 As shown, the conflicting constraints and conflicting data will be displayed.

[0051] It should also be noted that this also includes conflict detection between constraints. Conflicts between constraints usually occur when users modify the constraint space of the input and output tables, which conflicts with the constraints generated by the script. Prompting conflicts can help data workers find mismatches between data and requirements. At the same time, this method can help users understand the causes of two types of conflicts, that is, what constraints have caused the conflict, such as Figure 7 As shown in .

[0052] Step 9: Conflict location: After step 8 is completed, the location of the conflict in the script, that is, the conflicting code fragment, is located, which can help the script to be modified to address the conflict issue.

[0053] Specifically, the user can click on any conflicting constraint, and the present invention will highlight the specific script content that causes the conflict, such as Figure 7 Help users to modify and debug the corresponding script content.

[0054] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A constraint-based data organization script input and output space visualization method, characterized in that: The following steps are involved: Use the data table constraint generator to generate the constraint space of the input and output tables of the data sorting script; Visualize the constraint space of input and output tables; Supports modifying the constraint space of the input and output tables to align with the original data or the requirements of downstream tasks. After modifying the constraint space, the data table constraint generator will be called again to generate a new constraint space for the input and output tables. Detect conflicts between constraints, conflicts between data and constraints, and locate conflicts; The data table constraint generator includes an input table space constraint construction module, an output table space constraint construction module, a constraint back propagation module, and a constraint reverse construction module, and generates the constraint space of the input and output tables based on these four modules; The input table space constraint building module determines the constraints imposed on the input table by analyzing each data conversion operation in the data arrangement script; The output table space constraint construction module transfers the constraints generated by the input table to the output table through the content of the data conversion operation; When the input table space constraint construction module updates the constraints of a column of an original column in a non-input table and needs to update the constraint space of the generated column, the constraint backpropagation module locates the backpropagation path from the generated column to the original column, and backpropagates along the constraint propagation path until it stops at the original column, and then starts forward propagation from the end point; After locating the reverse propagation path, the constraint reverse construction module needs to propagate the constraints in the output table to the input table. Specifically, according to the content of the data conversion operation, the constraint content is transferred to the input table to make the space of the input table more clear.

2. The constraint-based data organization script input and output space visualization method according to claim 1 is characterized in that: Visualize the constraint space of the input and output tables, including visual constraint icons, constraint labels, and data range multi-layer bar charts. Constraint icons are used to indicate data types, data sorting, missing values ​​within the data, and duplicate values ​​within the data; data labels indicate all constraint types; and data range multi-layer bar charts are used to indicate data ranges.

3. The constraint-based data organization script input and output space visualization method according to claim 2, characterized in that: The constraint types include data type, data range, data format, data sorting, special values ​​within data, repeated values ​​within data, missing values ​​within data, comparison relationships between data, derived relationships between data, and joint relationships between data.

4. The constraint-based data organization script input and output space visualization method according to claim 2, characterized in that: Constraint Labels and Data Ranges Multi-layer bar charts use a unified color coding to indicate the data range of the constraint. Color 1 indicates that all data must satisfy the constraint, color 2 indicates that at least one data needs to satisfy the constraint, color 3 indicates that optional data must satisfy the constraint, and color 4 indicates that no data satisfies the constraint.

5. The constraint-based data organization script input and output space visualization method according to claim 1, characterized in that: The conflict between the constraints occurs when the user modifies the constraint space of the input and output tables, which conflicts with the constraints generated by the script. The conflict between the data and the constraints occurs when the user uploads a data table that requires the execution of the data sorting script, and detects the conflict between the data table and the constraints.

6. The constraint-based data sorting script input and output space visualization method according to claim 1, characterized in that: Also includes: Generate sample data that meets the constraints and is semantically rich based on the constraint space of the input and output tables.

7. The constraint-based data organization script input and output space visualization method according to claim 1, characterized in that: Also includes: Based on the data conversion operations in the data wrangling script and the constraint space of the input and output tables, the derivation of the data columns in the script is constructed, and a data column derivation visualization view is generated.

Images