Generation of reusable data processing programs

Abstract

The method involves developing a reusable data processing program that includes a set of data transformation steps, by displaying a set of records, allowing the user to select one or more data transformation steps, applying the data transformation steps to the records, and displaying the transformed records.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 472,445, filed on June 12, 2023, the contents of which are hereby incorporated in their entirety.

[0002] The present invention relates to the generation of reusable data processing programs based on user operations on tabular representations of data.

Background Art

[0003] Complex calculations may be represented as data flows using directed graphs, where the components of the calculation are associated with the vertices of the graph and the data flows between components correspond to the links (arcs, edges) of the graph. In some cases, the calculations associated with components may be described in a human - readable format called "rules". Rules include a set of criteria used to convert data from one format to another, make decisions about data, or generate new data based on a set of input data.

[0004] Referring to FIG. 1, in one methodology for developing data transformations using rules, a tabular user interface 100 simplifies the development process for less - specialized users. The user interacts with the tabular user interface 100 to specify conditions 110a - 110h (e.g., inequalities or calculations) applied to the input fields 102, 104, 106, 108 of the input record. In the output field 112, the user associates an output value with each condition. The conditions are applied to the input record in order from the first condition 110a to the last condition 110h, and the output value associated with the first - satisfied condition is output from the rule. Within the user interface, the user can easily apply the rule to the input record and iteratively test and adjust the functionality of the rule.

[0005] Referring to Figure 2, once the user is satisfied with their rule 113, the rule is compiled using the generator 114 to produce a transform 116. The transform 116 is ultimately used as a component in an executable dataflow graph 118 that runs within a graph-based computing system. The dataflow graph referred to herein is an executable computer program. Further details of the rule development methodology can be found in U.S. Patent No. 8,069,129.

[0006] In some examples, a dataflow graph is an executable computer program that includes vertices (representing data processing components or datasets) connected by directed links between vertices (representing work elements, i.e., data flows). For example, such an environment is described in detail in U.S. Patent No. 7,716,630, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference. A system for performing computations based on such graphs is described in U.S. Patent No. 5,966,072, entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," which is incorporated herein by reference. A dataflow graph created according to this system provides a method for moving information between processes and for providing information to and from individual processes represented by graph components in order to define the running order of processes. The system includes an algorithm for selecting a method of communication between processes from any available methods (for example, communication paths following links in the graph can pass data between processes using TCP / IP or UNIX area sockets, or using shared memory). [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] U.S. Patent No. 8,069,129 [Patent Document 2] U.S. Patent No. 7,716,630 [Patent Document 3] U.S. Patent No. 5,966,072 [Overview of the project]

[0008] The rule development methodology described above uses a tabular user interface to provide the user with a comprehensive view of the different conditions that define a single rule, but the tabular user interface does not provide a complete view of the resulting converted records that arise from applying that rule to a set of input records.

[0009] The embodiments described herein relate to alternative and improved methodologies for defining transformations based on user operations on data records in a tabular user interface. The data records are displayed to the user in the tabular user interface. User operations on data records within the tabular user interface (e.g., adding or removing columns, filtering data records, and defining calculations based on data records) are aggregated to form a comprehensive transformation. What the user sees in the tabular interface is the latest representation of the set of input records transformed by the comprehensive transformation. Once the user is satisfied with the transformed data records displayed in the tabular user interface, the user can export the comprehensive transformation (sometimes referred to as the "final transformation") as a reusable data processing program for processing other input data.

[0010] In a general embodiment, a method for developing a reusable data processing program includes the steps of: accessing a plurality of input records; rendering representations of the plurality of input records in one or more user interfaces; receiving a set of one or more data transformation steps; applying the set of data transformation steps to the plurality of input records to obtain a plurality of converted records; rendering representations of the plurality of converted records in one or more user interfaces; and receiving a first user input as a user manipulates the representations of the plurality of converted records using one or more user interfaces, wherein the first user input includes one or more data transformation steps. For each of the one or more data transformation steps provided by the first user input, the method includes the steps of: updating the set of data transformation steps by adding a data transformation step to the set of data transformation steps; updating the plurality of converted records, which includes applying the updated set of data transformation steps to the plurality of input records to obtain updated plurality of converted records; and rendering representations of the updated plurality of converted records in one or more user interfaces. The method further includes receiving a second user input that results in the export of a reusable data processing program, wherein the exported program is at least partially based on an updated set of data transformation steps, and the reusable data processing program is applicable to one or more records different from a plurality of input records.

[0011] The embodiment may include one or more of the following features:

[0012] A set of data transformation steps may include multiple data transformation steps. Multiple data transformation steps may be applied sequentially in an order specified by the user. The method may further include the step of rendering a representation of the set of data transformation steps within one or more user interfaces during the development of a reusable data processing program. The representation of the set of data transformation steps may display the data transformation steps in a list ordered according to an order specified by the user. The representation of the set of data transformation steps may include a data flow graph representation of the data transformation steps. The method may further include the step of receiving a third user input that results in the removal of one or more data transformation steps from the set of data transformation steps, and updating the set of data transformation steps accordingly. The method may further include the step of receiving a third user input that results in the modification of one or more data transformation steps within the set of data transformation steps, and updating the set of data transformation steps accordingly.

[0013] One or more user interfaces may include a tabular interface, in which representations of multiple input records and / or converted records are rendered in a tabular format in the tabular interface. One or more user interfaces may further include a list interface, in which a set of data transformation steps are rendered as a list in the list interface. The set of data transformation steps rendered in the list interface may be ordered according to the order in which the data transformation steps are applied to multiple input records. The method may further include the step of receiving a third user input to change the order in which the set of data transformation steps are applied, and updating the list in the list interface accordingly. The method may further include the step of interacting with the data transformation steps using the list interface to modify the data transformation steps. The method may further include the step of interacting with the data transformation steps using the list interface to remove a data transformation step from the set of data transformation steps, and updating the list in the list interface accordingly.

[0014] The set of data transformation steps may include one or more data transformation steps for filtering, adding fields, and selecting fields. The set of data transformation steps may include a data transformation step for filtering. Exporting a reusable data processing program may include compiling the updated set of data transformation steps to form a reusable data processing program. Exporting a reusable data processing program may include forming a data flow graph representation of the updated set of data transformation steps to form a reusable data processing program. The method may include the steps of calculating a data profile for a plurality of updated post-transformed records and drawing a representation of the data profile in one or more user interfaces.

[0015] A second user input may be received when it is determined that the data profiles for multiple updated post-converted records conform to a predetermined data profile. The predetermined data profile or predetermined profile rule may specify an acceptable range for some characteristic of the data profile. The method may include the steps of calculating the data quality of multiple post-converted records and rendering a representation of the data quality within the user interface. The data quality may include at least one of valid values, invalid values, NULL values, different values, unique values, and / or maximum and minimum value counts.

[0016] In another general embodiment, a system for developing a reusable data processing program includes an interface for accessing a plurality of input records; a first output for plotting representations of the plurality of input records in one or more user interfaces; a first input for receiving a set of one or more data transformation steps; one or more processors for applying the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; a second output for plotting representations of the plurality of transformed records in one or more user interfaces; and a second input for receiving a first user input as a user manipulates representations of the plurality of transformed records using one or more user interfaces, wherein the first user input includes one or more data transformation steps. One or more processors are further configured to update a plurality of post-converted records, which includes updating a set of data conversion steps by adding a data conversion step to the set of data conversion steps for each of the one or more data conversion steps resulting from a first user input, applying the updated set of data conversion steps to a plurality of input records to obtain an updated plurality of post-converted records, and rendering a representation of the updated plurality of post-converted records within one or more user interfaces. The system also includes a third input for receiving a second user input that results in the export of a reusable data processing program, wherein the exported program is at least partially based on the updated set of data conversion steps, and the reusable data processing program is applicable to one or more records different from a plurality of input records.

[0017] In another general embodiment, a non-temporary computer-readable medium stores instructions causing a computing system to carry out a method for developing a reusable data processing program. The instructions cause the computing system to access a plurality of input records, to draw representations of the plurality of input records in one or more user interfaces, to receive a set of one or more data transformation steps, to apply the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records, to draw representations of the plurality of transformed records in one or more user interfaces, and to receive a first user input as a user manipulates the representations of the plurality of transformed records using one or more user interfaces, wherein the first user input includes one or more data transformation steps. For each of the one or more data transformation steps resulting from the first user input, the instruction causes the computing system to update the set of data transformation steps by adding a data transformation step to the set of data transformation steps, to update a plurality of converted records by applying the updated set of data transformation steps to a plurality of input records to obtain an updated plurality of converted records, and to draw a representation of the updated plurality of converted records in one or more user interfaces. The instruction further causes the computing system to receive a second user input that results in the export of a reusable data processing program, wherein the exported program is at least partially based on the updated set of data transformation steps, and the reusable data processing program is applicable to one or more records different from the plurality of input records.

[0018] In another general embodiment, a system for developing a reusable data processing program includes means for accessing a plurality of input records; means for rendering representations of the plurality of input records in one or more user interfaces; means for receiving one or more sets of data transformation steps; means for applying the set of data transformation steps to the plurality of input records to obtain a plurality of converted records; means for rendering representations of the plurality of converted records in one or more user interfaces; and means for receiving a first user input as a user manipulates representations of the plurality of converted records using one or more user interfaces, wherein the first user input includes one or more data transformation steps. The system further includes processing means configured to update a plurality of converted records, including updating the set of data transformation steps by adding a data transformation step to the set of data transformation steps for each of the one or more data transformation steps of the first user input, and applying the updated set of data transformation steps to the plurality of input records to obtain updated plurality of converted records, and rendering representations of the updated plurality of converted records in one or more user interfaces. The system further includes means for receiving a second user input that results in the export of a reusable data processing program, wherein the exported program is at least partially based on an updated set of data transformation steps, and the reusable data processing program is applicable to one or more records different from a plurality of input records.

[0019] In another general embodiment, a method for developing a reusable data processing program including a set of data transformation steps, by displaying a set of records to allow a user to iteratively select one or more data transformation steps, iteratively applying the data transformation steps to records, and iteratively displaying the transformed records, includes the steps of accessing some input records; drawing representations of some transformed records in a user interface, wherein some transformed records are determined by applying a set of data transformation steps to some input records; and receiving a first user input as a user interacts with representations of some transformed records using the user interface, wherein the first user input includes one or more data transformation steps. For each of the one or more data transformation steps, the method includes the steps of adding a data transformation step to the set of data transformation steps; updating some transformed records, which includes applying the set of data transformation steps to some input records; and drawing representations of some transformed records in a user interface. The method includes receiving a second user input that causes an export of a reusable data processing program based at least in part on a set of data transformation steps, the reusable data processing program receiving a second user input that is applicable to one or more records different from some input records.

[0020] In another general embodiment, a computer program includes instructions that, when the program is executed by the computer, cause the computer to perform the method described in any of the prior claims.

[0021] Among its various advantages, the embodiment particularly provides a convenient graphical shortcut for configuring settings in data transformation, instructing the computer to transform data records in a specific form. This graphical shortcut may be accompanied by a tabular user interface for developing and / or adjusting complex transforms related to data processing programs. The graphical shortcut allows selecting data processing conditions, such as changing, adding, or removing transform steps, directly through a tabular representation of the records, without having to navigate through the transform code each time a transform step needs to be modified, added, or removed. This conserves computing resources and makes the implementation of changes, additions, or removals of transform steps efficient and reliable.

[0022] Furthermore, the configuration saves time and computing resources while ensuring proper execution of the resulting data processing program. For example, providing an environment for developing rules in a tabular interface conveniently allows users to see the results of applying those rules to the data in real time. This allows for more efficient use of computing resources, as seeing the results helps users immediately recognize errors in the code without relying on computationally inefficient trial and error. A shorter feedback cycle between editing and seeing the results can help users catch logical and conceptual errors more quickly. In some configurations, the interface operates on a subset of data (e.g., 500 records) so that logic runs more frequently to provide user feedback and shorter feedback loops, but also runs on smaller datasets cached during the session to reduce the cost of extracting from the source system (e.g., cloud providers such as Amazon S3 are paid only once for usable data, not once per edit). Some embodiments also optimize data processing programs by, for example, merging steps, bringing logic down to the source system, and rearranging logic to make it more efficient.

[0023] Aspects also advantageously ensure a higher likelihood of proper execution of the resulting data processing program. Software testing and debugging are difficult problems, and even the most well-developed software contains bugs. The aspects described herein provide a view of the output data as the software is being developed, facilitating the quick and easy discovery of errors within the software (i.e., based on visual inspection, the user can almost immediately know that the output data is different from what the user expected). As a result, the software developed using the present invention is more likely to execute without errors. That is, the interface provides a powerful debugging tool, where the tabular interface acts like a debugger probe, presenting results that help the user intuitively recognize and solve errors within the data processing program.

[0024] Some aspects advantageously eliminate the need for the user to manually construct a data flow graph. Since the construction of the data flow graph occurs behind the scenes, the user does not have to spend time configuring the layout of the data flow graph and waste computing resources. This corresponds to a graphical shortcut, where program development in the tabular view significantly reduces the processing load on the underlying computing hardware (e.g., by reducing the graphical processing load from activities such as dragging and dropping components, connecting components to each other, repositioning components, switching between development views and runtime views, etc.).

[0025] Still other aspects guide the user to obtain a resulting data processing program that reduces incorrect manipulation of tabular representations of data, reduces errors within the transform, and is more likely to be executable and properly applicable to data records. The tabular view is, by its nature, a restricted programming environment that aims to reduce some of the scenarios in which a user can introduce errors into a program. Compared to a spreadsheet program, the aspects operate based on semantically meaningful concepts such as "data fields" rather than interface elements such as "spreadsheet cells". Among other benefits, this prevents the types of errors that occur in spreadsheets when formulas are not correctly replicated between cells (not to mention saving the effort of having to replicate formulas across multiple cells). For example, spreadsheet formulas can potentially use relative references when absolute references should be used, or vice versa, or the formula may not be properly copied to inserted rows. Additionally, showing test data during the construction of a mathematical formula / transform can also help catch logical errors without waiting to run the formula / transform across the entire dataset.

[0026] Field-by-field profiling can also be used to catch some types of logical errors that arise from misunderstandings about the input data. For example, it helps the user notice that a numeric value probably came from a restricted set of values so that no one tries to perform mathematical operations on it. That type of information may not be immediately obvious from looking at the first few dozen values, but the data profile can clarify it.

[0027] The ability to perform test transforms on very large datasets conveniently facilitates the identification of outliers in the results. Local spreadsheets can suffer performance problems when run on extremely large datasets, so in practice, users write formulas in smaller spreadsheet files and then transfer them to larger files, whereas the transforms described herein are implemented as graphs and can be applied to the dataset from the start.

[0028] Other features and advantages of the present invention will become apparent from the following description and claims. [Brief explanation of the drawing]

[0029] [Figure 1] This is a diagram of the user interface for conventional technologies used to develop rules. [Figure 2] This figure shows the conventional process for compiling rules developed using the user interface shown in Figure 1 for use in a data processing system. [Figure 3] This is a schematic diagram of a system for developing a data flow graph from user operations on data records in a tabular user interface. [Figure 4] This is a diagram of a data flow graph development environment. [Figure 5A] This diagram shows the user interface while a user is adding a field selection transformation. [Figure 5B] This figure shows the result after applying the field selection transformation in Figure 5A and the user selecting the button to add a "Add Field" transformation. [Figure 5C] This figure shows how a user configures the field addition transformation shown in Figure 5B. [Figure 5D] Figure 5B shows the user interface with the "Add Field" transformation applied and the user selected the button to add a "Add Filter" transformation. [Figure 5E] This figure shows how a user configures the filter addition transformation shown in Figure 5D. [Figure 5F] This figure shows the user interface of Figure 5D with the filter addition transformation applied. [Figure 6] This figure shows the data flow graph representation of the transformation set shown in Figures 5A to 5F. [Figure 7A] This diagram shows the user interface with the button for adding a "Summary Addition" transformation selected. [Figure 7B]This diagram shows how users configure aggregation, additions, and transformations. [Figure 7C] Figure 7A shows the user interface with the aggregated transformation applied and the user selected the button to view the data profile. [Figure 7D] This figure shows the representation of the data profile as drawn on the user interface in Figure 7A. [Figure 7E] Figure 7A shows the representation of data quality analysis as depicted in the user interface. [Modes for carrying out the invention]

[0030] 1. Overview Referring to Figure 3, user 320 manipulates a tabular view 321 of the test data 334 within the user interface 322 according to an iterative development method 326 to develop a set of data transformations called the set of “final” transformations 324. Generally, the tabular view 321 provides a familiar, easy-to-use spreadsheet-type interface that allows the user to iteratively manipulate the test data to develop the set of final transformations 324, which may be a complex data processing program.

[0031] The iterative development method 326 includes a first step 328 in which the user 320 adds, removes, or modifies data transformations by manipulating the test data displayed in the tabular view 321 (for example, by adding or removing fields and / or filtering the data). The result of the first step 328 is a set of “working” transformations 330, which is fed back to the user interface 322 and displayed to the user 320 in the transformation history view 323 (for example, as a list of transformations, as described in more detail below).

[0032] The iterative development method 326 includes a second step 332 in which a set of working transformations 330 is applied to working test data 336 to form converted test data 334. As part of the second step 332, the converted test data 334 is also fed back to the user interface 322 and displayed to the user 320 in a tabular view 321. As part of the second step 332, the converted test data 334 may be processed by a data analyzer 337 to generate profile / quality data 339 for the converted test data 334. The profile / quality data 339 is displayed to the user 320 in the user interface 322.

[0033] As the user manipulates the data in the tabular view 321, the first and second steps 328, 332 of the iterative development method 326 are repeated, and the user interface 322 is repeatedly updated to reflect the current working transformation 330 and the transformed test data 334. At any point during the iterative development method 326, the user 320 can view the data profile 339 of the transformed test data 334 to verify that the data 334 matches the desired data profile. For example, the user can compare the data profile obtained from the transformed data with a given data profile to identify any errors. In some examples, errors are communicated to the user as a warning, which includes information guiding the user on how to correct the errors. Finally, in the third step of the iterative development method 326, once the user is satisfied with the state of the transformed test data 334 displayed in the tabular view 321, the user exports the set of working transformations 330 as the set of final transformations 324. The set of final transformations 324 is exported in a format that can be used to transform data other than the working test data 326. One example of such a form is a component for use in data flow graphs.

[0034] 2. Example 1 Referring to Figures 4 to 6, a first step-by-step example is provided illustrating the use of the user interface 322 described above, following an iterative development method 326 for generating a final set of transformations 324.

[0035] Referring to Figure 4, in some examples, the user begins the development process in the data flow graph development environment 436. The data flow graph development environment 436 in Figure 4 includes a canvas 438, where the user "drags" data processing components 440 from the component list 442 and data sources and data sinks 444 from the data catalog 446. The user then "wires" the input and output ports of the components, data sources, and data sinks together to establish the flow of data through the data flow graph 448. The user can run the data flow graph 448 and view the results on the console 450.

[0036] In Figure 4, to create the data flow graph 448, the user drags the "Country" data source 452, the "ActiveViewData" component 454, and the "Density" data sink 456 onto the canvas 438. The output port of the Country data source 452 is connected to the input port of the ActiveViewData component 454, and the output port of the ActiveViewData component 454 is connected to the input port of the Density data sink 456, forming the data flow graph 448.

[0037] Next, the Active View Data component 454 is configured by the user to process data from the country data source 452 and generate density data that is written to the density data sink 456. To do this, the user double-clicks the Active View Data component 454 to open the component. Throughout the rest of this example, a user interacting with an element of the user interface is represented by the outline of that element being shown with a thick line. For example, in Figure 4, the user is interacting with (e.g., "clicking") the Active View Data component 454, and the outline of this component is shown with a thick line.

[0038] Referring to Figure 5A, opening the ActiveViewData component 454 displays the ActiveViewData user interface 322. The user interface 322 includes a tabular view 321, a field selection menu 319, a conversion history view 323, and several buttons 556.

[0039] The tabular view 321 displays the test data 334 in a tabular format, with rows 558 corresponding to record numbers (e.g., records 1-7) and columns 560 corresponding to the record fields (e.g., "Country Name", "Country Code", etc.). The field values ​​for a specific record are displayed in cell 562 at the intersection of a specific row and column (e.g., the code for record 3 is "MK"). The user 320 can scroll through the tabular view 321 to view the test data 334 (note that the tabular view 321 displays the original working test data because the working transformation set in Figure 5A does not include any transformations).

[0040] The field selection menu 319 contains checkboxes 325 for each field in the original working test data. Each checkbox can be toggled to select whether the associated field is displayed in the tabular view 321. In Figure 5A, all fields are displayed in the tabular view 321 because the checkboxes for all fields were checked beforehand. The conversion history view 323 displays an ordered list of the current set of working conversions 330. In Figure 5A, nothing is shown in the conversion history view 323 because the set of working conversions is empty.

[0041] Button 556 is associated with a set of data transformations or other actions that can be applied to the working test data 336. In some examples, the button includes an "Add Filter" button 563 that allows the user to add a filter transformation to a set of working transformations 330, an "Add Field" button 564 for adding an "Add Field" transformation to a set of working transformations, an "Add Aggregation" button 590 for adding an "Add Aggregation" transformation to a set of working transformations, a "Show Profile" button 591 for the user to view the data profile of the transformed test data 334, and a "Show Data Quality" button 592 for the user to view the results of a data quality analysis of the transformed test data 334. The transformations and actions associated with button 556 are described in more detail below. The "View Graph" button 567 displays a data flow graph associated with a set of working transformations to the user.

[0042] In Figure 5A, the user has "unchecked" the checkboxes 325 associated with the Code, Capital, and Region fields in the field selection menu 319, while leaving the checkboxes for Name, Area, and Population checked. Referring to Figure 5B, unchecking the checkboxes 325 associated with the Code, Capital, and Region fields adds a field selection transformation set to Working Transformation 330, which is applied to the Working Test Data to generate the Post-Transformed Test Data 334. As a result of applying the field selection transformation, the only fields remaining in the Post-Transformed Test Data 324 and displayed in the Tabular View 321 are the Name, Area, and Population fields. Furthermore, the field selection transformation is added to the Transformation History View 323 as the first transformation 572.

[0043] In Figure 5B, the user then clicks the Add Field button 564 to begin adding a field addition transformation to the set of working transformations 330. Referring to Figure 5C, when the user clicks the Add Field button 566, the Add Field dialog 573 appears. The dialog 573 includes the Name field 575, the Data Type field 574, and the Formula field 576.

[0044] The Name field 575 prompts the user to specify a name for the new column. In this case, the user has selected "Density" as the name for the new column. In the Data Type field 574, the user can select the data type for the new column from a list of data types, such as Number, String, and Boolean (or the user can choose to have the data type automatically detected). In this case, the user has chosen to have the data type automatically detected, and as a result, Number is the data type for the new column.

[0045] Formula field 576 requires the user to specify a formula to be used to populate a new column with values ​​(for example, a calculation based on the values ​​of one or more fields in a record). In this case, the user specifies a formula to calculate the population density of each country, as follows: “= round(Country.population / Country.area)” (That is, divide the country's population by the country's area and round to the nearest integer.) When finished, the user clicks the save button to return to user interface 322.

[0046] Referring to Figure 5D and returning to the user interface 322, at this point a set of working transformations 330, including the field addition transformation, is applied to the working test data 336 to generate the transformed test data 334. As a result of applying the field addition transformation, a new density field 577 now exists in the tabular view 321, showing the calculated population density for each country (i.e., row) shown in the tabular view 321. Note that a pencil icon 593 next to the density field heading indicates that the field was added by the user. Furthermore, the field addition transformation is added to the transformation history view 323 as a second transformation 578.

[0047] In Figure 5D, the user then clicks the "Add Filter" button 563 to add a "Filter" transformation to the set of working transformations 330. Referring to Figure 5E, when the user clicks the "Add Filter" button 566, the "Add Filter" dialog box 579 appears. The dialog box 579 includes a record selection field 580, a formula definition field 581, and a formula output indicator 582. In the formula definition field 581, the user can specify conditions to be applied to the values ​​of one or more fields in the working test data 336 to determine which records in the working test data are "filtered out" from the converted test data 334 displayed in the tabular view 321. For example, in Figure 5E, the user defines a formula that retains only records where the density field has a value equal to "1". The user can use the record selection field 580 to navigate and view records in the converted test data, and as they navigate and view, the filter formula is evaluated for the currently selected record and input into the formula output indicator 582. For example, in Figure 5E, the user has selected record "1" using the record selection field 580, and the formula indicator contains the value "false," indicating that the density value for record 1 is not equal to "1" (recall from Figure 5D that the population density value for record 1 is equal to 113). The false value means that record 1 will be filtered out from the transformed test data that will ultimately be displayed in the tabular view 321. Once the user is satisfied with their filtered transformation, they click the save button to return to the user interface 322.

[0048] Referring to Figure 5F and returning to the user interface 322, at this point a set of working transformations, including the filter transformation, is applied to the working test data 336 to generate the transformed test data 334. As a result of applying the filter transformation, only one record, "Western Sahara," remains in the transformed test data 334 displayed in the tabular view 321, which is the only country in the working test data with a population density equal to approximately "1". Furthermore, the additional filter transformation is added to the transformation history view 323 as a third transformation 583.

[0049] In some cases, once the user is satisfied with the transformed test data 334 shown in the tabular view 321, the user can click the save button to export the set of working transformations as the set of final transformations 324. Clicking the save button returns the user to the data flow graph development environment 436 in Figure 4, where the active view data component 454 is configured according to the set of final transformations and can be reused to process other data sources taken from the data catalog 446.

[0050] Referring to Figure 6, in another example, the user can click the "View Graph" button 567 to display a dataflow graph representation of a set of working transformations 684 on the canvas 438 of the dataflow graph development environment 436. In this example, the dataflow graph representation of the set of working transformations 684 includes a field selection component 685, a field addition component 686, and a filter component 687, all of which are interconnected according to the order of transformations in the set of working transformations (for example, as shown in the transformation history view 323).

[0051] 3. Example 2 Referring to Figures 7A to 7E, a second step-by-step example is provided illustrating the use of the user interface 322 described above to generate the final set of transformations 324. In the second example, the final set of transformations 324 includes rollup aggregations, and the user utilizes data profiles to develop the final set of transformations.

[0052] Referring to Figure 7A, user 320 has already added a field selection transformation to the set of working transformations using the field selection menu 319. The field selection transformation is applied to the working test data 336 to generate converted test data 334, which includes the customer ID ("cust_ID") field and the charge amount ("charge_amt") field. In this example, each row of the converted test data represents a different transaction in which a charge amount was imposed on a customer with a specific customer ID (for example, the customer made a purchase using a credit card). The field selection transformation is shown in the transformation history view 323 as the first transformation 772.

[0053] In Figure 7A, the user clicks the "Add Aggregation" button 590 to add the "Aggregation" transformation to the set of working transformations 330. Referring to Figure 7B, clicking the Add Aggregation button brings up the "Add Aggregation" dialog 792. The dialog 792 includes the "Field Name" field 793, the Aggregation "Key" field 794, and the Aggregation "Formula" field 795. The Field Name field 793 is the name of the field where the result of the aggregation transformation will be stored. In the example in Figure 7B, the user enters "total_charges" in the Field Name field 793 because they want to determine the total amount of charges per customer ID.

[0054] In the summary key field 794, user 320 has selected the charge_amt field as the summary key. The user has entered a formula in the summary formula field 795. “rollup_sum(purchase_details.charge_amt)” This incorporates the information, indicating that the aggregate transformation is a rollup aggregate that determines the sum of charge_amt values ​​for each unique cust_ID. In Figure 7B, the user clicks the save button and returns to the user interface 322.

[0055] Referring to Figure 7C and returning to the user interface 322, a set of working transformations 330, including the Add Aggr. transformation, is applied to the working test data to generate the converted test data 334. As a result of applying the Add Aggr., the cardinality of the converted test data 334 changes. In this case, rows representing multiple transactions corresponding to cust_ID are collapsed into a single row representing the sum of charge_amt corresponding to cust_ID, thus reducing the number of rows. To represent this change in cardinality, a new page 796 called "charge_amt" is added to the tabular view 321 of the user interface 322. The user can switch between the charge_amt page 796 and the tabs of the original "main" page 797 by clicking the tabs associated with each page. The Add Aggr. transformation is added to the transformation history view 323 as a second transformation 778 (abbreviated as "Add Aggr.").

[0056] In Figure 7C, the user then clicks the profile display button 591 to view the data profile of the converted test data 334 shown in the tabular view 321. Referring to Figure 7D, when the user clicks the profile display button 591, the data profiler 337 calculates the profile data 339 of the converted test data 334. A graphical representation of the profile data for each field in the converted test data 334 is then displayed in the column for that field in the tabular view 321.

[0057] In Figure 7D, the first graphic representation of the profile data 798 is shown in the cust_ID column, and the second graphic representation of the profile data 799 is shown in the charge_amt column. The first graphic representation 798 includes a histogram 701 which shows that the cust_ID field has values ​​in the range of approximately 1000 to 2000 and there are no duplicate values ​​(i.e., there is "1" instance for each unique cust_ID). The first graphic representation 798 includes data quality bars 702 which indicate high data quality for that field (e.g., data with few duplicate entries, few blank entries, and / or few entries with invalid or inaccurate values). The second graphic representation of the profile data 799 includes a histogram 703 which shows that while the majority of customers have a charge_amt value of less than approximately $7,000, there are also customers with charge_amt values ​​exceeding that number, up to approximately $15,000. A second graphical representation of the profile data, 799, also includes data quality bars, 704, that indicate several minor data quality issues, shown as areas of different patterns within the data quality bars. More generally, a data profile provides information about specific characteristics of the data. This information can be used to determine whether a data transformation applied to the data is suitable for deployment. For example, a data profile could group customers into bins based on how much money they spend. Developers may use the data profile to identify problems (e.g., bugs) in the transformation. For example, if all customers fall into the same bin, or if the bin values ​​are not what the developer expected, the developer may rethink and debug their transformation. Data quality characterizes how complete and accurate the data is. For example, data quality characterizes aspects of the data such as duplicate records, records with missing fields, and records with inaccurate data (e.g., misspellings, invalid zip codes, invalid, etc.).Developers can use data quality to measure the quality of the results produced by their transformations and, if necessary, correct the transformations to address data quality issues.

[0058] In Figure 7D, user 320 clicks on carat 705 to investigate further the data quality issues related to the charge_amt field. Referring to Figure 7E, when the user clicks on carat 705, the data profile view 706 is displayed. The data profile view 706 includes a pie chart 707 of different charge_amt values ​​and a value summary 708 that includes valid values, invalid values, NULL values, different values, unique values, and maximum and minimum value counts. In this case, user 320 can see that there are 50 invalid values ​​and 5 NULL values, which are the cause of the data quality issues indicated by the data quality bar 704. Furthermore, user 320 can see that there are 905 different values, 750 unique values, a maximum value of $14,995, and a minimum value of $10. After viewing the profile data 339, user 320 may decide to change the set of working transformations to adjust the data profile.

[0059] In some cases, a more complete characterization of data quality can be accessed by clicking the "Data Quality Display" button 592 on the user interface 322. A further description of this characterization of data quality is beyond the scope of the present invention and is not discussed further herein.

[0060] 4. Alternative examples In some cases, the user may need to modify a conversion or remove a conversion from the set of working conversions 330. The user can do this within the user interface 322 by interacting with the list of conversions in the conversion history view 323 (for example, by clicking the modify or remove button associated with a conversion in the list). Furthermore, there may be situations where the user wants to rearrange the conversions in the conversion history view. In such cases, the user can change the order shown in the conversion history view, for example, by dragging the conversions.

[0061] It should be noted that the types of transformations described in the above example are merely examples of transformations that may be available in user interface 322, and that other transformations may be available to the user.

[0062] In the example described above, the data accessed by the data processing program is represented as a dataset (for example, a database or other set of data stored on disk or in memory). However, the data flow can also be used to develop the data processing program, and it should be recognized that it can be processed by the data processing program in the runtime configuration.

[0063] The step of exporting the final set of transformations 324 may be a compilation step that translates the final set of transformations into a low-level programming language such as assembly language, object code, or machine code to create an executable program. Alternatively, the export step may translate the final set of transformations into Ab Initio's DML programming language, Ab Initio's dataflow graph, or Ab Initio's “EZ graph,” which is an easily modifiable and optimizable computation graph (as described in U.S. Patent Publication No. 2021-0232579, which is incorporated herein by reference). Finally, the export step may export the set of transformations without any translation at all.

[0064] The data processing programs exported from the iterative development method described above can be used not only to process working test data, but also to process other real-world data, both for bulk and streaming purposes.

[0065] 5. Embodiments The computational resource allocation methods described above can be implemented, for example, using a programmable computing system that executes appropriate software instructions, or as appropriate hardware such as a field-programmable gate array (FPGA), or in some hybrid form. For example, in a programmed method, the software may include procedures in one or more computer programs running on one or more programmed or programmable computing systems (which may be of various architectures such as distributed, client / server, or grid), each including at least one processor, at least one data storage system (including volatile and / or non-volatile memory, and / or storage elements), and at least one user interface (for receiving input using at least one input device or port, and for providing output using at least one output device or port). The software may include, for example, one or more modules of a larger program that provide services related to the design, configuration, and execution of a data processing graph. The modules of the program (e.g., elements of a data processing graph) may be implemented as data structures or other organized data that match a data model stored in a data repository.

[0066] Software may be stored in a non-temporary form, such as being embodied as a volatile or non-volatile storage medium, or any other non-temporary medium, that utilizes the physical properties of the medium (e.g., surface pores and protrusions, magnetic areas, or electric charge) for a period of time (e.g., the time between refresh cycles of a dynamic memory device such as dynamic RAM). In preparation for instruction loading, software may be provided on a tangible non-temporary medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or it may be transported via a network communication medium to a tangible non-temporary medium of the computing system on which it is executed (e.g., it may be encoded in a propagating signal). Some or all of the processing may be performed on a special-purpose computer, or using special-purpose hardware such as a coprocessor, a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Processing may be performed in a distributed manner, where different parts of the computation specified by the software are performed by different computing elements. Each such computer program is preferably stored in or downloaded to a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) of a general-purpose or special-purpose programmable computer, in order to configure and operate the computer to perform the processing described herein when the storage medium is read by the computer. The system of the present invention may also be considered to be implemented as a tangible non-temporary medium composed of computer programs, wherein the medium is configured to operate the computer in a specific predetermined manner to perform one or more of the processing steps described herein.

[0067] Several embodiments of the present invention have been described. However, it should be understood that the foregoing description is for illustrative purposes only and is not intended to limit the scope of the present invention as defined by the following claims. Therefore, other embodiments are also within the scope of the following claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be order-independent and may be performed in an order different from that described.

Claims

1. A method for developing a reusable data processing program, Steps to access multiple input records, The steps include: rendering the representation of the plurality of input records within one or more user interfaces; A step of receiving a set of one or more data conversion steps, A step of applying the set of data conversion steps described above to the plurality of input records to obtain a plurality of converted records, The steps include: drawing the representations of the plurality of converted records within the one or more user interfaces; A step of receiving a first user input as the user manipulates the representation of the plurality of converted records using the one or more user interfaces, wherein the first user input includes one or more data conversion steps, For each of the one or more data conversion steps based on the first user input, A step of updating the set of data conversion steps by adding the data conversion steps to the set of data conversion steps, A step to update a plurality of converted records, which includes applying the updated set of data conversion steps to the plurality of input records to obtain a plurality of updated converted records, and The steps include: rendering the updated representation of the multiple converted records within the one or more user interfaces; A step of receiving a second user input that causes the export of the reusable data processing program, wherein the exported program is at least partially based on an updated set of the data conversion steps, and the reusable data processing program is applicable to one or more records different from the plurality of input records. The method, including the method described above.

2. The method according to claim 1, wherein the set of data conversion steps includes a plurality of data conversion steps.

3. The method according to claim 2, wherein the plurality of data conversion steps are applied sequentially in an order specified by the user.

4. The method according to claim 3, further comprising the step of drawing a representation of the set of data conversion steps within the one or more user interfaces during the development of the reusable data processing program.

5. The method according to claim 4, wherein the representation of the set of data conversion steps displays the data conversion steps in a list ordered according to an order specified by the user.

6. The method according to claim 4, wherein the representation of the set of data transformation steps includes a data flow graph representation of the data transformation steps.

7. The method according to claim 4, further comprising the steps of receiving a third user input that results in the removal of one or more data conversion steps from the set of data conversion steps, and updating the set of data conversion steps accordingly.

8. The method according to claim 4, further comprising the steps of receiving a third user input that causes a change in one or more data conversion steps within the set of data conversion steps, and updating the set of data conversion steps accordingly.

9. The method according to claim 1, wherein the one or more user interfaces include a tabular interface, and the representations of the plurality of input records and / or converted records are rendered in a tabular format on the tabular interface.

10. The method according to claim 1, wherein the one or more user interfaces further include a list interface, and the set of data conversion steps is rendered as a list in the list interface.

11. The method according to claim 10, wherein the set of data transformation steps drawn on the list interface is ordered according to the order in which the data transformation steps are applied to the plurality of input records.

12. The method according to claim 11, further comprising the steps of receiving a third user input to change the order in which the set of data transformation steps are applied, and updating the list in the list interface accordingly.

13. The method according to claim 11, further comprising the step of interacting with a data transformation step using the list interface and modifying the data transformation step.

14. The method according to claim 11, further comprising the steps of interacting with a data transformation step using the list interface, removing a data transformation step from the set of data transformation steps, and updating the list in the list interface accordingly.

15. The method according to claim 1, wherein the set of data transformation steps includes one or more of a filter data transformation step, a field addition data transformation step, and a field selection data transformation step.

16. The method according to claim 1, wherein the set of data conversion steps includes a filter data conversion step.

17. The method according to claim 1, wherein causing the export of the reusable data processing program includes compiling an updated set of the data conversion steps to form the reusable data processing program.

18. The method according to claim 1, wherein generating the export of the reusable data processing program comprises forming a dataflow graph representation of the updated set of data transformation steps to form the reusable data processing program.

19. The method according to claim 1, further comprising the steps of: calculating a data profile for the plurality of updated post-conversion records; and rendering a representation of the data profile within the one or more user interfaces.

20. The method according to claim 1, wherein the second user input is received when it is determined that the data profiles for the plurality of updated converted records conform to a predetermined data profile.

21. The method according to claim 20, wherein the predetermined data profile or predetermined profile rule specifies an acceptable range for some characteristic of the data profile.

22. The method according to claim 1, further comprising the steps of calculating the data quality of the plurality of converted records and rendering a representation of the data quality within the user interface.

23. The method according to claim 22, wherein the data quality includes at least one of valid values, invalid values, NULL values, different values, unique values, and / or maximum and minimum value counts.

24. A system for developing reusable data processing programs, An interface for accessing multiple input records, A first output for rendering the representation of the plurality of input records within one or more user interfaces, A first input for receiving a set of one or more data conversion steps, One or more processors for applying the set of data conversion steps described above to the plurality of input records to obtain a plurality of converted records, A second output for rendering the representation of the plurality of converted records within the one or more user interfaces, As the user manipulates the representation of the plurality of converted records using the one or more user interfaces, a second input for receiving a first user input, wherein the first user input includes one or more data conversion steps, and Includes, The one or more processors, for each of the one or more data conversion steps of the first user input, Updating the set of data conversion steps by adding the data conversion steps to the set of data conversion steps, Updating the plurality of converted records, which includes applying the updated set of the data conversion steps to the plurality of input records to obtain the updated plurality of converted records, and To render the representation of the updated multiple converted records within the one or more user interfaces. Further configured to perform, A third input for receiving a second user input that results in the export of the reusable data processing program, wherein the exported program is at least partially based on an updated set of the data conversion steps, and the reusable data processing program is applicable to one or more records different from the plurality of input records. The system comprising the above.

25. A non-temporary computer-readable medium storing instructions for causing a computing system to execute a method for developing a reusable data processing program, wherein the instructions are transmitted to the computing system. Accessing multiple input records, The representation of the aforementioned plurality of input records is rendered within one or more user interfaces, Receiving a set of one or more data conversion steps, Applying the set of data conversion steps described above to the multiple input records to obtain multiple converted records, The representation of the plurality of converted records is drawn within the one or more user interfaces, Receiving a first user input as the user manipulates the representation of the plurality of converted records using the one or more user interfaces, wherein the first user input includes one or more data conversion steps. For each of the one or more data conversion steps based on the first user input, Updating the set of data conversion steps by adding the data conversion steps to the set of data conversion steps, Updating the plurality of converted records, which includes applying the updated set of the data conversion steps to the plurality of input records to obtain the updated plurality of converted records, and The updated representation of the multiple converted records is rendered within the one or more user interfaces, Receiving a second user input that causes the export of the reusable data processing program, wherein the exported program is at least partially based on an updated set of the data conversion steps, and the reusable data processing program is applicable to one or more records different from the plurality of input records. The non-temporary computer-readable medium that causes the execution of the above.

26. A system for developing reusable data processing programs, A means of accessing multiple input records, Means for rendering the representation of the plurality of input records within one or more user interfaces, Means for receiving one or more data conversion steps, means for applying the set of data conversion steps described above to the plurality of input records to obtain a plurality of converted records, Means for rendering the representation of the plurality of converted records within the one or more user interfaces, A means for receiving first user input as the user manipulates the representation of the plurality of converted records using the one or more user interfaces, wherein the first user input includes one or more data conversion steps, For each of the one or more data conversion steps based on the first user input, Updating the set of data conversion steps by adding the data conversion steps to the set of data conversion steps, Updating the plurality of converted records, which includes applying the updated set of the data conversion steps to the plurality of input records to obtain the updated plurality of converted records, and To render the representation of the updated multiple converted records within the one or more user interfaces. A means of processing configured to perform, Means for receiving a second user input that causes the export of the reusable data processing program, wherein the exported program is at least partially based on an updated set of the data conversion steps, and the reusable data processing program is applicable to one or more records different from the plurality of input records. The system comprising the above.

27. A method for developing a reusable data processing program that includes a set of data transformation steps, by displaying a set of records, allowing a user to iteratively select one or more data transformation steps, iteratively applying the data transformation steps to the records, and iteratively displaying the transformed records, Steps to access multiple input records, A step of drawing representations of the plurality of converted records within a user interface, wherein the plurality of converted records are determined by applying the set of data conversion steps to the plurality of input records, A step of receiving a first user input as the user manipulates the representation of the plurality of converted records using the user interface, wherein the first user input includes one or more data conversion steps, For each of the one or more data conversion steps described above, A step of adding the data conversion step to the set of data conversion steps, A step of updating the plurality of converted records, which includes applying the set of data conversion steps to the plurality of input records, and The steps include: drawing representations of the multiple converted records within the user interface; A step of receiving a second user input that causes the export of the reusable data processing program based at least in part on the set of data conversion steps, wherein the reusable data processing program is applicable to one or more records different from the plurality of input records; The method, including the method described above.

28. A computer program including instructions, wherein when the program is executed by a computer, the instructions cause the computer to perform the method according to any one of claims 1 to 27.

Images