Reusable data processing program generation

Abstract

A method for developing a reusable data processing program comprising a set of data transformation steps by displaying a set of records and enabling a user to select one or more data transformation steps, applying the data transformation steps to the records, and displaying transformed records.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 472,445, filed June 12, 2023, the contents of which are incorporated in their entirety. BACKGROUND

[0003] The present invention relates to generating data processing programs that can be reused based on user manipulation of tabular representations of data.

[0004] Complex computations can be represented as dataflow through a directed graph, where components of the computation are associated with vertices of the graph, and dataflow between components corresponds to links (arcs, edges) of the graph. In some cases, the computation associated with a component is described in a human-readable form called a “rule.” A rule includes a set of criteria for transforming data from one format to another, determining data, or generating new data based on a set of input data.

[0005] Reference Figure 1 In one example for developing data transformations using rules, a tabular user interface 100 simplifies the development process for less specialized users. A user interacts with the tabular user interface 100 to specify conditions 110a-110h (e.g., inequalities or computations) that are applied to input fields 102, 104, 106, 108 of input records. In an output field 112, the user associates an output value with each condition. The conditions are applied to input records in order from the first condition 110a to the last condition 110h, and the output value associated with the first condition that is satisfied is output according to the rule. Within the user interface, the user can easily apply the rule to input records to iteratively test and adjust the functionality of the rule.

[0006] Reference Figure 2 When the user is satisfied with their rule 113, the rule is compiled using a generator 114 to generate a transformation 116. The transformation 116 is ultimately used as a component in an executable dataflow graph 118 that is executed in a graph-based computing system. A dataflow graph as referred to herein is an executable computer program. Further details of the rule development example can be found in U.S. Patent No. 8,069,129.

[0007] In some examples, a dataflow graph is an executable computer program that includes vertices connected by directed links between vertices (representing the flow of work elements (i.e., data)). Such an environment is described in greater detail in U.S. Patent 7,716,630, entitled "Managing Parameters for Graph-based Applications," which is incorporated herein by reference. A system for executing such graph-based computations is described in U.S. Patent 5,966,072, entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," which is incorporated herein by reference. Dataflow graphs made according to this system provide a way for information to be brought into and out of individual processes represented by graph components, for moving information between processes, and for defining the order in which processes are run. The system includes algorithms that select the method of inter-process communication from any available method (e.g., the communication path according to the links of the graph can use TCP / IP or UNIX domain sockets or use shared memory to pass data between processes). SUMMARY

[0008] While the above rule development paradigm uses a table user interface to provide the user with a comprehensive view of the different conditions that define the rules, the table user interface does not provide an overall view of the transformed records that result from applying the rules to a collection of input records.

[0009] Aspects described herein relate to alternative and improved paradigms for defining transformations based on user manipulations of data records in a table user interface. Data records are displayed to a user in a table user interface. The user's manipulations of the data records in the table user interface (e.g., adding or removing columns, filtering the data records, and defining computations based on the data records) are aggregated and together form an aggregate transformation. What the user sees in the table interface is a latest representation of a collection of input records transformed by the aggregate transformation. When the user is satisfied with the transformed data records displayed in the table user interface, they can export the aggregate transformation (sometimes referred to as a "final transformation" collection) as a reusable data processing program for processing other input data.

[0010] In general aspects, a method for developing a reusable data processing program includes accessing a plurality of input records; rendering a representation of the plurality of input records in one or more user interfaces; receiving a set of one or more data transformation steps; applying the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; rendering a representation of the plurality of transformed records in the one or more user interfaces; receiving a first user input while the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps. For each data transformation step in the one or more data transformation steps of the first user input, the method includes adding the data transformation step to the set of data transformation steps to update the set of data transformation steps; updating the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records; and rendering a representation of the updated plurality of transformed records in the one or more user interfaces. The method further includes receiving a second user input that causes the reusable data processing program to be exported, the exported program based at least in part on the updated set of data transformation steps, the reusable data processing program applicable to one or more sets of a plurality of records different from the plurality of input records.

[0011] Aspects can include one or more of the following features.

[0012] The set of data transformation steps can include a plurality of data transformation steps. The plurality of data transformation steps can be applied sequentially according to an order specified by the user. The method can further include rendering a representation of the set of data transformation steps in the one or more user interfaces during development of the reusable data processing program. The representation of the set of data transformation steps can display the data transformation steps in a list ordered according to an order specified by the user. The representation of the set of data transformation steps can include a dataflow graph representation of the data transformation steps. The method can further include receiving a third user input that causes one or more data transformation steps to be removed from the set of data transformation steps and the set of data transformation steps to be updated accordingly. The method can further include receiving a third user input that causes one or more data transformation steps in the set of data transformation steps to be modified and the set of data transformation steps to be updated accordingly.

[0013] The one or more user interfaces can include a table interface, and representations of the plurality of input records and / or the plurality of transformed records are rendered in the table interface in a tabular form. The one or more user interfaces can also include a list interface, where the set of data transformation steps are rendered as a list in the list interface. The set of data transformation steps that can be rendered in the list interface are ordered according to an order in which the data transformation steps are applied to the plurality of input records. The method can also include receiving a third user input to change an order in which the set of data transformation steps are applied, and updating the list in the list interface accordingly. The method can also include using the list interface to interact with a data transformation step to modify the data transformation step. The method can also include using the list interface to interact with a data transformation step to remove the data transformation step from the set of data transformation steps, and updating the list in the list interface accordingly.

[0014] The set of data transformation steps can include one or more of a filter data transformation step, an add field data transformation step, and a select field data transformation step. The set of data transformation steps can include a filter data transformation step. Causing the reusable data processing program to be exported can include compiling the updated set of data transformation steps to form the reusable data processing program. Causing the reusable data processing program to be exported can include forming a dataflow graph representation of the updated set of data transformation steps to form the reusable data processing program. The method can include computing a data profile of the updated plurality of transformed records, and rendering a representation of the data profile in the one or more user interfaces.

[0015] The second user input can be received upon determining that a data profile of the updated plurality of transformed records is consistent with a predetermined data profile. The predetermined data profile or predetermined profile rule can specify an allowable range of some characteristic of the data profile. The method can include computing a data quality of the plurality of transformed records, and rendering a representation of the data quality in the user interface. Data quality can include at least one of a count of valid values, invalid values, null values, distinct values, unique values, and / or maximum and minimum values.

[0016] In another general aspect, a system for developing a reusable data processing program includes an interface to access a plurality of input records, a first output to render a representation of the plurality of input records in one or more user interfaces, a first input to receive a set of one or more data transformation steps, one or more processors to apply the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records, a second output to render a representation of the plurality of transformed records in the one or more user interfaces, and a second input to receive a first user input as the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps. The one or more processors are further configured to, for each data transformation step in the one or more data transformation steps of the first user input: add the data transformation step to the set of data transformation steps to update the set of data transformation steps; update the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records; and render a representation of the updated plurality of transformed records in the one or more user interfaces. The system further includes a third input to receive a second user input that causes the reusable data processing program to be exported, the exported program based at least in part on the updated set of data transformation steps, the reusable data processing program adapted for use with one or more sets of records different from the plurality of input records.

[0017] In another general aspect, a non-transitory computer-readable medium stores instructions to cause a computing system to implement a method for developing a reusable data processing program. The instructions cause the computing system to access a plurality of input records, render a representation of the plurality of input records in one or more user interfaces, receive a set of one or more data transformation steps, apply the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records, render a representation of the plurality of transformed records in the one or more user interfaces, and receive a first user input while the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps. For each data transformation step in the one or more data transformation steps of the first user input, the instructions cause the computing system to add the data transformation step to the set of data transformation steps to update the set of data transformation steps, update the plurality of transformed records including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records, and render a representation of the updated plurality of transformed records in the one or more user interfaces. The instructions further cause the computing system to receive a second user input that causes the reusable data processing program to be exported, the exported program based at least in part on the updated set of data transformation steps, the reusable data processing program applicable to one or more sets of multiple records different from the plurality of input records.

[0018] In another general aspect, a system for developing a reusable data processing program includes means for accessing a plurality of input records; means for rendering a representation of the plurality of input records in one or more user interfaces; means for receiving a set of one or more data transformation steps; means for applying the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; means for rendering a representation of the plurality of transformed records in the one or more user interfaces; and means for receiving a first user input as the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps. The system further includes processing means configured to, for each data transformation step in the one or more data transformation steps of the first user input: add the data transformation step to the set of data transformation steps to update the set of data transformation steps; update the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records; and render a representation of the updated plurality of transformed records in the one or more user interfaces. The system further includes means for receiving a second user input that causes the reusable data processing program to be exported, the exported program being based at least in part on the updated set of data transformation steps, the reusable data processing program being applicable to one or more sets of multiple records different from the plurality of input records.

[0019] In another general aspect, a method for developing a reusable data processing program comprising a set of data transformation steps by displaying a set of records and iteratively enabling a user to select one or more data transformation steps, iteratively applying the data transformation steps to the records, and iteratively displaying transformed records includes accessing a plurality of input records; rendering, in a user interface, a representation of the plurality of transformed records determined by applying the set of data transformation steps to the plurality of input records; and receiving a first user input as the user manipulates the representation of the plurality of transformed records using the user interface, the first user input comprising one or more data transformation steps. For each data transformation step in the one or more data transformation steps, the method includes adding the data transformation step to the set of data transformation steps; updating the plurality of transformed records including applying the set of data transformation steps to the plurality of input records; and rendering, in the user interface, a representation of the updated plurality of transformed records. The method further includes receiving a second user input causing the reusable data processing program to be derived based at least in part on the set of data transformation steps, the reusable data processing program being applicable to one or more sets of multiple records different from the plurality of input records.

[0020] In another aspect, a computer program comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any of the preceding claims.

[0021] Among other advantages, aspects advantageously provide a graphical shortcut for setting up in data transformations to instruct a computer to transform data records in a particular way. The graphical shortcut can involve a table user interface for developing and / or adjusting complex transformations for data processing programs. The graphical shortcut allows data processing conditions to be selected, such as directly via a table form of displayed records, without having to cycle through code for transformations each time a change, addition, or removal of a transformation step is needed. This saves computing resources and makes the changing, adding, or removing of transformation steps efficient and reliable.

[0022] Further, aspects save time and computing resources while ensuring correct execution of the resulting data processing program. For example, providing an environment for developing rules in a tabular interface advantageously lets the user see in real-time the results of applying the rules to the data. This can more efficiently use computing resources because seeing the results helps the user immediately identify errors in their code without having to take a computationally wasteful trial-and-error approach. The tighter feedback loop between making edits and seeing results can help the user catch logical and conceptual errors much more quickly. In some aspects, the interface works on a subset of the data (e.g., 500 records), so the logic can run more frequently to provide user feedback and tighter loops, but the interface also runs on a small cached dataset during the session to reduce the cost of extraction from the source system (e.g., pay a cloud provider such as Amazon S3 only once for ready data, rather than once per edit). Some aspects also optimize the data processing program to be more efficient by, for example, merging steps, pushing logic down to the source system, and reordering logic.

[0023] Aspects also advantageously ensure that the resulting data processing program is more likely to execute correctly. Software testing and debugging is a difficult problem - even the most mature software includes bugs. Aspects described herein provide for viewing of output data while developing the software, thereby facilitating quick and easy discovery of errors in the software (i.e., based on visual inspection, the user knows almost immediately that the output data is different from what it is expected to be). As a result, software developed using the invention is more likely to have bug-free execution. That is, the interface provides a powerful debugging tool, where the tabular interface acts like a probe in a debugger, presenting results that help the user intuitively identify and resolve errors in the data processing program.

[0024] Some aspects advantageously eliminate the need for the user to manually construct a dataflow graph. Construction of the dataflow graph is done behind the scenes, so the user does not have to spend time and waste computing resources configuring the layout of the dataflow graph. This is equivalent to a graphical shortcut, where program development in the tabular view (e.g., by reducing the graphical processing load from dragging and dropping components, wiring components together, rearranging components, switching between development and runtime views, etc.) significantly reduces the processing load on the underlying computing hardware.

[0025] Other aspects guide the user to reduce erroneous manipulation of tabular representations of data, reduce errors in transformations, and obtain resulting data processing programs that are more likely to be executable and correctly applied to data records. The tabular view is a constrained programming environment that by its nature aims to reduce the number of ways a user can introduce errors into a program. In contrast to spreadsheet programs, aspects operate on semantically meaningful concepts (such as "data fields") rather than interface elements (such as "spreadsheet cells"). Among other benefits, this prevents a class of errors that can occur in spreadsheets when formulas are not correctly copied between cells (not to mention saving the effort of having to copy a formula into many cells). For example, a spreadsheet formula can use a relative reference when it should use an absolute reference, or vice versa, or a formula can not be correctly copied onto an inserted row. Furthermore, displaying test data while an expression / transformation is being constructed can also help catch logical errors without having to wait to run the expression / transformation on a full data set.

[0026] Establishing a profile per field can also be used to catch some classes of logical errors caused by misinterpretations about the input data. For example, helping the user to notice that a numeric value can come from a constrained set of values so that someone does not attempt to mathematically manipulate it. This type of information can not be immediately obvious by seeing the first tens of values, but the data profile will make it clear.

[0027] Being able to test a transformation on very large data sets also advantageously facilitates the identification of outliers in the results. Native spreadsheets encounter performance issues when run on very large data sets, so in practice users write formulas in smaller spreadsheet files and then migrate them to larger spreadsheet files, whereas the transformations described herein are implemented in graphs and can be applied to a data set from scratch.

[0028] Other features and advantages of the present application will be apparent from the following description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a prior art user interface for developing rules.

[0030] Figure 2 is a prior art process for compiling rules developed using Figure 1 the user interface development for use in a data processing system.

[0031] Figure 3 is a schematic diagram of a system for developing a dataflow graph from user manipulations of data records in a tabular user interface.

[0032] Figure 4 is a dataflow graph development environment.

[0033] Figure 5A A user interface showing the user adding a field selection transform.

[0034] Figure 5B A user interface showing the user applying Figure 5A a field selection transform and the user selecting a button to add an "add field" transform.

[0035] Figure 5C A user interface showing the user configuring Figure 5B an add field transform.

[0036] Figure 5D A user interface showing the user applying an add field transform and the user selecting a button to add an "add filter" transform. Figure 5B

[0037] Figure 5E A user interface showing the user configuring Figure 5D an add filter transform.

[0038] Figure 5F A user interface showing the user applying an add filter transform. Figure 5D

[0039] Figure 6 A dataflow graph representation of Figures 5A-5F a transform collection.

[0040] Figure 7A A user interface showing the user selecting a button to add an "add aggregate" transform.

[0041] Figure 7B A user configuring an add aggregate transform.

[0042] Figure 7C A user interface showing the user applying an add aggregate transform and the user selecting a button to view a data profile. Figure 7A

[0043] Figure 7D A representation of a data profile rendered in a user interface of Figure 7A .

[0044] Figure 7E A representation of a data quality analysis rendered in a user interface of Figure 7A . DETAILED DESCRIPTION

[0045] 1. SUMMARY

[0046] Reference is made to Figure 3 ​​​The user 320 manipulates the table view 321 of test data 334 in the user interface 322 according to an iterative development method 326 to develop a set of data transformations referred to as a "final" transformation set 324. Typically, the table view 321 provides a familiar and easy-to-use spreadsheet-like interface in which the user can iteratively manipulate the test data to develop the final transformation set 324, which can be a complex data processing program.

[0047] The iterative development method 326 includes a first step 328 in which the user 320 adds, removes, or modifies data transformations by manipulating the test data displayed in the table view 321 (e.g., by adding or removing fields and / or filtering the data). The result of the first step 328 is a "work" transformation set 330, which is fed back to the user interface 322 and displayed to the user 320 in the transformation history view 323 (e.g., as a list of transformations described in more detail below).

[0048] The iterative development method 326 includes a second step 332 in which the work transformation set 330 is applied to the work test data 336 to form transformed test data 334. As part of the second step 332, the transformed test data 334 is also fed back to the user interface 322, where it is displayed to the user 320 in the table view 321. Optionally, as part of the second step 332, the transformed test data 334 can be processed in a data profiler 337 to generate profile / quality data 339 for the transformed test data 334. The profile / quality data 339 is displayed to the user 320 in the user interface 322.

[0049] The first step 328 and the second step 332 of the iterative development method 326 are repeated as the user manipulates the data in the table view 321, and the user interface 322 is repeatedly updated to reflect the current work transformations 330 and the transformed test data 334. The user 320 can review the data profile 339 of the transformed test data 334 at any point during the iterative development method 326 to ensure that the data 334 meets a desired data profile. For example, the data profile obtained from the transformed data can be checked against a predetermined data profile to identify any errors. In some examples, errors are communicated to the user as warnings, including information instructing the user how to fix the error. Ultimately, in a third step of the iterative development method 326, when the user is satisfied with the state of the transformed test data 334 displayed in the table view 321, they export the work transformation set 330 as the final transformation set 324. The final transformation set 324 is exported in a form that can be used to transform data other than the work test data 326. One example of such a form is for a component in a dataflow graph.

[0050] 2 Example 1

[0051] Referring Figures 4-6 A first step-by-step example is provided to illustrate the use of the user interface 322 described above to generate a final transform set 324 according to the iterative development method 326.

[0052] Referring Figure 4 In some examples, a user begins developing a process in a dataflow development environment 436. Figure 4 The dataflow development environment 436 of FIG. 4A includes a canvas 438 onto which a user "drags" data processing components 440 from a component list 442 and data sources and sinks 444 from a data catalog 446. The user then "wires" together the input and output ports of the components, data sources, and data sinks to establish data flow through a dataflow graph 448. The user can execute the dataflow graph 448 and view the results in a console 450.

[0053] In Figure 4 To create the dataflow graph 448, the user has dragged a "Countries" data source 452, an "Activity View Data" component 454, and a "Density" data sink 456 onto the canvas 438. The output port of the Countries data source 452 is wired to the input port of the Activity View Data component 454, and the output port of the Activity View Data component 454 is wired to the input port of the Density data sink 456 to form the dataflow graph 448.

[0054] Next, the Activity View Data component 454 is configured by the user to process data from the Countries data source 452 to generate density data that is written to the Density data sink 456. To do this, the user double-clicks on the Activity View Data component 454 to open the component. Throughout the remainder of this example, user interaction with elements of the user interface is indicated by outlining the element in bold lines. For example, in Figure 4 the user has interacted with (e.g., "clicked on") the Activity View Data component 454, and the outline of the component is shown in bold lines.

[0055] Referring Figure 5A Opening the Activity View Data component 454 causes the Activity View Data user interface 322 to be displayed. The user interface 322 includes a table view 321, a field selection menu 319, a transform history view 323, and a plurality of buttons 556.

[0056] The table view 321 displays the test data 334 in tabular form, with rows 558 corresponding to record numbers (e.g., records 1-7) and columns 560 corresponding to fields of the records (e.g., "country name," "country code," etc.). Field values for a particular record are displayed in cells 562 at the intersection of a particular row and column (e.g., record 3 has a code of "MK"). The user 320 can scroll through the table view 321 to view the test data 334 (note that Figure 5A The table view 321 displays the original work test data because no transformations are included in the work transformation set 330.

[0057] The field selection menu 319 includes checkboxes 325 for each field in the original work test data. Each checkbox can be toggled to select whether its associated field is displayed in the table view 321. In Figure 5A In this example, the checkboxes for all fields were previously "checked," so all fields are displayed in the table view 321. The transformation history view 323 displays an ordered list of the current work transformation set 330. In Figure 5A In this example, the work transformation set is empty, so nothing is shown in the transformation history view 323.

[0058] The buttons 556 are associated with a set of data transformations or other operations that can be applied to the work test data 336. In some examples, the buttons include an "add filter" button 563 that allows the user to add a filter transformation to the work transformation set 330, an "add field" button 564 for adding a "add field" transformation to the work transformation set, an "add aggregate" button 590 that allows the user to add an "add aggregate" transformation to the work transformation set, a "show profile" button 591 that allows the user to view a data profile of the test data 334 after the transformations, and a "show data quality" button 592 that allows the user to view the results of a data quality analysis of the test data 334 after the transformations. The transformations and operations associated with the buttons 556 are described in more detail below. The "view graph" button 567 causes a data flow graph associated with the work transformation set to be displayed to the user.

[0059] In Figure 5A the user "unchecks" the checkboxes 325 associated with the code, capital, and province fields in the field selection menu 319, while leaving the name, area, and population checkboxes checked. Referring to Figure 5Bunchecking the checkboxes 325 associated with the code, capital, and province fields causes the select fields transform to be added to the working transform set 330, which is applied to the working test data to generate the transformed test data 334. As a result of applying the select fields transform, the name, area, and population fields are the only fields that are retained in the transformed test data 324 and displayed in the table view 321. In addition, the select fields transform is added as the first transform 572 in the transform history view 323.

[0060] In Figure 5B , the user then clicks the add field button 564 to begin adding the add field transform to the working transform set 330. Referring to Figure 5C , the user clicks the add field button 566 to cause the add field dialog 573 to appear. The dialog 573 includes a "name" field 575, a "data type" field 574, and an "expression" field 576.

[0061] The name field 575 requires the user to specify the name of the new column. In this case, the user has selected "density" as the name of the new column. The data type field 574 allows the user to select the data type of the new column from a list of data types such as number, string, and boolean data types (or the user can select auto-detect data type). In this case, the user has selected auto-detect data type, which results in number being the data type of the new column.

[0062] The expression field 576 requires the user to specify an expression (e.g., a calculation based on the values in one or more fields in the record) that is used to populate the values in the new column. In this case, the user has specified the expression to calculate the population density of a country as:

[0063] "= round(Country.population / Country.area)"

[0064] (i.e., the population of a country divided by the area of the country, rounded to the nearest integer value). When finished, the user clicks the save button to return to the user interface 322.

[0065] Referring to Figure 5DOn return to the user interface 322, the work transform set 330, now including the add field transform, is applied to the work test data 336 to generate the transformed test data 334. As a result of applying the add field transform, a new density field 577 now exists in the table view 321, showing the population density calculated for each country (i.e., row) shown in the table view 321. Note that the pencil icon 593 is next to the title of the density field, indicating that the field was added by the user. Further, the add field transform is added as the second transform 578 in the transform history view 323.

[0066] In Figure 5D the user then clicks the "add filter" button 563 to add a "filter" transform to the work transform set 330. Referring to Figure 5E , the user clicks the add filter button 566 causing an "add filter" dialog 579 to appear. The dialog 579 includes a record selection field 580, an expression definition field 581, and an expression output indicator 582. The expression definition field 581 allows the user to specify a condition to be applied to the values in one or more fields in the work test data 336 to determine which records in the work test data are to be "filtered out" of the transformed test data 334 displayed in the table view 321. For example, in Figure 5E the user has defined an expression that keeps only the records for which the density field has a value equal to "1". The user can use the record selection field 580 to navigate through the records of the transformed test data, and as the user navigates, the filter expression is evaluated on the currently selected record to populate the expression output indicator 582. For example, in Figure 5E the user has selected record "1" using the record selection field 580, and the expression indicator includes a "false" value, indicating that the density value for record 1 is not equal to "1" (recall from Figure 5D that the population density value for record 1 is equal to 113). The false value indicates that record 1 will be filtered out of the transformed test data ultimately displayed in the table view 321. When the user is satisfied with their filter transform, they click the save button to return to the user interface 322.

[0067] Referring to Figure 5F , on return to the user interface 322, the work transform set, now including the filter transform, is applied to the work test data 336 to generate the transformed test data 334. As a result of applying the filter transform, only a single record is retained in the transformed test data 334 displayed in the table view 321 - "Western Sahara", which is the only country in the work test data for which the population density is approximately equal to "1". Further, the add filter transform is added as the third transform 583 in the transform history view 323.

[0068] In some examples, when the user is satisfied with the transformed test data 334 as shown in the table view 321, they can click the save button to export the working transform set as the final transform set 324. Clicking the save button returns the user to the dataflow graph development environment 436 with the final transform set configured to the active view data component 454 and can be reused to process other data sources from the data catalog 446. Figure 4

[0069] Referring to Figure 6 , in other examples, the user can click the "view graph" button 567 to display a dataflow graph representation 684 of the working transform set on the canvas 438 of the dataflow graph development environment 436. In this example, the dataflow graph representation 684 of the working transform set includes a select fields component 685, an add fields component 686, and a filter component 687, all interconnected according to the order of the transforms in the working transform set (e.g., as shown in the transform history view 323).

[0070] 3Example 2

[0071] Referring to Figures 7A-7E , a second step-by-step example is provided to illustrate the use of the user interface 322 described above to generate a final transform set 324. In the second example, the final transform set 324 includes a roll-up aggregation, and the user utilizes a data profile to develop the final transform set.

[0072] Referring to Figure 7A , the user 320 has used the field selection menu 319 to add a select fields transform to the working transform set. The select fields transform is applied to the working test data 336 to generate transformed test data 334 that includes a customer ID ("cust ID") field and a charge amount ("charge amt") field. In this example, the rows of the transformed test data represent different transactions in which a customer with a particular customer ID is charged an amount (e.g., the customer makes a purchase using their credit card). The select fields transform is shown as the first transform 772 in the transform history view 323.

[0073] In Figure 7A , the user clicks the "add aggregation" button 590 to add an "aggregate" transform to the working transform set 330. Referring to Figure 7B , clicking the add aggregation button causes an "add aggregation" dialog 792 to appear. The dialog 792 includes a "field name" field 793, an aggregation "key" field 794, and an aggregation "expression" field 795. The field name field 793 is the name of the field that stores the results of the aggregation transform. In Figure 7B ​In the example of FIG. 8, the user has entered "total_charges" into the field name field 793, as they want to determine the total charge amount for each customer ID.

[0074] In the aggregate key field 794, the user 320 has selected the charge_amt field as the aggregate key. The user has populated the aggregate expression field 795 with the following expression:

[0075] "rollup_sum(purchase_details.charge_amt)"

[0076] indicating that the aggregate transform is a rollup aggregate that determines the sum of the charge_amt values for each unique cust_ID. In Figure 7B the user has clicked the save button to return to the user interface 322.

[0077] Referring back to Figure 7C , upon returning to the user interface 322, the working transform set 330 including the add aggregate transform is applied to the working test data to generate the transformed test data 334. As a result of applying the add aggregate, the cardinality of the transformed test data 334 changes - in this case there are fewer rows, as rows representing multiple transactions for a cust_ID are collapsed into a single row representing the sum of the charge_amt for the cust_ID. To represent this change in cardinality, a new page 796 entitled "charge_amt" is added to the table view 321 of the user interface 322. The user can toggle between the charge_amt page 796 and the tab for the original "home" page 797 by clicking the tab associated with the page. The add aggregate transform is added as the second transform 778 (abbreviated "add aggregate") in the transform history view 323.

[0078] In Figure 7C the user then clicks the show profile button 591 to view the data profile for the transformed test data 334 shown in the table view 321. Referring back to Figure 7D , the user clicking the show profile button 591 causes the data profiler 337 to calculate profile data 339 for the transformed test data 334. A graphical representation of the profile data for each field in the transformed test data 334 is then displayed in the column for that field in the table view 321.

[0079] In Figure 7DIn the chart, a first graphical representation of profile data 798 is shown in the `cust_ID` column, and a second graphical representation of profile data 799 is shown in the `charge_amt` column. The first graphical representation 798 includes a histogram 701 showing that the `cust_ID` field has values ​​ranging from approximately 1000 to 2000, and that there are no duplicate values ​​(i.e., each unique `cust_ID` has "1" instance). The first graphical representation 798 also includes a data quality bar 702 showing that the field has high data quality (e.g., the data has almost no duplicate entries, almost no blank entries, and / or almost no entries with invalid or incorrect values). The second graphical representation of profile data 799 includes a histogram 703 showing that most customers have `charge_amt` values ​​less than ~$7,000, but there are some customers with `charge_amt` values ​​higher than that, up to ~$15,000. The second graphical representation of profile data 799 also includes a data quality bar 704 showing some minor data quality issues, indicated as areas with different patterns in the data quality bar. More generally, a data profile provides information about certain characteristics of the data. This information can be used to determine whether data transformations applied to the data are suitable for deployment. For example, a data profile can group customers into bins based on how much money a user has already spent. Developers can use data profiles to identify problems in the transformations (e.g., vulnerabilities). For example, if all customers fall into one bin, or the bin value is not what the developer expected, the developer can revisit their transformations for debugging. Data quality characterizes how complete and correct the data is. For example, data quality characterizes aspects of the data, such as duplicate records, records with missing fields, and records with incorrect data (e.g., spelling errors, invalid postal codes, invalid data, etc.). Developers can use data quality to measure the quality of the results generated by their transformations and correct their transformations as needed to address data quality issues.

[0080] exist Figure 7D In the middle, user 320 clicked the dropdown arrow (carat) 705 to explore details regarding a data quality issue with the charge_amt field. Reference Figure 7EUpon clicking the drop-down arrow 705, the user causes the display of a data profile view 706. The data profile view 706 includes a pie chart 707 of different charge_amt values and a summary 708 of values, including valid values, invalid values, NULL values, distinct values, unique values, and counts of maximum and minimum values. In this case, the user 320 can see that there are 50 invalid values and 5 NULL values, which are the source of the data quality issues shown in the data quality bar 704. In addition, the user 320 can see that there are 905 distinct values, 750 unique values, a maximum value of $14,995, and a minimum value of $10. After viewing the profile data 339, the user 320 can decide to modify the set of work transformations to adjust the data profile.

[0081] In some examples, a more complete characterization of the data quality is accessed by clicking the "Show Data Quality" button 592 of the user interface 322. A discussion of this characterization of the data quality is outside the scope of the present invention and is not further discussed herein.

[0082] 4 Alternatives

[0083] In some examples, the user can need to modify or remove a transformation from the set of work transformations 330. The user can do so in the user interface 322 by interacting with the list of transformations in the transformation history view 323 (e.g., by clicking a modify or remove button associated with a transformation in the list). In addition, there can be cases where the user wants to reorder the transformations in the transformation history view. In this case, the user can modify the order shown in the transformation history view, for example, by dragging the transformations.

[0084] It will be appreciated that the types of transformations described in the above examples are merely examples of transformations available in the user interface 322, and that other transformations can be available to the user.

[0085] In the above examples, the data being accessed by the data processing program is shown as a data set (e.g., a database or other data set stored on disk or in memory). However, it will be appreciated that a data stream can also be used to develop a data processing program and can be processed by the data processing program in a run-time setting.

[0086] The step of exporting the final set of transformations 324 can be a compilation step that translates the final set of transformations into a lower level programming language such as assembly language, object code, or machine code to create an executable program. Alternatively, the exporting step can translate the final set of transformations into the Ab Initio DML programming language, into an Ab Initio dataflow graph, or into an Ab Initio “EZ graph,” which is an easily modified and optimizable computational graph (described in U.S. Patent Publication 2021-0232579, the contents of which are incorporated by reference herein). Finally, the exporting step can export the set of transformations without any translation.

[0087] The data processing programs derived from the above-described iterative development method can be used not only to process work test data, but also to process other real-world data in both batch processing and streaming applications.

[0088] 5Implementation

[0089] The above-described computing resource allocation method can be implemented, for example, using a programmable computing system executing suitable software instructions, or it can be implemented in suitable hardware such as a field programmable gate array (FPGA), or in some hybrid form. For example, in a programmed approach, the software can include processes in one or more computer programs executing on one or more programmed or programmable computing systems (which can have various architectures such as distributed, client / server or grid, etc.), each including at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), at least one user interface (for receiving input using at least one input device or port, and for providing output using at least one output device or port). The software can include one or more modules of a larger program, for example, that provide services related to the design, configuration, and execution of data processing graphs. The modules of the program (e.g., elements of data processing graphs) can be implemented as data structures or other organized data conforming to a data model stored in a data repository.

[0090] Software can be stored in a non-transitory form for a sustained period of time (e.g., between refresh cycles of a dynamic memory device such as dynamic RAM) using the physical properties of the medium (e.g., surface pits and plateaus, magnetic domains, or charges), such as embodied in volatile or non-volatile storage media or any other non-transitory medium. When preparing to load instructions, the software can be provided on a tangible, non-transitory medium, such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or transmitted via a network communication medium (e.g., encoded with propagation signals) to the computing system executing the software. Some or all of the processing can be executed on a dedicated computer or using dedicated hardware, such as a coprocessor or field-programmable gate array (FPGA) or a specialized application-specific integrated circuit (ASIC). The processing can be implemented in a distributed manner, where different parts of the computation specified by the software are executed by different computing elements. Each such computer program is preferably stored or downloaded to a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) accessible by a general-purpose or special-purpose programmable computer, for configuring and operating the computer when the storage medium is read by the computer to perform the processes described herein. The system of the present invention can also be considered as a tangible, non-transitory medium configured with computer programs, wherein such a medium causes the computer to operate in a specific and predefined manner to perform one or more processing steps described herein.

[0091] Several embodiments of the invention have been described. However, it should be understood that the foregoing description is intended to be illustrative and not to limit the scope of the invention, which is defined by the appended claims. Therefore, other embodiments are also within the scope of the appended claims. For example, various modifications can be made without departing from the scope of the invention. Furthermore, some of the steps described above may be independent of the order and therefore may be performed in a different order than described.

Claims

1. A method for developing a reusable data processing program, the method comprising: accessing a plurality of input records; rendering a representation of the plurality of input records in one or more user interfaces; receiving a set of one or more data transformation steps; applying the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; rendering a representation of the plurality of transformed records in the one or more user interfaces; receiving a first user input while the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps; for each data transformation step in the one or more data transformation steps of the first user input: adding the data transformation step to the set of data transformation steps to update the set of data transformation steps, updating the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records, and rendering a representation of the updated plurality of transformed records in the one or more user interfaces; and receiving a second user input that causes the reusable data processing program to be exported, the exported program being based at least in part on the updated set of data transformation steps, the reusable data processing program being applicable to one or more sets of multiple records different from the plurality of input records.

2. The method of claim 1, wherein, The set of data transformation steps comprises a plurality of data transformation steps.

3. The method of claim 2, wherein, The plurality of data transformation steps are applied sequentially in an order specified by the user.

4. The method of claim 3, further comprising rendering a representation of the set of data transformation steps in the one or more user interfaces during development of the reusable data processing program.

5. The method of claim 4, wherein, The representation of the set of data transformation steps displays the data transformation steps in a list ordered according to an order specified by the user.

6. The method of claim 4, wherein, The representation of the set of data transformation steps comprises a dataflow graph representation of the data transformation steps.

7. The method of claim 4, further comprising receiving a third user input that causes one or more data transformation steps to be removed from the set of data transformation steps and the set of data transformation steps to be updated accordingly.

8. The method of claim 4, further comprising receiving a third user input that causes one or more data transformation steps in the set of data transformation steps to be modified and the set of data transformation steps to be updated accordingly.

9. The method of claim 1, wherein, The one or more user interfaces comprise a table interface, and the representation of the plurality of input records and / or the plurality of transformed records is rendered in the table interface in tabular form.

10. The method of claim 1, wherein, The one or more user interfaces further comprise a list interface, wherein the set of data transformation steps is rendered as a list in the list interface.

11. The method of claim 10, wherein, the set of data transformation steps rendered in the list interface is ordered according to an order in which the data transformation steps are applied to the plurality of input records.

12. The method of claim 11, further comprising receiving a third user input to change an order in which the set of data transformation steps are applied, and updating the list in the list interface accordingly.

13. The method of claim 11, further comprising using the list interface to interact with data transformation steps to modify the data transformation steps.

14. The method of claim 11, further comprising using the list interface to interact with data transformation steps to remove data transformation steps from the set of data transformation steps, and updating the list in the list interface accordingly.

15. The method of claim 1, wherein, the set of data transformation steps includes one or more of a filter data transformation step, an add field data transformation step, and a select field data transformation step.

16. The method of claim 1, wherein, the set of data transformation steps includes a filter data transformation step.

17. The method of claim 1, wherein, deriving the reusable data processing program includes compiling the updated set of data transformation steps to form the reusable data processing program.

18. The method of claim 1, wherein, deriving the reusable data processing program includes forming a dataflow graph representation of the updated set of data transformation steps to form the reusable data processing program.

19. The method of claim 1, further comprising computing a data profile for the updated plurality of transformed records, and rendering a representation of the data profile in the one or more user interfaces.

20. The method of claim 1, wherein, receiving the second user input upon determining that a data profile for the updated plurality of transformed records is consistent with a predetermined data profile.

21. The method of claim 20, wherein, the predetermined data profile or predetermined profile rule specifies an allowable range of some characteristic of the data profile.

22. The method of claim 1, further comprising computing a data quality of the plurality of transformed records, and rendering a representation of the data quality in the user interface.

23. The method of claim 22, wherein, the data quality includes at least one of a count of valid values, invalid values, null values, distinct values, unique values, and / or maximum and minimum values.

24. A system for developing a reusable data processing program, the system comprising: an interface to access a plurality of input records; a first output to render a representation of the plurality of input records in one or more user interfaces; a first input to receive a set of one or more data transformation steps; one or more processors to apply the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; a second output to render a representation of the plurality of transformed records in the one or more user interfaces; a second input to receive a first user input as the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps; The one or more processors are further configured to, for each data transformation step of the one or more data transformation steps of the first user input: add the data transformation step to the set of data transformation steps to update the set of data transformation steps, update the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records, and render a representation of the updated plurality of transformed records in the one or more user interfaces; and a third input to receive a second user input that causes the derivation of the reusable data processing program, the derived program based at least in part on the updated set of data transformation steps, the reusable data processing program applicable to one or more sets of records different from the plurality of input records.

25. A non-transitory computer readable medium storing instructions for causing a computing system to implement a method for developing a reusable data processing program, the instructions causing the computing system to: access a plurality of input records; render a representation of the plurality of input records in one or more user interfaces; receive a set of one or more data transformation steps; apply the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; render a representation of the plurality of transformed records in the one or more user interfaces; receive a first user input while the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps; for each data transformation step of the one or more data transformation steps of the first user input: add the data transformation step to the set of data transformation steps to update the set of data transformation steps, update the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records, and render a representation of the updated plurality of transformed records in the one or more user interfaces; and receive a second user input that causes the derivation of the reusable data processing program, the derived program based at least in part on the updated set of data transformation steps, the reusable data processing program applicable to one or more sets of records different from the plurality of input records.

26. A system for developing a reusable data processing program, the system comprising: means for accessing a plurality of input records; means for rendering a representation of the plurality of input records in one or more user interfaces; means for receiving a set of one or more data transformation steps; means for applying the set of data transformation steps to the plurality of input records to obtain a plurality of transformed records; means for rendering a representation of the plurality of transformed records in the one or more user interfaces; means for receiving a first user input while the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps; for each data transformation step of the one or more data transformation steps of the first user input: add the data transformation step to the set of data transformation steps to update the set of data transformation steps, update the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records, and render a representation of the updated plurality of transformed records in the one or more user interfaces; and receive a second user input that causes the derivation of the reusable data processing program, the derived program based at least in part on the updated set of data transformation steps, the reusable data processing program applicable to one or more sets of records different from the plurality of input records. means for receiving a first user input while the user manipulates the representation of the plurality of transformed records using the one or more user interfaces, the first user input comprising one or more data transformation steps; processing means configured to, for each data transformation step of the one or more data transformation steps of the first user input: add the data transformation step to the set of data transformation steps to update the set of data transformation steps, update the plurality of transformed records, including applying the updated set of data transformation steps to the plurality of input records to obtain an updated plurality of transformed records, and render a representation of the updated plurality of transformed records in the one or more user interfaces; and means for receiving a second user input that causes the derivation of the reusable data processing program, the derived program being based at least in part on the updated set of data transformation steps, the reusable data processing program being applicable to one or more sets of multiple records different from the plurality of input records.

27. A method for developing a reusable data processing program comprising a set of data transformation steps by displaying a set of records and iteratively enabling a user to select one or more data transformation steps, iteratively applying the data transformation steps to the records, and iteratively displaying transformed records, the method comprising: accessing a plurality of input records; rendering a representation of a plurality of transformed records in a user interface, the plurality of transformed records being determined by applying the set of data transformation steps to the plurality of input records; receiving a first user input while the user manipulates the representation of the plurality of transformed records using the user interface, the first user input comprising one or more data transformation steps; for each data transformation step of the one or more data transformation steps: adding the data transformation step to the set of data transformation steps, updating the plurality of transformed records, including applying the set of data transformation steps to the plurality of input records, and rendering a representation of the transformed plurality of transformed records in the user interface; and receiving a second user input that causes the derivation of the reusable data processing program based at least in part on the set of data transformation steps, the reusable data processing program being applicable to one or more sets of multiple records different from the plurality of input records.

28. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any preceding claim.

Images