Dataflow Graph Datasets

Abstract

Described herein is a technique performed by a data processing system for enabling efficient development of software application programs in a dynamic environment having multiple datasets by generating entries in a dataset catalog to provide software application programs with access to output data dynamically generated by a dataflow graph, the entries being associated with each software application program developed as a dataflow graph. The technique includes identifying a subgraph that, when executed, generates output data by applying one or more data processing operations to data obtained from one or more data sources, creating a new entry in the dataset catalog associated with the identified subgraph, the new entry being associated with information indicative of nodes, links, and configuration parameters of the identified subgraph, and configuring the dataset catalog to enable access to the new entry in the dataset catalog associated with the identified subgraph.

Description

[Technical field]

[0001] Related Applications This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 338,855, filed May 5, 2022, and entitled "DATAFLOW GRAPH DATASETS," and to U.S. Provisional Patent Application No. 63 / 432,615, filed December 14, 2022, and entitled "DATAFLOW GRAPH DATASETS," each of which is incorporated by reference in its entirety herein.

[0002] Aspects of the present disclosure relate to techniques for enabling efficient data analysis in dynamic environments with multiple datasets, where software application programs are developed as dataflow graphs that access datasets through a dataset catalog, allowing subgraphs of a dataflow graph to be saved as datasets that are accessible to other dataflow graphs through the dataset catalog. [Background technology]

[0003] Modern data processing systems manage vast amounts of data (e.g., millions, billions, or trillions of data records) and how this data can be accessed (e.g., created, updated, read, or deleted). Large institutions (e.g., multinational banks, global technology companies, etc.) may have millions of datasets. For example, a dataset may store transaction records, documents, tables, files, or any other suitable type of data. As another example, a dataset may store "metadata," which is data that contains information about other data (e.g., stored in the same data processing system and / or another data processing system) and / or processes (e.g., in the same data processing system and / or another data processing system). For example, a data processing system may store metadata about credit card transaction data stored in a table in a credit card company's database. Non-limiting examples of such metadata include the size of the table in memory, when the table was created, when the table was last updated, the number of rows and / or columns in the table, where the table is stored, who has permission to read, update, delete, and / or perform any other suitable action or actions on the table.

[0004] A data processing system may execute software application programs to support various functions. Software application programs may be used to provide functionality that supports an institution's processes. As part of performing such functions, software application programs may perform operations on data sets. For example, a bank may develop software application programs that support various aspects of its business, such as programs that generate credit reports, bank account histories, transaction reports, and / or other data. Software application programs may also be used to extract information from data sets. Summary of the Invention [Means for solving the problem]

[0005] Some embodiments provide a method performed by a data processing system to enable efficient development of software application programs in a dynamic environment having multiple datasets by using entries in a dataset catalog to provide software application programs developed as dataflow graphs with access to output data dynamically generated by one or more other dataflow graphs, the entries being associated with each software application program developed as a dataflow graph having nodes representing data processing operations and links representing flows of data. The method includes, using at least one computer hardware processor, providing a user interface by which a user can identify one or more entries in a dataset catalog associated with one or more respective cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph, the first cataloged dataflow graph having one or more nodes representing the one or more respective data sources and one or more nodes representing the one or more respective data processing operations, where when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, an identification of the first entry associated with the first cataloged dataflow graph; and configuring a dataflow graph of the software application program to receive as an input the output data generated when the first cataloged dataflow graph was executed, where configuring includes associating one of the input nodes of the dataflow graph with the first cataloged dataflow graph.

[0006] In some embodiments, receiving an identification of a first entry associated with the first cataloged dataflow graph includes receiving a selection of the first entry via a user interface.

[0007] In some embodiments, providing the user interface includes generating a graphical user interface having a searchable menu of one or more entries in the dataset catalog, and receiving an identification of a first entry associated with the first cataloged dataflow graph includes receiving user input via the user interface indicating a selection of the first entry in the searchable menu.

[0008] In some embodiments, the method includes executing a configured dataflow graph of the software application program. In some embodiments, executing the configured dataflow graph of the software application program includes executing a first cataloged dataflow graph to generate output data and providing the generated output data as input to the dataflow graph of the software application to perform at least one of the one or more data processing operations using the output data. In some embodiments, executing the configured dataflow graph results in execution of the first cataloged dataflow graph. In some embodiments, the output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

[0009] In some embodiments, the dataset catalog contains a number of entries associated with each cataloged dataflow graph and a number of entries associated with each dataset previously stored in memory.

[0010] In some embodiments, the user interface allows a user to identify at least one entry in the dataset catalog associated with at least one respective cataloged physical dataset previously stored in the memory, the at least one entry including a second entry associated with the physical dataset stored in the memory. In some embodiments, the one or more input nodes include a plurality of input nodes, and the method further includes receiving, via the user interface, an identification of the second entry associated with the physical dataset stored in the memory, and configuring a dataflow graph of the software application program to receive as an input data from the physical dataset, where configuring includes associating another one of the plurality of input nodes in the dataflow graph with data from the physical dataset.

[0011] In some embodiments, the method further includes transforming a data flow graph that includes an input node associated with the first cataloged data flow graph to obtain a transformed data flow graph, compiling the transformed data flow graph into a software application program, and executing the software application program. In some embodiments, transforming a data flow graph that includes an input node associated with the first cataloged data flow graph to obtain a transformed data flow graph includes incorporating the first cataloged data flow graph as a first sub-graph into the data flow graph at the input node associated with the first cataloged data flow graph, and transforming the first sub-graph to obtain a second sub-graph different from the first sub-graph.

[0012] In some embodiments, transforming the first subgraph to obtain the second subgraph includes transforming the first subgraph based at least in part on at least one operation represented by at least one node downstream of the input node in the dataflow graph. In some embodiments, transforming the first subgraph to obtain the second subgraph includes applying at least one optimization to the first subgraph to obtain the second subgraph. In some embodiments, the at least one optimization includes at least one of removing at least one node in the first subgraph, replacing at least one node in the first subgraph, reordering nodes in the first subgraph, merging nodes in the first subgraph, parallelizing processing of at least one operation represented by at least one node in the first subgraph, or deleting data of at least one node in the first subgraph such that the data is not used in a subsequent operation represented by a node downstream of the at least one node in the first subgraph. In some embodiments, the transforming includes identifying at least one portion of the dataflow graph to transform, where the at least one portion comprises a first cataloged dataflow graph associated with the input node, and transforming the at least one portion of the dataflow graph to obtain a transformed dataflow graph.

[0013] In some embodiments, the first cataloged dataflow graph is generated from a sub-graph embedded in another dataflow graph, the other dataflow graph having nodes representing data processing operations and links representing flows of data between the nodes, the other dataflow graph being separate from the dataflow graph of the software application. In some embodiments, the method further includes displaying a graphical representation of the other dataflow graph in the UI, receiving user input via the UI indicating that the sub-graph in the dataflow graph is to be cataloged, and saving the sub-graph as the first cataloged dataflow graph in response to receiving user input indicating that the sub-graph in the dataflow graph is to be cataloged.

[0014] In some embodiments, the first cataloged dataflow graph has only a single output link representing data output by the first cataloged dataflow graph by applying one or more data processing operations to data obtained from one or more data sources. In some embodiments, the first cataloged dataflow graph is stored in a data storage of the data processing system, and the first entry stores a reference to a location of the first cataloged dataflow graph in the data storage.

[0015] In some embodiments, the first entry stores a reference to a file that stores information indicative of nodes of the first cataloged dataflow graph and / or configuration parameters of the first cataloged dataflow graph.

[0016] In some embodiments, configuring a dataflow graph of the software application program to receive as input output data generated when the first cataloged dataflow graph is executed includes receiving, via a user interface, an association of the first entry with an input node in the dataflow graph, and in response to receiving user input associating the first entry with an input node in the dataflow graph, configuring the dataflow graph to receive at the input node data output via an output link of the first cataloged dataflow graph as a result of execution of the first cataloged dataflow graph.

[0017] In some embodiments, receiving the association of the first entry with the input node in the dataflow graph includes receiving user input via a user interface indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node in the dataflow graph. In some embodiments, the user input indicating the association of the first graphical element representing the first entry with the second graphical element representing the input node includes dragging the first graphical element to the second graphical element in the user interface.

[0018] Some embodiments provide a data processing system for enabling efficient development of software application programs in a dynamic environment having multiple datasets by using entries in a dataset catalog to provide software application programs developed as dataflow graphs with access to output data dynamically generated by one or more other dataflow graphs, the entries being associated with each software application program developed as a dataflow graph having nodes representing data processing operations and links representing flows of data.The system includes at least one computer hardware processor and, when executed by the at least one computer hardware processor, provides to the at least one computer hardware processor a user interface by which a user can identify, in a dataset catalog, one or more entries associated with one or more respective cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph, the first cataloged dataflow graph having one or more nodes representing one or more respective data sources and one or more nodes representing one or more respective data processing operations, and wherein the first cataloged dataflow graph is executed. and at least one non-transitory computer-readable storage medium having instructions stored thereon to cause the first cataloged dataflow graph to generate output data by applying one or more data processing operations to data obtained from one or more data sources; receiving, via a user interface, an identification of a first entry associated with the first cataloged dataflow graph; and configuring a dataflow graph of the software application program to receive as input the output data generated when the first cataloged dataflow graph is executed, where configuring includes associating one of the input nodes of the dataflow graph with the first cataloged dataflow graph.

[0019] In some embodiments, receiving an identification of a first entry associated with the first cataloged dataflow graph includes receiving a selection of the first entry via a user interface. In some embodiments, providing the user interface includes generating a graphical user interface having a searchable menu of one or more entries in the dataset catalog, and receiving an identification of the first entry associated with the first cataloged dataflow graph includes receiving user input via the user interface indicating a selection of the first entry in the searchable menu.

[0020] In some embodiments, the instructions further cause the at least one computer hardware processor to execute a dataflow graph of the software application program. In some embodiments, executing the dataflow graph of the software application program includes executing the first cataloged dataflow graph to generate output data and providing the generated output data as input to the dataflow graph of the software application to perform at least one of the one or more data processing operations using the output data. In some embodiments, executing the configured dataflow graph results in the execution of the first cataloged dataflow graph. In some embodiments, the output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

[0021] In some embodiments, the dataset catalog contains a number of entries associated with each cataloged dataflow graph and a number of entries associated with each physical dataset previously stored in memory.

[0022] In some embodiments, the user interface allows a user to identify at least one entry in the dataset catalog associated with at least one respective cataloged physical dataset previously stored in the memory, the at least one entry containing a second entry associated with the physical dataset stored in the memory. In some embodiments, the one or more input nodes include a plurality of input nodes, and the instructions further cause the at least one computer hardware processor to receive, via the user interface, an identification of the second entry associated with the physical dataset stored in the memory, and configure a dataflow graph of the software application program to receive as an input data from the physical dataset, where configuring includes associating another one of the plurality of input nodes in the dataflow graph with the data from the physical dataset.

[0023] In some embodiments, the dataset catalog includes a plurality of entries associated with each cataloged dataflow graph and a plurality of entries associated with each physical dataset previously stored in memory. In some embodiments, the user interface allows a user to identify at least one entry in the dataset catalog associated with at least one respective cataloged physical dataset previously stored in memory, the at least one entry including a second entry associated with the physical dataset stored in memory.

[0024] In some embodiments, the one or more input nodes include a plurality of input nodes, and the instructions further cause the at least one computer hardware processor to receive, via the user interface, an identification of a second entry associated with the physical dataset stored in the memory, and configure a dataflow graph of the software application program to receive as an input data from the physical dataset, where configuring includes associating another one of the plurality of input nodes in the dataflow graph with the data from the physical dataset.

[0025] In some embodiments, the instructions further cause the at least one computer hardware processor to transform a data flow graph that includes an input node associated with the first cataloged data flow graph to obtain a transformed data flow graph, compile the transformed data flow graph into a software application program, and execute the software application program. In some embodiments, transforming a data flow graph that includes an input node associated with the first cataloged data flow graph to obtain a transformed data flow graph includes incorporating the first cataloged data flow graph as a first sub-graph into the data flow graph at the input node associated with the first cataloged data flow graph, and transforming the first sub-graph to obtain a second sub-graph different from the first sub-graph.

[0026] In some embodiments, transforming the first subgraph to obtain the second subgraph includes transforming the first subgraph based at least in part on at least one operation represented by at least one node downstream of the input node in the dataflow graph. In some embodiments, transforming the first subgraph to obtain the second subgraph includes applying at least one optimization to the first subgraph. In some embodiments, the at least one optimization includes at least one of removing at least one node in the first subgraph, replacing at least one node in the first subgraph, reordering nodes in the first subgraph, merging nodes in the first subgraph, parallelizing processing of at least one operation represented by at least one node in the first subgraph, or deleting data of at least one node in the first subgraph such that the data is not used in a subsequent operation represented by a node downstream of the at least one node in the first subgraph. In some embodiments, the transforming includes identifying at least one portion of the dataflow graph to transform, where the at least one portion comprises a first cataloged dataflow graph associated with the input node, and transforming the at least one portion of the dataflow graph to obtain a transformed dataflow graph.

[0027] In some embodiments, the first cataloged dataflow graph was generated from a sub-graph embedded in another dataflow graph, the other dataflow graph having nodes representing data processing operations and links representing flows of data between the nodes, the other dataflow graph being separate from the dataflow graph of the software application. In some embodiments, the instructions further cause the at least one computer hardware processor to display a graphical representation of the other dataflow graph in the UI and to receive user input via the UI indicating a sub-graph within the dataflow graph.

[0028] In some embodiments, the first cataloged dataflow graph has a single output link that represents data output by the first cataloged dataflow graph by applying one or more data processing operations to data obtained from one or more data sources.

[0029] In some embodiments, the first entry stores a reference to a file that stores information indicative of nodes of the first cataloged dataflow graph and / or configuration parameters of the first cataloged dataflow graph.

[0030] In some embodiments, configuring a dataflow graph of the software application program to receive as input output data generated when the first cataloged dataflow graph is executed includes receiving, via a user interface, an association of the first entry with an input node in the dataflow graph, and in response to receiving user input associating the first entry with an input node in the dataflow graph, configuring the dataflow graph to receive at the input node data output via an output link of the first cataloged dataflow graph as a result of execution of the first cataloged dataflow graph.

[0031] In some embodiments, receiving the association of the first entry with the input node in the dataflow graph includes receiving user input via a user interface indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node in the dataflow graph. In some embodiments, the user input indicating the association of the first graphical element representing the first entry with the second graphical element representing the input node includes dragging the first graphical element to the second graphical element in the user interface.

[0032] Some embodiments provide at least one non-transitory computer readable storage medium having instructions stored thereon that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for enabling efficient development of software application programs in a dynamic environment having multiple datasets by using entries in a dataset catalog to provide software application programs developed as dataflow graphs with access to output data dynamically generated by one or more other dataflow graphs, the entries being associated with each software application program developed as a dataflow graph having nodes representing data processing operations and links representing flows of data.The method includes providing a user interface by which a user can identify one or more entries in a dataset catalog associated with one or more respective cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph, the first cataloged dataflow graph having one or more nodes representing the one or more respective data sources and one or more nodes representing the one or more respective data processing operations, where when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, an identification of the first entry associated with the first cataloged dataflow graph; and configuring a dataflow graph of the software application program to receive as an input the output data generated when the first cataloged dataflow graph was executed, where configuring includes associating one of the input nodes of the dataflow graph with the first cataloged dataflow graph.

[0033] Some embodiments provide a method performed by a data processing system for enabling efficient development of software application programs in a dynamic environment having multiple datasets by generating entries in a dataset catalog to provide software application programs with access to output data dynamically generated by a dataflow graph, the entries being associated with each software application program developed as a dataflow graph having nodes representing data processing operations and links representing flow of data. The method includes causing at least one computer hardware processor to: identify a sub-graph, the sub-graph having one or more nodes representing one or more respective data sources, one or more nodes representing one or more respective data processing operations, and output links representing data output by the sub-graph, where when the sub-graph is executed, the sub-graph generates the output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry in a dataset catalog associated with the identified sub-graph, the new entry being associated with information indicative of the nodes, links, and configuration parameters of the identified sub-graph; and configuring the dataset catalog to enable access to the new entry in the dataset catalog associated with the identified sub-graph for incorporating the sub-graph into a software application program.

[0034] In some embodiments, identifying the subgraph includes displaying a graphical representation of the dataflow graph in a user interface and receiving a first user input via the user interface indicating a subgraph in the dataflow graph. In some embodiments, the method further includes receiving a second user input via the user interface directing creation of a new entry associated with the indicated subgraph, the creation of the new entry associated with the identified subgraph occurring in response to receiving the second user input.

[0035] In some embodiments, the data processing system includes data storage that stores previously created dataflow graphs, and identifying the subgraph includes receiving user input via a user interface identifying the previously created dataflow graph as the subgraph. In some embodiments, the new entry contains information indicative of nodes, links, and configuration parameters of the identified subgraph. In some embodiments, configuring the dataset catalog to enable access to the new entry associated with the identified subgraph includes providing a user interface through which a user can identify a new entry associated with the identified subgraph in the dataset catalog.

[0036] In some embodiments, the one or more respective data sources include a physical dataset previously stored in memory, and the dataset catalog includes an entry associated with the physical dataset. In some embodiments, the data processing system includes a data storage, and the method further includes storing the sub-graph in the data storage of the data processing system, and creating the new entry includes storing in the new entry a reference to a location of the stored sub-graph in the data storage of the data processing system.

[0037] Some embodiments provide a data processing system for enabling efficient development of software application programs in a dynamic environment having multiple datasets by generating entries in a dataset catalog to provide software application programs with access to output data dynamically generated by a dataflow graph, the entries being associated with each software application program developed as a dataflow graph having nodes representing data processing operations and links representing flow of data. The system includes at least one computer hardware processor and at least one non-transitory computer-readable storage medium having instructions stored thereon that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to: identify a sub-graph of a data flow graph, the sub-graph having one or more nodes representing one or more respective data sources, one or more nodes representing one or more respective data processing operations, and output links representing data output by the sub-graph, where when the sub-graph is executed, the sub-graph generates the output data by applying the one or more data processing operations to data obtained from the one or more data sources; create a new entry in a dataset catalog associated with the identified sub-graph, the new entry being associated with information indicative of the nodes, links, and configuration parameters of the identified sub-graph; and configure the dataset catalog to enable access to the new entry in the dataset catalog associated with the identified sub-graph.

[0038] In some embodiments, identifying the sub-graph includes displaying a graphical representation of the dataflow graph in a user interface and receiving a first user input via the user interface indicating a sub-graph in the dataflow graph. In some embodiments, the instructions further cause the at least one computer hardware processor to receive a second user input via the user interface directing creation of a new entry associated with the indicated sub-graph, the creation of the new entry associated with the identified sub-graph being performed in response to receiving the second user input.

[0039] In some embodiments, the data processing system includes data storage that stores previously created dataflow graphs, and identifying the subgraph includes receiving user input via a user interface identifying the previously created dataflow graph as the subgraph. In some embodiments, the new entry contains information indicative of nodes, links, and configuration parameters of the identified subgraph. In some embodiments, configuring the dataset catalog to enable access to the new entry associated with the identified subgraph includes providing a user interface through which a user can identify a new entry associated with the identified subgraph in the dataset catalog.

[0040] In some embodiments, the one or more respective data sources include a physical dataset previously stored in memory, and the dataset catalog includes an entry associated with the physical dataset. In some embodiments, the data processing system includes a data storage, and the instructions further cause the at least one computer hardware processor to store the sub-graph in the data storage of the data processing system, and creating the new entry includes storing in the new entry a reference to a location of the stored sub-graph in the data storage of the data processing system.

[0041] Some embodiments provide at least one non-transitory computer readable storage medium having instructions stored thereon that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for enabling efficient development of software application programs in a dynamic environment having multiple datasets by generating entries in a dataset catalog to provide software application programs with access to output data dynamically generated by a dataflow graph, the entries being associated with each software application program developed as a dataflow graph having nodes representing data processing operations and links representing flows of data. The method includes identifying a sub-graph of a data flow graph, the sub-graph having one or more nodes representing one or more respective data sources, one or more nodes representing one or more respective data processing operations, and output links representing data output by the sub-graph, where when the sub-graph is executed, the sub-graph generates the output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry in a dataset catalog associated with the identified sub-graph, the new entry being associated with information indicative of the nodes, links, and configuration parameters of the identified sub-graph; and configuring the dataset catalog to enable access to the new entry in the dataset catalog associated with the identified sub-graph.

[0042] In some embodiments, identifying the sub-graph includes displaying a graphical representation of the dataflow graph in a user interface and receiving a first user input via the user interface indicating the sub-graph within the dataflow graph.

[0043] In some embodiments, the method further includes receiving a second user input via the user interface directing creation of a new entry associated with the indicated subgraph, wherein creation of the new entry associated with the identified subgraph is performed in response to receiving the second user input.

[0044] Some embodiments provide a method performed by a data processing system for providing a software application program developed as a dataflow graph having nodes representing data processing operations and links representing flows of data between the nodes, access to output data dynamically generated by one or more other dataflow graphs, the method including providing, using at least one computer hardware processor, a user interface configured to receive from a user an identification of one or more entries in a dataset catalog associated with one or more respective cataloged dataflow graphs having nodes representing data processing operations and links representing flows of data between the nodes, the one or more entries including a first entry associated with a first cataloged dataflow graph, the first cataloged dataflow graph having one or more input nodes representing one or more respective data sources, one or more nodes representing one or more respective data processing operations, and output links representing data output from the first cataloged dataflow graph, The method includes: when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates and provides output data at output links by applying one or more data processing operations to data obtained from one or more data sources; receiving, via a user interface, an identification of a first entry associated with the first cataloged dataflow graph; and configuring a dataflow graph of the software application program using the first entry to receive as input the output data dynamically generated when the first cataloged dataflow graph is executed, where configuring includes associating one of the input nodes in the dataflow graph of the software application program with the first cataloged dataflow graph.

[0045] In some embodiments, receiving an identification of a first entry associated with the first cataloged dataflow graph includes receiving a selection of the first entry via a user interface.

[0046] In some embodiments, providing the user interface includes generating a graphical user interface having a searchable menu of one or more entries in the dataset catalog, and receiving an identification of a first entry associated with the first cataloged dataflow graph includes receiving user input via the user interface indicating a selection of the first entry in the searchable menu.

[0047] In some embodiments, the method further includes executing the configured dataflow graph of the software application program. In some embodiments, executing the configured dataflow graph of the software application program includes executing the first cataloged dataflow graph to generate output data, and providing the generated output data as input to the dataflow graph of the software application for performing at least one of the one or more data processing operations using the output data.

[0048] In some embodiments, executing the configured dataflow graph results in execution of the first cataloged dataflow graph. In some embodiments, output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph. In some embodiments, executing the configured dataflow graph of the software application program includes compiling the configured dataflow graph of the software application program to obtain a compiled software application program and executing the compiled software application program.

[0049] In some embodiments, the method further includes maintaining a dataset catalog, the dataset catalog containing a plurality of entries associated with each cataloged dataflow graph and preferably a plurality of entries associated with each physical dataset previously stored in the memory.

[0050] In some embodiments, the user interface allows a user to identify at least one entry in the dataset catalog associated with at least one respective cataloged physical dataset previously stored in the memory, the at least one entry containing a second entry associated with the physical dataset stored in the memory. In some embodiments, the dataflow graph includes a plurality of input nodes containing the first input node, and the method further includes receiving, via the user interface, an identification of the second entry associated with the physical dataset stored in the memory, and configuring a dataflow graph of the software application program to receive as an input data from the physical dataset, where configuring includes associating a second input node of the plurality of input nodes in the dataflow graph with data from the physical dataset.

[0051] In some embodiments, the method further includes transforming a dataflow graph of the software application program that includes a first input node associated with the first cataloged dataflow graph to obtain a transformed dataflow graph. In some embodiments, the transforming includes incorporating the first cataloged dataflow graph into the dataflow graph at the first input node associated with the first cataloged dataflow graph, identifying at least one portion of the dataflow graph to transform, the at least one portion including the first cataloged dataflow graph incorporated at the first input node, and transforming the first cataloged dataflow graph incorporated at the first input node to obtain a transformed dataflow graph. In some embodiments, the transforming includes incorporating the first cataloged dataflow graph into the dataflow graph as a first sub-graph at the first input node associated with the first cataloged dataflow graph, and transforming the first sub-graph to obtain a second sub-graph different from the first sub-graph. In some embodiments, transforming the first subgraph to obtain the second subgraph includes transforming the first subgraph based at least in part on at least one operation represented by at least one node downstream of the first input node in the dataflow graph. In some embodiments, transforming the first subgraph to obtain the second subgraph includes applying at least one optimization to the first subgraph to obtain the second subgraph.

[0052] In some embodiments, the at least one optimization includes at least one of removing at least one node in the first subgraph, replacing at least one node in the first subgraph, reordering multiple nodes in the first subgraph, merging multiple nodes in the first subgraph, parallelizing processing of at least one operation represented by at least one node in the first subgraph, or deleting data of at least one node in the first subgraph such that the data is not used in a subsequent operation represented by a node downstream of the at least one node in the first subgraph.

[0053] In some embodiments, the method further includes compiling the transformed data flow graph into a software application program and executing the software application program.

[0054] In some embodiments, the first cataloged dataflow graph is generated from a sub-graph embedded in another dataflow graph, the other dataflow graph having nodes representing data processing operations and links representing flows of data between the nodes, the other dataflow graph being separate from the dataflow graph of the software application. In some embodiments, the method further includes displaying a graphical representation of the other dataflow graph in the UI and receiving user input via the UI indicating a sub-graph within the dataflow graph. In some embodiments, the output link is the only output link of the first cataloged dataflow graph.

[0055] In some embodiments, the first cataloged dataflow graph is stored in a data storage of the data processing system and the first entry stores a reference to a location of the first cataloged dataflow graph in the data storage. In some embodiments, the first entry stores a reference to a file that stores information identifying nodes of the first cataloged dataflow graph and / or operations at the nodes of the first cataloged dataflow graph.

[0056] In some embodiments, configuring the dataflow graph of the software application program to receive as input the output data generated when the first cataloged dataflow graph is executed includes receiving, via a user interface, an association of a first entry with a first input node in the dataflow graph of the software application program, and in response to receiving user input associating the first entry with the first input node in the dataflow graph of the software application program, configuring the dataflow graph of the software application program to receive at the first input node the data output via an output link of the first cataloged dataflow graph as a result of the execution of the first cataloged dataflow graph.

[0057] In some embodiments, receiving the association of the first entry with a first input node in the dataflow graph of the software application program includes receiving user input via a user interface indicating an association of a first graphical element representing the first entry with a second graphical element representing the first input node in the dataflow graph of the software application program. In some embodiments, the user input indicating the association of the first graphical element representing the first entry with the second graphical element representing the first input node includes dragging the first graphical element to the second graphical element in the user interface.

[0058] Some embodiments provide a data processing system for providing a software application program developed as a data flow graph having nodes representing data processing operations and links representing flows of data between the nodes, access to output data dynamically generated by one or more other data flow graphs.The data processing system includes at least one computer hardware processor and a user interface configured, when executed by the at least one computer hardware processor, to cause the at least one computer hardware processor to receive from a user an identification of one or more entries in a data set catalog associated with one or more respective cataloged dataflow graphs having nodes representing data processing operations and links representing flows of data between the nodes, the one or more entries including a first entry associated with a first cataloged dataflow graph, the first cataloged dataflow graph having one or more input nodes representing one or more respective data sources, one or more nodes representing one or more respective data processing operations, and output links representing data output from the first cataloged dataflow graph, and at least one non-transitory computer-readable storage medium having instructions stored thereon to cause the first cataloged dataflow graph, when executed, to generate output data at an output link by applying one or more data processing operations to data obtained from one or more data sources; receiving, via a user interface, an identification of a first entry associated with the first cataloged dataflow graph; and configuring a dataflow graph of the software application program using the first entry to receive as input the output data dynamically generated when the first cataloged dataflow graph is executed, where configuring includes associating one of the input nodes in the dataflow graph of the software application program with the first cataloged dataflow graph.

[0059] Some embodiments provide at least one non-transitory computer readable storage medium having stored thereon instructions that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing a software application program developed as a dataflow graph having nodes representing data processing operations and links representing flows of data between the nodes, access to output data dynamically generated by one or more other dataflow graphs, the method comprising: providing a user interface configured to receive from a user an identification of one or more entries in a dataset catalog associated with one or more respective cataloged dataflow graphs having nodes representing data processing operations and links representing flows of data between the nodes, the one or more entries including a first entry associated with a first cataloged dataflow graph, the first cataloged dataflow graph having one or more input nodes representing one or more respective data sources, one or more nodes representing one or more respective data processing operations, and output links representing data output from the first cataloged dataflow graph, the first cataloged dataflow graph, when executed, The method includes providing a first cataloged dataflow graph generating output data at an output link by applying one or more data processing operations to data obtained from one or more data sources; receiving, via a user interface, an identification of a first entry associated with the first cataloged dataflow graph; and configuring a dataflow graph of the software application program using the first entry to receive as input the output data dynamically generated when the first cataloged dataflow graph is executed, where configuring includes associating one of the input nodes in the dataflow graph of the software application program with the first cataloged dataflow graph.

[0060] The above is a non-limiting summary.

[0061] Various aspects and embodiments are described with reference to the following drawings, which it should be understood are not necessarily drawn to scale, and items that appear in more than one drawing are designated with the same or similar reference numerals in all the drawings in which they appear. [Brief description of the drawings]

[0062] [Figure 1] FIG. 1 is a diagram of a data processing system. [Figure 2A] FIG. 2 illustrates cataloging of subgraphs as a dataflow graph dataset in a data processing system in accordance with some embodiments of the techniques described herein. [Figure 2B] FIG. 2B illustrates the incorporation of the cataloged subgraph of FIG. 2A into a software application program in accordance with some embodiments of the techniques described herein. [Figure 3A] 1 is a software application program development user interface (UI) for a data processing system in accordance with some embodiments of the technology described herein. [Figure 3B] 3B illustrates identification of a sub-graph within the data flow graph of the software application program development UI of FIG. 3A in accordance with some embodiments of the techniques described herein. [Figure 3C] 3C illustrates a subgraph UI menu associated with the subgraph identified in FIG. 3B in accordance with some embodiments of the techniques described herein. [Figure 3D] 3C in accordance with some embodiments of the techniques described herein, illustrates cataloging the subgraph of FIG. 3B in a dataset catalog of a data processing system in response to user input received via the subgraph UI menu of FIG. [Figure 3E]3D illustrates a dataset catalog and data storage 202 of a data processing system updated by cataloging the subgraph in FIG. 3D according to some embodiments of the techniques described herein. [Figure 3F] 1 illustrates a software application program development UI with another data flow graph for another software application program in accordance with some embodiments of the techniques described herein. [Figure 3G] 3F with cataloged subgraphs associated with input nodes in accordance with some embodiments of the techniques described herein. [Figure 4A] 1 is a block diagram illustrating a dataset catalog of a data processing system in accordance with some embodiments of the techniques described herein. [Figure 4B] 1 is a block diagram of system modules of a data processing system in accordance with some embodiments of the techniques described herein. [Figure 4C] 4C illustrates interactions between the system modules of FIG. 4B in accordance with some embodiments of the techniques described herein. [Figure 4D] FIG. 2 illustrates a dataflow graph that includes a subgraph incorporated into the dataflow graph as an input, in accordance with some embodiments of the techniques described herein. [Figure 4E] FIG. 4E illustrates a transformation of the dataflow graph of FIG. 4D in accordance with some embodiments of the techniques described herein. [Figure 4F] FIG. 4E illustrates a transformed dataflow graph obtained after the transformation of FIG. 4D, according to some embodiments of the techniques described herein. [Figure 4G] FIG. 4B illustrates a dataset catalog entry of the dataset catalog of FIG. 4A associated with a physical dataset, according to some embodiments of the techniques described herein. [Diagram 5] 1 is an exemplary data flow graph of a software application program in accordance with some embodiments of the techniques described herein. [Figure 6] 1 is a flowchart of an exemplary process for configuring a software application program developed as a dataflow graph to receive as input output data dynamically generated by the dataflow graph, in accordance with some embodiments of the techniques described herein. [Figure 7] 1 is a flowchart of an example process for providing a software application program with access to output data dynamically generated by a dataflow graph in accordance with some embodiments of the techniques described herein. [Figure 8] 1 is a screenshot of a software application program development UI according to some embodiments of the technology described herein. [Figure 9] 9 is a screenshot of the UI of FIG. 8 including a dataflow graph, according to some embodiments of the techniques described herein. [Figure 10] 10 is a screenshot of a UI in which a portion of the dataflow graph of FIG. 9 has been configured as a subgraph, according to some embodiments of the techniques described herein. [Figure 11] 11 is a screenshot of a UI having a menu for saving the subgraph of FIG. 10 as a dataset accessible via a dataset catalog according to some embodiments of the techniques described herein. [Figure 12] 11 is a screenshot of a UI for configuring details of a dataflow graph dataset in accordance with some embodiments of the techniques described herein. [Figure 13] 1 is a screenshot of a UI for cataloging a dataflow graph according to some embodiments of the techniques described herein. [Figure 14] 14 is a screenshot of a UI having a dataflow graph incorporating the cataloged dataflow graph of FIG. 13 in accordance with some embodiments of the techniques described herein. [Figure 15]1 is a screenshot of a UI displaying information about a cataloged dataflow graph in accordance with some embodiments of the technology described herein. [Figure 16] 1 is a screenshot of a UI displaying output data generated when a cataloged dataflow graph is executed, according to some embodiments of the technology described herein. [Figure 17] 11 is a screenshot of a UI displaying information for a dataset catalog entry associated with a physical dataset, according to some embodiments of the technology described herein. [Figure 18] 11 is a screenshot of a UI displaying information for a dataset catalog entry associated with a dataflow graph dataset, according to some embodiments of the technology described herein. [Figure 19] FIG. 1 is a block diagram of an illustrative computing system that can be used in implementing some embodiments of the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0063] A data processing system may use one or more software application programs to process data. One or more of the software application programs utilized by the data processing system may be developed as a data flow graph. A data flow graph may contain (1) components called "nodes" or "junctions," which represent data processing operations performed on input data, and (2) "links" or "edges" between the components, which represent the flow of data. The nodes of a data flow graph may include one or more nodes representing respective input data sources, one or more output nodes representing respective output data sets, and / or one or more nodes representing data processing operations performed on the data.

[0064] To illustrate, Figure 5 is an example data flow graph 500 of a software application program in accordance with some embodiments of the techniques described herein. The data flow graph 500 receives data sets as inputs at nodes 502A, 502B. The data set input at node 502A is provided to a filtering operation at node 504, as shown by link 504A. The output of the filtering operation at node 504 is then provided as input to a sorting operation at node 506, as shown by link 506A. The data set input at node 502B is provided to a filter operation at node 508, as shown by link 508A. The output of the sorting operation at node 506 and the output of the filter operation at node 508 are then provided as inputs to a combine operation at node 550, as shown by links 550A, 550B. The output of the combine operation 550 is then provided to an output data sink 552, as shown by link 552A.

[0065] The inventors have developed techniques to allow software applications and / or portions thereof developed as data flow graphs to be saved and subsequently accessed by other software applications (e.g., developed as other data flow graphs) via a data catalog as a source of dynamic input data in the other software applications. The saved data flow graphs can be incorporated (e.g., as inputs) into one or more other software application programs (e.g., programs developed as data flow graphs or other types of software programs) through the use of a data catalog. This allows data dynamically generated by executing the cataloged data flow graph to be provided as inputs to one or more other software application programs. The catalog holds entries corresponding to the data flow graphs. These entries allow reuse of the data flow graphs in other software applications instead of requiring redevelopment and / or duplication of code in the software application. This allows for more efficient use of computing resources and thus efficient execution of data operations in dynamic environments with multiple data sets.

[0066] The term "dataflow graph" may refer to a dataflow graph of a software application or a portion of a given dataflow graph. A stored dataflow graph accessible via a data catalog may be referred to herein as a "cataloged dataflow graph" or a "dataflow graph dataset." A stored dataflow graph accessible via a data catalog has a single link (e.g., in some embodiments, only a single link) that represents data output by the dataflow graph. A dataflow graph that has only a single link that represents data output by the dataflow graph may be referred to herein as a "subgraph." A cataloged dataflow graph may be incorporated into one or more other dataflow graphs. Output data provided via an output link of a cataloged dataflow graph may be used in downstream operations of the dataflow graph in which the cataloged dataflow graph is incorporated.

[0067] FIG. 1 is a diagram of a data processing system 100. As shown in FIG. 1, the data processing system 100 includes a data storage 102 that stores datasets including datasets 102A and 102B. The data processing system 100 includes a dataset catalog 104 having entries corresponding to the datasets stored in the data storage 102. The entries include an entry 104A corresponding to the dataset 102A and an entry 104B corresponding to the dataset 102B. A user of the device 110 may develop software application programs in the data processing system 100 that perform operations using data that is part of one or more datasets stored in the data storage 102 or data derived from such datasets. The software application programs may be developed as data flow graphs. FIG. 1 shows software application programs 106A, 106B, 106C developed by users. The software application programs 106A, 106B, 106C operate on input datasets.

[0068] Entries 104A, 104B of the dataset catalog 104 may be used to incorporate datasets into a software application program. A user of the device 110 may use entries 104A, 104B to associate datasets 102A, 102B with input sinks of a dataflow graph. As shown in the example of FIG. 1, dataset 102A is included as an input to each of software application programs 106A, 106B, 106C using entry 104A of the dataset catalog 104. Each of the software application programs 106A, 106B, 106C performs one or more operations using data from dataset 102A. When one of the software application programs 106A, 106B, 106C is executed by the data processing system 100, the data processing system 100 may use data from dataset 102A to perform a set of operations indicated by the dataflow graph of the software application program. The data processing system 100 may generate output data as a result of the execution of the software application program.

[0069] Data processing system 100 may have numerous (e.g., hundreds or thousands) of software application programs developed as data flow graphs to perform data processing using data sets managed by the data processing system. The data sets may change over time (e.g., as a result of operations performed on data stored in the data sets). Additionally, users may frequently develop new data flow graphs for new software applications. For example, data processing system 100 may be used to manage a data set for a multinational bank. The multinational bank may develop thousands of data flow graphs to process customer data associated with millions of bank accounts. In another example, data processing system 100 may manage a data set for a credit card company. The credit card company may develop thousands of data flow graphs to process transaction data generated from millions of credit card transactions occurring per day.

[0070] The inventors recognize that a given dataflow graph (e.g., a standalone graph or a subgraph of a larger dataflow graph) may be useful as part of one or more other software application programs. For example, the output of the sort operation at node 506 in FIG. 5 may be useful in one or more other dataflow graphs of one or more other respective software application programs. Traditionally, a user would need to recreate a given dataflow graph in each of the other dataflow graphs in which the given dataflow graph is used. This requires a user to recreate the same dataflow graph multiple times to incorporate it into other dataflow graphs. For example, the same subgraph may be recreated tens, hundreds, or thousands of times by a user for different software application programs. Furthermore, if a dataflow graph needs to be updated, the user would need to update the dataflow graph in each of the different software application programs. Such traditional approaches are inefficient due to duplication of effort in recreating the dataflow graph.

[0071] One possible solution is to store the output data generated when a dataflow graph is executed for later use as input in other dataflow graphs. However, datasets managed by a data processing system may be frequently updated, deleted, or otherwise modified. For example, if data processing system 100 manages a dataset that stores information (e.g., customer information, account information, transaction information) for a multinational bank, the dataset may be frequently updated in the course of the bank's operations. As another example, if data processing system 100 manages a dataset that stores data (e.g., transaction history, account balance, credit utilization, etc.) for a credit card company, the dataset may be frequently updated based on millions of credit card transactions that occur daily. Furthermore, a dataflow graph may need to be modified after execution of the dataflow graph such that a subsequent execution of the dataflow graph results in different output data compared to the initial execution. Given the dynamic nature of data and dataflow graphs, output data generated by a dataflow graph when executed at one time may differ from output data generated when the dataflow graph is executed at another time. Thus, previously generated output data obtained from an execution of a dataflow graph may become outdated or stale. Furthermore, storing the output generated by executing a large number (e.g., thousands or millions) of dataflow graphs would require significant additional storage resources, making such an approach untenable.

[0072] Accordingly, the inventors have developed a data processing system that stores data flow graphs and enables them to be accessed by software application programs via a dataset catalog of the data processing system. A software application program (e.g., developed as a data flow graph) can access the data flow graph datasets via the dataset catalog in order to incorporate output data generated by the data flow graph at run time. Unlike previous approaches in which only datasets could be registered in the dataset catalog for access by software application programs, the inventors have developed a system in which the data flow graph itself can be registered in the dataset catalog and later accessed by software application programs via the dataset catalog. This allows cataloged data flow graphs to be efficiently incorporated into multiple software application programs using the dataset catalog.

[0073] The techniques described herein improve the efficiency of software application program development by allowing previously developed software to be easily incorporated into some applications. The cataloged data flow graph incorporated into the software application program becomes a subgraph embedded in the software application program that is compiled and executed along with the software application program. Execution of the subgraph generates data that can be used in downstream operations of the software application. Thus, the cataloged data flow graph allows the software application program to incorporate data generated at run time into the operation of the software application program.

[0074] The dataflow graph may be compiled into an executable software application program and then executed. In some embodiments, the data processing system may transform the dataflow graph prior to compilation to improve the execution efficiency of the software application program. If a cataloged dataflow graph is incorporated into the dataflow graph (e.g., as a subgraph), transforming the dataflow graph may include transforming the subgraph. The data processing system may transform the subgraph in a manner customized for the software application (e.g., based on operations in the dataflow graph downstream of the subgraph), resulting in a more efficient software application program.

[0075] 2A illustrates a data processing system 200 in which a dataflow graph may be registered in a dataset catalog 204 of the data processing system 200, according to some embodiments of the techniques described herein. As shown in the example of FIG. 2A, the data processing system 200 stores a subgraph 208 identified in a dataflow graph of a software application program 206A in a data storage 202 of the data processing system 200. The data processing system 200 further generates an entry 204A in the dataset catalog 204 associated with the cataloged dataflow graph 202A. The entry 204A may then be used by other software application programs (e.g., developed as a dataflow graph) to incorporate the cataloged dataflow graph 202A. In contrast to the data processing system 100 of FIG. 1, which contains only entries for physical datasets in the data catalog 104, the data processing system 200 of FIG. 2A may contain entries for both physical datasets and dataflow graph datasets that may be incorporated into a software application program.

[0076] As shown in FIG. 2A, the system developed by the inventors allows a user to identify a dataflow graph or a subgraph of a larger dataflow graph (e.g., in storage) and saves the identified dataflow graph as a dataflow graph dataset that can be accessed via the data catalog. For example, a user may determine that a particular subgraph in a dataflow graph may be useful in one or more other dataflow graphs. The system allows a user to identify a particular subgraph and then saves the subgraph as a dataflow graph dataset that can be accessed by software applications via the data catalog. In some embodiments, the system may allow a user to highlight a portion of the dataflow graph to indicate the subgraph and then provide an input via a UI menu indicating a command to save the highlighted portion as a dataflow graph dataset. The saved dataflow graph dataset can then be utilized like an input dataset in other software application programs (e.g., being developed as a dataflow graph). In some embodiments, the system may allow a user to provide an input indicating a command to make a previously saved subgraph into a dataflow graph dataset that can be accessed by software applications via the data catalog.

[0077] 2B illustrates how the entry 204A associated with the cataloged dataflow graph 202A in FIG. 2A can be used to incorporate the cataloged dataflow graph 202A into other software application programs 206B, 206C. As shown in the example of FIG. 2B, the cataloged dataflow graph 202A is incorporated into the software application programs 206B, 206C by associating the entry 204A with input nodes 208A, 208B of the dataflow graphs of the software application programs 206B, 206C. For example, the entry 204A may contain a reference to the dataflow graph 202A. The system 200 may be configured to use the entry 204A to access the input nodes 208A, 208B, respectively, of the dataflow graph 202A (e.g., prior to compiling the dataflow graph into an executable software application program). The incorporated dataflow graph may be compiled and executed as part of the software application program. The data output by the embedded dataflow graph is obtained by executing the operations indicated by the embedded dataflow graph as part of the execution of the software application. In this manner, data processing system 200 enables cataloged dataflow graphs to be used as a source of dynamic data that is generated during execution of a software application program.

[0078] As shown in FIG. 2B, a software application program (e.g., developed as a dataflow graph) may access the dataflow graph dataset via the dataset catalog to incorporate output data generated by the subgraph. Thus, the system allows a subgraph to be used repeatedly in multiple different software application programs without the user having to recreate the subgraph. Also, the output data generated by the subgraph when it is executed as part of a software application is current at the time the subgraph is executed. The cataloged subgraph may be further updated after its creation. The cataloged subgraph may be executed in its latest state when it is executed as part of a software application. Thus, the cataloged subgraph allows a software application program to incorporate the latest output data generated by the current version of the cataloged subgraph.

[0079] In some embodiments, the system may optimize the dataflow graph resulting from incorporating the cataloged subgraphs into a given dataflow graph. The system may analyze the structure of the resulting dataflow graph (e.g., by analyzing the structure of the cataloged subgraphs, other dataflow graphs, and how they are linked) to identify transformations that may be applied to the resulting dataflow graph such that a software application program obtained by compiling the dataflow graph may be more efficiently executed by a data processing system. As part of optimizing the resulting dataflow graph, the system may apply transformations to the subgraphs incorporated into the resulting dataflow graph. These transformations may be specific to the given dataflow graph into which the cataloged subgraphs were incorporated. After these transformations, the resulting subgraphs may have a different structure than the cataloged subgraphs that were initially incorporated into the given dataflow graph. The resulting subgraphs may be optimized for execution as part of the given dataflow graph.

[0080] For example, as part of optimizing a given dataflow graph to incorporate a cataloged subgraph, the system may transform the incorporated subgraph by removing redundant operations in the subgraph, reducing the amount of data read from the input data set, and / or parallelizing operations in the subgraph. The transformed subgraph may have a different structure than the cataloged subgraph. The system may then compile the given dataflow graph that includes the transformed subgraph to obtain an executable software application program. The software application program may run more efficiently than if it were obtained by compiling the given dataflow graph without the transformation of the incorporated subgraph.

[0081] In some embodiments, a data processing system may have a dataset catalog for providing access to data sets managed by the data processing system to a data flow graph of a software application program. The dataset catalog may have an entry associated with each dataset. A software application program being developed as a data flow graph may access an entry in the dataset catalog to incorporate the dataset as an input in the data flow graph. In some embodiments, the system may store a subgraph as a dataset by (1) identifying a subgraph (e.g., in the data flow graph or in storage), (2) creating an entry in the dataset catalog associated with the identified subgraph, where the entry is associated with information indicative of the subgraph, and (3) configuring the dataset catalog to enable access to the entry associated with the subgraph. The cataloged subgraph may then be used as a dataset in another data flow graph by accessing the entry associated with the subgraph from the dataset catalog.

[0082] In some embodiments, a subgraph may have a single link that represents the data output by the subgraph. This property of a subgraph allows the subgraph to be registered in a data catalog as a subgraph data set that can be used by a software application. The output data generated by the execution of the cataloged subgraph may be provided at the input node with which the entry associated with the subgraph is associated. When a dataflow graph having an input node associated with the subgraph is compiled into an executable software application program, the output link of the subgraph may be connected to the downstream node to which the input node is connected. The data generated by the execution of the subgraph is provided as an input to the dataflow graph in the compiled software application program.

[0083] In some embodiments, identifying a sub-graph of the dataflow graph may include (1) displaying a graphical representation of the dataflow graph (e.g., having graphical elements representing input data sets and operations) in a UI, and (2) receiving a first user input via the UI indicating a sub-graph within the dataflow graph. The system may receive user input via the UI directing the creation of an entry associated with the indicated sub-graph. The entry may include information indicative of information about the sub-graph, such as nodes, links, configuration parameters, and / or other information.

[0084] In some embodiments, the system may provide a UI (e.g., a dataset catalog UI) that allows a user to identify an entry associated with the identified subgraph in the dataset catalog (e.g., by selecting a graphical element that represents an entry from a list of elements that represent entries in the dataset catalog). Dataflow graphs of other software application programs may access output data dynamically generated by the subgraphs. The system may (1) receive, via the UI, an identification of an entry associated with the cataloged subgraph, and (2) configure a dataflow graph of another software application program to receive as input the output data generated when the cataloged subgraph is executed (e.g., by associating one of the input nodes of the dataflow graph with the cataloged subgraph).

[0085] In some embodiments, the system may provide a UI with a searchable menu of entries in the dataset catalog. The system may receive an identification of an entry associated with the cataloged subgraph by receiving user input via the UI indicating a selection of an entry from the searchable menu (e.g., from a list of search results). In some embodiments, the system may execute a dataflow graph of another software application program by (1) executing the cataloged subgraph to generate an output dataset and (2) providing the generated output dataset as an input to a dataflow graph for performing at least one of one or more data processing operations using data in the output dataset.

[0086] In some embodiments, the system may be configured to optimize the dataflow graph of the other software application prior to execution. The system may be configured to optimize the dataflow graph by transforming the dataflow graph to obtain an optimized dataflow graph. The system may be configured to transform the dataflow graph to obtain a transformed dataflow graph that is computationally more efficient to execute. For example, the system may remove redundant operations in the dataflow graph, parallelize one or more operations in the dataflow graph, partition data in the dataflow graph, and / or limit reads of input data to one or more fields used downstream in the dataflow graph. Transforming the dataflow graph may include transforming cataloged subgraphs that are incorporated into the dataflow graph (e.g., because the cataloged subgraphs are associated with input nodes of the dataflow graph). For example, the system may identify various portions of the dataflow graph that may be optimized. Those portions may include at least some of the cataloged subgraphs that are incorporated into the dataflow graph (e.g., by associating data catalog entries with input nodes of the dataflow graph). The system may transform some of the cataloged subgraphs as part of transforming the dataflow graph.

[0087] In some embodiments, a dataset may include data and information about the data. In some embodiments, the information about the data may be stored in attribute-value pairs. For example, a dataset may include one or more attributes having values, and the information about the data in the dataset may include the values ​​of the attributes. A dataset may be stored by a data processing system in any suitable format and / or using any suitable data structure or structures (although aspects of the technology described herein are not limited in this respect).

[0088] In some embodiments, the data processing system may manage datasets for an organization, such as, for example, a multinational corporation (e.g., a financial institution, a utility company, an automobile company, an electronics company, etc.) or other business or organization. A large organization may have a vast number of datasets, and thus, in some embodiments, the data processing system may be used to manage a large number of datasets (e.g., millions, billions, or trillions) for the organization. For example, in some embodiments, the data processing system may be configured to manage millions or billions of datasets. In some such embodiments, the data processing system may be used for metadata management in an enterprise environment, whereby a dataset stores information about other datasets (e.g., tables, transactions, documents, data records, etc.) stored across a globally distributed information technology (IT) infrastructure, including many databases, data warehouses, data lakes, etc. In this context, a dataset may store information about a corresponding object, such as, for example, when the object was created, where the object is stored, its size, the identity of one or more users authorized to edit the object, information identifying which application programs are using the object, information identifying the sensitivity level of the data, etc. Large organizations (e.g., financial institutions such as banks or credit card companies, public utilities such as telephone companies or electric companies, etc.) typically manage millions or billions of such data sets, and so there may be millions or billions of data sets that store information about such data sets managed by data processing systems. In this exemplary application, the data processing system stores information about other data (sometimes referred to as "metadata"), and so this exemplary application may be referred to as "metadata management." However, it should be understood that the techniques described herein are not limited to data processing systems used for metadata management, but may be applied to any data processing system that uses data sets to manage data, whether the data managed is metadata or any other type of data.

[0089] The techniques described herein may be implemented in any of a number of ways (the techniques are not limited to any particular manner of implementation). Detailed examples of implementations are provided herein for illustrative purposes only. Furthermore, the techniques disclosed herein may be used individually or in any suitable combination (aspects of the techniques described herein are not limited to the use of any particular technique or combination of techniques).

[0090] FIG 2A illustrates the cataloging of a subgraph as a dataflow graph dataset in a data processing system 200 in accordance with some embodiments of the techniques described herein. FIG 2A illustrates the identification of a subgraph in a software application program 206A and the saving of the identified subgraph as a cataloged dataflow graph in the data processing system 200. FIG 2B illustrates the subsequent incorporation of the cataloged dataflow graph into another software application program. The data processing system 200 includes a data storage 202 and a dataset catalog 204. The data processing system 200 is in communication with a client device 210.

[0091] The client device 210 may be any suitable type of computing device. For example, the client device 210 may be a desktop computer, a laptop computer, a smartphone, or a tablet. The data processing system 200 may provide a canvas on which a user can create and / or edit a dataflow graph. The data processing system 200 may be configured to provide the canvas as part of a UI that a user of the client device 210 may use to develop a software application program. The UI may include a display of a dataflow graph of the software application program. The UI may allow a user to modify the dataflow graph, catalog a subgraph of the dataflow graph, and / or incorporate a cataloged dataflow graph into the dataflow graph. The UI may include an interface that displays dataset catalog entries. The UI may allow a user to incorporate a cataloged dataset (e.g., a physical dataset and / or a dataflow graph dataset) using the interface. A user may select an entry from the interface to incorporate a dataset associated with the entry into the displayed dataflow graph. For example, a user may drag an entry from the interface to an input node of the dataflow graph to associate a cataloged dataset with the input node.

[0092] 2A , a user of device 210 is developing software application program 206A in data processing system 200. Software application program 206A is developed as a dataflow graph. The dataflow graph of software application program 206A encompasses sub-graph 208. In some embodiments, data processing system 200 may be configured to identify sub-graph 208 based on user input. For example, data processing system 200 may receive input via a GUI in which a user has identified sub-graph 208 embedded within the dataflow graph of software application program 206A.

[0093] 2A , the subgraph 208 has a single link that represents the output of the subgraph 208. Data output via the output link of the subgraph 208 may then be used in downstream operations. For example, data output by the subgraph 208 may be used in one or more downstream operations in the dataflow graph. The single output link of the subgraph 208 may enable the subgraph 208 to be registered as a dataflow graph dataset in the dataset catalog 204 and then incorporated into other software application programs (e.g., to provide input data to the software application programs).

[0094] As shown in the exemplary embodiment of FIG. 2A, data processing system 200 stores subgraph 208 as cataloged dataflow graph 202A in data storage 202 of data processing system 200. Dataflow graph 202A may be executed as part of a software application program to acquire data. This is in contrast to a physical data set that is composed of previously stored data. In some embodiments, data processing system 200 may be configured to generate a data record that stores dataflow graph 202A. For example, data processing system 200 may use a file configured for storing the dataflow graph. In this example, data processing system 200 may generate a file that stores dataflow graph 202A. The file may then be used to execute dataflow graph 202A (e.g., as part of the execution of software application program 206A or another software application program that incorporates dataflow graph 202A).

[0095] In some embodiments, data processing system 200 may be configured to save subgraph 208 in data storage 202 in response to a user command. Data processing system 200 may be configured to receive input via a GUI indicating a command to save subgraph 208 as a dataset. An example of how data processing system 200 may save a subgraph as a dataset is described herein with reference to FIGS. 3A-3G. In some embodiments, data processing system 200 may be configured to allow updates to dataflow graph 202A stored in data storage 202. For example, a user may modify dataflow graph 202A in data storage 202 to modify the operations performed by dataflow graph 202A.

[0096] As shown in FIG. 2A, the data processing system 200 registers the dataflow graph 202A in the dataset catalog 204. In some embodiments, the data processing system 200 may be configured to register the dataflow graph 202A in the dataset catalog 204 by generating an entry 204A corresponding to the dataflow graph 202A. The entry 204A may be used to incorporate the dataflow graph 202A into another software application program (e.g., as shown in FIG. 2B). In some embodiments, the entry 204A may provide a reference to the dataflow graph 202A that may be used to incorporate the dataflow graph 202A into another software application program. For example, the entry 204A may be used to associate the dataflow graph 202A with an input node of the dataflow graph.

[0097] 2B is a diagram illustrating the use of dataflow graph 202A in software application programs 206B, 206C of data processing system 200 in accordance with some embodiments of the techniques described herein. As shown in FIG. 2B, data processing system 200 incorporates dataflow graph 202A (which is a subgraph 208 of software application program 206A) into software application programs 206B, 206C using entry 204A. Data processing system 200 associates entry 204A with (1) input node 208A of software application program 206B and (2) input node 208B of software application program 206C. Data processing system 200 can be configured to execute dataflow graph 202A to generate data at input nodes 208A, 208B for use by the respective software application programs 206B, 206C.

[0098] For example, when software application program 206B executes, dataflow graph 202A may be executed to obtain data at input node 208A. Dataflow graph 202A is a sub-graph 208 from software application program 206A. Thus, the data at input node 208A may be output resulting from the execution of dataflow graph 202A at the time software application program 206B executes. Similarly, when software application program 206C executes, dataflow graph 202A may be executed to obtain data at input node 208B. Thus, the data at input node 208B may be output resulting from the execution of sub-graph 208 at the time software application program 206C executes.

[0099] 2A, in some embodiments, dataflow graph 202A may be updated. For example, dataflow graph 202A may initially be a copy of subgraph 208 from which dataflow graph 202A was created. A user may subsequently modify dataflow graph 202A in data storage 202 (e.g., to modify the operations performed by dataflow graph 202A). Thus, when dataflow graph 202A is incorporated into a software application program as a data set, dataflow graph 202A may be executed in its current form at the time the software application program is executed.

[0100] In some embodiments, data processing system 200 may be configured to compile a dataflow graph and execute the compiled dataflow graph. In some cases, data processing system 200 may be configured to transform a dataflow graph prior to compilation. Data processing system 200 may be configured to transform a dataflow graph to optimize execution of the dataflow graph. Data processing system 200 may apply a series of one or more transformations to the dataflow graph to obtain an optimized dataflow graph. Data processing system 200 may then compile and execute the optimized dataflow graph instead of the original dataflow graph. The compiled, optimized dataflow graph may be executed by data processing system 200 more efficiently than the compilation of the original dataflow graph. For example, the optimized dataflow graph may read less data than the original dataflow graph, parallelize certain operations of the original dataflow graph, and / or eliminate operations from the original dataflow graph.

[0101] As an illustrative example, data processing system 200 may transform the dataflow graph of software application program 206B and / or the dataflow graph of software application program 206C to obtain an optimized dataflow graph. Data processing system 200 may be configured to transform dataflow graph 202A as part of transforming the dataflow graphs of software application programs 206B, 206C (e.g., because dataflow graph 202A executes along with the execution of software application programs 206B, 206C). In some embodiments, data processing system 200 may be configured to transform dataflow graph 202A of a software application program based on the operation of the software application program. For example, data processing system 200 may transform dataflow graph 202A as part of transforming the dataflow graph of software application program 206B. In this example, dataflow graph 202A may be optimized for software application program 206B. As another example, data processing system 200 may transform dataflow graph 202A as part of transforming the dataflow graph of software application program 206C. In this example, dataflow graph 202A may be optimized for software application program 206C. Each of software application programs 206B, 206C may be compiled with a respective transformed version of dataflow graph 202A. In some cases, the transformation of dataflow graph 202A for one software application program may be different from the transformation of dataflow graph 202B for another software application program. In other cases, the transformation of dataflow graph 202A for the two software application programs may be identical.

[0102] The transformation performed by data processing system 200 may include optimizations to process the data according to one or more of the operations specified in dataflow graph 202A, compared to processing the data without the optimization or transformation, or both. For example, data processing system 200 adds one or more sort operations, data type operations, join operations including join operations based on keys specified in dataflow graph 202A, partition operations, automatic parallelization operations, or operations that specify metadata, among others, to generate a transformed dataflow graph that has the desired functionality of dataflow graph 202A. In some implementations, the dataflow graph is transformed into an optimized dataflow graph by applying one or more dataflow graph optimization rules to the dataflow graph to improve the computational efficiency of the transformed dataflow graph, compared to the computational efficiency of the dataflow graph before applying the optimizations. The dataflow graph optimization rules may include, for example, elimination of dead or redundant components, early filtering, or record narrowing, among others, as described herein with reference to transformation engine 412 of data processing system 200.

[0103] Figures 3A-3D show an exemplary sequence of steps for saving a subgraph of dataflow graph 306 as a cataloged dataflow graph in data processing system 200. Figure 3E shows data processing system 200 after the subgraph has been saved as a cataloged dataflow graph. Figures 3F-3G show an exemplary sequence of steps for incorporating a cataloged dataflow graph into another dataflow graph.

[0104] 3A is a software application program development UI 220 of the data processing system 200 shown on a display of a device 210 that interacts with the data processing system 200 according to some embodiments of the techniques described herein. The software application program development UI 220 displays a data flow graph 306 of the software application program. As shown in FIG. 3A, the software application program receives data from multiple input nodes 310A, 310B, 310C, 310D and generates data at an output node 312 by performing various data processing operations (e.g., filter, sort, join, etc.) shown at the nodes of the data flow graph 306. The software application program development UI 220 may be used by a user of the device 210 to generate the data flow graph 306. In some embodiments, the data flow graph 306 may be automatically generated by the data processing system 200. For example, the data flow graph 306 may be generated by the data processing system 200 based on a query entered by a user.

[0105] FIG. 3B illustrates the identification of a sub-graph 308 in the dataflow graph 306 in the software application program development UI 220 of FIG. 3A according to some embodiments of the techniques described herein. In some embodiments, the data processing system 200 may be configured to identify the sub-graph 308 based on user input. For example, the data processing system 200 may identify the sub-graph 308 based on user input highlighting a portion of the dataflow graph 306 (e.g., by clicking and dragging a box around the portion). In another example, the data processing system 200 may identify the sub-graph 308 based on user input clicking a node to be included in the sub-graph 308. In another example, the data processing system 200 may identify the sub-graph 308 based on user selection of a link marking an output of a portion of the dataflow graph 306. In the example of FIG. 3B, the identified sub-graph 308 is a portion of the dataflow graph 306 that performs data processing operations on data received at input nodes 310A, 310B to generate output data that is provided to a sorting operation at node 314.

[0106] FIG. 3C illustrates a subgraph UI menu 320 associated with the subgraph 308 identified in FIG. 3B according to some embodiments of the techniques described herein. The subgraph menu 320 may include one or more options associated with functions related to the subgraph 308. In the example of FIG. 3C, the subgraph UI menu 320 includes an option 320A to copy the subgraph, an option 320B to delete the subgraph, and an option 320C to save the subgraph as a cataloged dataflow graph. A user may select option 320C to save the identified subgraph 308 as a cataloged dataflow graph (e.g., in data storage 202). For example, a user may select option 320C by clicking, tapping, or providing another form of input indicating selection of option 320C. In some embodiments, a user may enter a command to save the subgraph 308 as a cataloged dataflow graph without using the subgraph menu 320. For example, a user may enter a shortcut key or select an icon in the software application program development UI that triggers saving the identified subgraph 308 as a cataloged dataflow graph.

[0107] In some embodiments, data processing system 200 may be configured to save sub-graph 308 in response to a user selection of option 320C. For example, data processing system 200 may be configured to save sub-graph 308 to a file stored in data storage 202 of data processing system 202. Data processing system 200 may further determine a reference (e.g., a URL, a file path, or other type of reference) to the saved file. Data processing system 200 may include the reference in an entry registered for the sub-graph dataset in data catalog 204.

[0108] In some embodiments, data processing system 200 may organize data catalog 204 into multiple applications and / or projects. Data processing system 200 may be configured to determine an application or project in which a cataloged dataflow graph is to be placed. For example, data processing system 200 may (1) provide a list of applications and / or projects in a GUI in which a user can specify an application and / or project, and (2) determine the application and / or project based on a user selection from the list. In some embodiments, data processing system 200 may be configured to determine a schema in which data output by a cataloged dataflow graph is to be stored. A schema may be an organizational container (e.g., a directory) that stores data sources. In some embodiments, data processing system 200 may be configured to (1) provide a list of schemas in a GUI in which a user can select a schema for a cataloged dataflow graph, and (2) determine the schema based on user input indicating a selection of one of the schemas.

[0109] In some embodiments, data processing system 200 may be configured to determine a name for the cataloged dataflow graph. For example, data processing system 200 may generate a GUI that includes a field for input of a name for the subgraph dataset. Data processing system 200 may name the cataloged dataflow graph as indicated in the field. Data processing system 200 may be configured to display the determined name in a data catalog entry list interface.

[0110] In some embodiments, the dataset catalog 204 can organize data into multiple collections (e.g., carts). The data processing system 200 can be configured to determine the collection to which a cataloged dataflow graph belongs. The data processing system 200 can be configured to provide a GUI in which a user can specify the collection. For example, the data processing system 200 can (1) provide a list of collections in the GUI and (2) determine the collection of the cataloged dataflow graph based on a user selection of a collection from the list of collections.

[0111] In some embodiments, data processing system 200 may be configured to determine a record format for data output from execution of a cataloged dataflow graph. The record format may describe the data output by the cataloged dataflow graph. For example, the record format may indicate the fields of the data, the format of the data within the fields, primary keys, foreign keys, and / or other information. In some embodiments, data processing system 200 may be configured to determine an inherited record format from an upstream dataset (e.g., an input dataset). If data processing system 200 does not identify an inherited record format, data processing system 200 may allow a user to specify a record format. For example, data processing system 200 may determine the record format using a user-specified file that indicates the record format. In some embodiments, data processing system 200 may be configured to (1) provide a GUI that allows a user to define the record format, and (2) determine the record format based on a user definition of the record format obtained using the GUI.

[0112] In some embodiments, data processing system 200 may be configured to determine one or more keys (e.g., primary and / or foreign keys) for the data. In some embodiments, data processing system 200 may be configured to determine inherited keys from upstream data. For example, data processing system 200 may use keys from one or more input data sets used by subgraph 208. In some embodiments, data processing system 200 may be configured to obtain a user-specified key if data processing system 200 does not identify an inherited key. For example, data processing system 200 may provide an interface that allows a user to specify a key.

[0113] 3D illustrates cataloging the subgraph 308 of FIG. 3B in the dataset catalog 204 of the data processing system 200 in response to user input received via the subgraph UI menu 320 of FIG. 3C according to some embodiments of the techniques described herein. As indicated by the arrow from the identified subgraph 308 to the dataset catalog 220, the data processing system 200 may be configured to save the subgraph 308 in the data storage 202 as a cataloged dataflow graph 202B. The data processing system 200 may be configured to generate an entry 204B in the dataset catalog 204 associated with the dataflow graph 202B. The entry 204B may then be used to incorporate the dataflow graph 202B into other software application programs (e.g., by associating the entry 204B with an input node of the dataflow graph).

[0114] 3E illustrates the dataset catalog 204 and data storage 202 of the data processing system 200 updated after cataloging the subgraph 308 in accordance with some embodiments of the techniques described herein. As shown in FIG. 3E, the data storage 202 now contains a cataloged dataflow graph 202B that is comprised of the subgraph 308 identified in the dataflow graph 306. The dataset catalog 204 includes an additional entry 204B associated with the cataloged dataflow graph 202B. The entry 204B may be used to incorporate the cataloged dataflow graph 202B into a software application program.

[0115] 3F illustrates the software application program development UI 220 with another data flow graph 326 on the display of the device 210 interacting with the data processing system 200 in accordance with some embodiments of the techniques described herein. As shown in FIG. 3F, the software application program development UI 220 includes a dataset catalog UI 324 that displays graphical elements representing the entries 204A, 204B, 204C, 204D, 204E of the updated dataset catalog 204 of FIG. 3E. A user of the device 300 can use the dataset catalog UI 324 to incorporate datasets of the data processing system 200 into the data flow graph 326 shown in the software application program development UI 220.

[0116] In some embodiments, the software application program development UI 220 may allow a user to associate a cataloged data set with an input node of the data flow graph 326. In the example of FIG. 3F, the user has selected entry 204B to associate with input node 328. For example, the software application program development UI 220 may allow a user to drag a graphical element representing entry 204B onto the input node 328 to associate the cataloged data flow graph 202B with the input node 328. In another example, the UI 302 may allow a user to select the input node 328 and then select the graphical element representing entry 204B to associate the cataloged data flow graph 202B with the input node 328.

[0117] FIG. 3G is the dataflow graph 326 of FIG. 3F with cataloged dataflow graph 202B associated with input node 328, according to some embodiments of the techniques described herein. Cataloged dataflow graph 202B is incorporated into dataflow graph 326 by associating entry 204B with input node 328. As described herein with reference to FIGS. 3A-3E, cataloged dataflow graph 202B is a subgraph 308 identified from dataflow graph 306. Thus, the data at input node 328 may be output data generated when cataloged dataflow graph 202B is executed. In some embodiments, the data processing system may be configured to compile and execute cataloged dataflow graph 202B or a transformation thereof when dataflow graph 326 is executed. For example, data processing system 200 may compile and execute cataloged dataflow graph or a transformation thereof when a software application program is compiled and executed. In some embodiments, the data processing system may independently compile and execute cataloged dataflow graph 202B. For example, data processing system 200 may locally compile and execute cataloged dataflow graph 202B in response to a user command, and thus data received at input node 328 may be data output by cataloged dataflow graph 202B.

[0118] In some embodiments, the cataloged dataflow graph may be compiled and executed in its current state at the time dataflow graph 326 is updated. In some cases, the cataloged dataflow graph 202B may be different from when it was originally saved as a data set. For example, the input data used in dataflow graph 202B may be different from when subgraph 308 was originally saved as dataflow graph 202B. As another example, one or more operations of dataflow graph 202B may have changed since dataflow graph 202B was created. Thus, the cataloged dataflow graph 202B may be the latest version at the time dataflow graph 326 was compiled and executed.

[0119] Figure 4A illustrates a block diagram of a data processing system 200 in accordance with some embodiments of the techniques described herein. As shown in Figure 2A, the data processing system 200 includes a dataset catalog 204, data storage 202, a software application program 206, and a system module 400. As shown in Figure 4A, the software application program 206 may access entries from the dataset catalog 204 to incorporate cataloged data flow graphs and / or datasets into the software application program 206.

[0120] The dataset catalog 204 includes entries for accessing each dataset (e.g., dataflow graph dataset and stored dataset) managed by the data processing system 200. In the example of FIG. 2A, the entries 202A, 202B, 202C, 202D, 202E may be used to access the datasets 202A, 202B, 202C, 202D, 202E from the data storage 202 of the data processing system 200 (e.g., by associating the dataset with an input node). In some embodiments, the dataset catalog 204 may include entries for accessing one or more datasets in a data storage external to the data processing system 200. The entries may include entries associated with physical datasets. For example, the entries 204C, 204D of the dataset catalog 204 are associated with respective physical datasets 202C, 202D. The entries may include entries associated with cataloged dataflow graphs. Each cataloged dataflow graph may include a dataflow graph that generates output data when executed. For example, entries 204A, 204B, 204C are associated with respective cataloged dataflow graphs 202A, 202B, 202E. Examples of entries in the dataset catalog 204 are described herein with reference to FIG.

[0121] In some embodiments, the dataset catalog 204 may be any suitable data structure for storing a collection of entries. For example, the dataset catalog 204 may be an array, where each element of the array is an entry in the dataset catalog 204. In another example, the dataset catalog 204 may be a hash table, where each entry contains a hashed set of information for accessing the dataset and is associated with a key that can be used to look up the entry.

[0122] In some embodiments, the dataset catalog 204 may be searchable. The data processing system 200 may enable a user of the data processing system 200 to search among the entries of the dataset catalog 204. For example, the data processing system 200 may provide a UI through which a user can enter a search query (e.g., keywords) that the data processing system 200 uses to identify one or more entries in the dataset catalog 204. In some embodiments, the data processing system 200 may be configured to search the contents of the entries in the data catalog 202 and / or the contents of the datasets in the data storage 202 to determine matches with the search query. For example, the data processing system 200 may perform a keyword search on the entries and / or the contents of the datasets.

[0123] In some embodiments, the dataset catalog 204 can provide software application programs 206 of the data processing system 200 with access to datasets as input data used by the software application programs. In the example of FIG. 4A, a physical dataset from the data storage 202 or output data generated when a cataloged dataflow graph is executed can be incorporated as input into a software application program using an entry in the dataset catalog 204. For example, to include data from a physical dataset 202C as input into a software application program (e.g., software application program 206A), the software application program can access entry 204C in the dataset catalog 204. Entry 204C can provide information to the software application program 206A to obtain dataset 202C. Entry 204C can contain a reference to dataset 202C in the data storage 202 that can be used by the software application program 206A to access dataset 202C. In another example, entry 204D can contain a reference to dataset 202D stored in the data storage 202 that can be used by the software application program 206A to access the dataflow graph. In some embodiments, entries in the dataset catalog 204 may be used to associate cataloged dataflow graphs with input nodes of a software application program. When the software application program is executed, it may use information from the entry to obtain data (e.g., by reading from a physical dataset or by executing the dataflow graph or a transformation thereof to generate output data).

[0124] Data storage 202 may be any suitable storage of data processing system 200. In some embodiments, data storage 202 may be comprised of storage hardware for storing datasets, dataflow graph information, and / or other information. For example, storage hardware may include one or more hard drives (e.g., disk drives, solid state drives, and / or other hard drives). In some embodiments, data storage 202 may include a distributed database. A distributed database may include data storage resources in multiple geographic regions. A distributed database may be comprised of one or more data centers that store datasets and dataflow graph information. In the example of FIG. 4A, dataset catalog 204 is shown separate from data storage 202, but in some embodiments, dataset catalog 204 may also be stored within data storage 202.

[0125] As shown in FIG. 4A, data storage 202 may store physical datasets, such as datasets 202C, 202D, and cataloged data flow graphs, such as data flow graphs 202A, 202B, 202E. A physical dataset may be a previously generated dataset. For example, a physical dataset may be a SQL table, an ORACLE database dataset, a TERADATA database dataset, a flat file, a multifile data store, a HADOOP database dataset, a DB2 data store, a Microsoft SQL SERVER dataset, an INFORMIX dataset, a table, or other type of dataset. In another example, a physical dataset may be a document that stores values ​​in one or more fields (e.g., structured or unstructured based on a schema). In another example, a physical dataset may be an XML file, a JSON file, or other type of storage file. In another example, a physical dataset may be an array, a linked list, a queue, or any other suitable data structure that stores data.

[0126] As shown in FIG. 4A, data processing system 200 includes software application program 206. Software application program 206D is developed as a dataflow graph. The dataflow graph of software application program 206D includes an input node (also called a "read component") configured to receive a dataset via dataset catalog 204. The dataflow graph is configured to receive as input (1) dataset 202C using entry 204C, and (2) output data generated when cataloged dataflow graph 202B is executed using entry 204B. The dataflow graph includes nodes where operations are performed on the input data to generate an output dataset. Software application program 206E is another software application program developed as a dataflow graph. The dataflow graph is configured to receive as input output data generated when dataflow graph 202E is executed using entry 204E. Although the example of FIG. 4A shows software application programs 206D, 206E, data processing system 200 may include any number of software application programs. For example, data processing system 200 may include hundreds or thousands of such software application programs.

[0127] In some embodiments, a software application program developed as a dataflow graph may be created manually by a user. For example, data processing system 200 may provide a UI (e.g., UI 220) that a user can use to develop a dataflow graph. In some embodiments, a software application program developed as a dataflow graph may be generated programmatically. For example, data processing system 200 may include a software application program that generates a dataflow graph.

[0128] In some embodiments, a software application program can access a physical dataset using an entry in the dataset catalog 204 by accessing the physical dataset using information in the entry. For example, a software application program can access the physical dataset using a reference (e.g., a URL, key, or other type of reference) to the physical location stored in the entry. In some embodiments, an entry associated with a physical dataset may contain a reference to a software application program for accessing the physical dataset. For example, the software application program may convert the physical dataset into a logical dataset that the software application program uses as input. In some embodiments, a software application program for accessing a physical dataset may be developed as a dataflow graph. For example, a software application program developed as a dataflow graph may perform various data processing operations to change the format of data from a physical dataset into a logical dataset for use by other software application programs. An example of an application for accessing a physical dataset using an entry in the dataset catalog is described in U.S. Patent Application Publication No. 2022 / 0245125, entitled "Dataset Multiplexer for Data Processing System," which is incorporated herein by reference in its entirety.

[0129] In some embodiments, a software application program can use the information in the entry to incorporate the cataloged dataflow graph. The entry can be used to configure the software application program to (1) execute the cataloged dataflow graph and (2) receive as input the output data generated when the cataloged dataflow graph is executed. For example, the output data generated from the execution of the cataloged dataflow graph can be used in one or more operations performed by the software application program.

[0130] In some embodiments, the data processing system 200 may be configured to configure the data flow graph of the software application to receive output data generated by the cataloged data flow graph using the entry. The data processing system 200 may be configured to configure the data flow graph by associating the cataloged data flow graph with an input node of the data flow graph. For example, the data processing system 200 may associate the cataloged data flow graph with an input node using a reference to information indicative of the cataloged data flow graph in the data storage 202. In another example, the entry may store the cataloged data flow graph and the data processing system 200 may associate the cataloged data flow graph from the entry with an input node. In some embodiments, the data processing system 200 may be configured to associate the cataloged data flow graph with an input node by including a reference to the cataloged data flow graph in the input node. When the cataloged data flow graph is associated with an input node of an incorporating data flow graph, the cataloged data flow graph may be executed as part of the incorporating data flow graph. The execution of the cataloged data flow graph may generate output data that the incorporating data flow graph can use for subsequent operations.

[0131] Although not shown in FIG. 4A , the data processing system 200 may also include software application programs that are not developed as dataflow graphs. In some embodiments, the entries in the dataset catalog 204 may be used by software application programs that are developed as dataflow graphs (e.g., software application programs 206D, 206E) and software application programs that are not developed as dataflow graphs. For example, the software application programs may perform operations using metadata about the datasets stored in the data storage 202. In this example, the software application programs may access information about the datasets from entries in the dataset catalog 204. In another example, the software application programs may be developed as software code and access the datasets using associated entries in the dataset catalog 204.

[0132] 4B illustrates a block diagram of system modules 400 of data processing system 200 in accordance with some embodiments of the techniques described herein. As shown in FIG. 4, system modules 400 include a dataflow graph generator 402, a dataset catalog module 404, a dataflow graph storage module 406, a software application development UI module 408, a dataset catalog UI module 410, a transformation engine 412, a compiler 414, and a dataflow graph execution engine 416.

[0133] The dataflow graph generator 402 may be configured to generate a dataflow graph for a software application program. In some embodiments, the dataflow graph generator 402 may be configured to generate the dataflow graph by obtaining user input indicating a dataflow graph via a graphical UI. A user may lay out nodes and links in the graphical UI that represent input data sources, data processing operations, outputs, and / or flows of data. In some embodiments, the dataflow graph generator 402 may be configured to automatically generate the dataflow graph. For example, the dataflow graph generator 402 may be configured to generate a dataflow graph based on a query. The dataflow graph generator 402 may automatically lay out nodes and links to perform the query.

[0134] In some embodiments, dataflow graph generator 402 may be configured to generate a dataflow graph. In some embodiments, dataflow graph generator 402 may be configured to generate a dataflow graph for an application by (1) obtaining a user definition of a dataflow graph (e.g., in a software application program development UI) and (2) generating a dataflow graph for the application based on the user definition. In some embodiments, dataflow graph generator 402 may be configured to save the user-defined dataflow graph as a software application program in data processing system 200. The software application program may be accessed and executed by data processing system 200 (e.g., to analyze data or perform processing as part of a task). In some embodiments, dataflow graph generator 402 may be configured to compile the dataflow graph into a software application program.

[0135] In some embodiments, the dataflow graph generator 402 may be configured to identify a subgraph in the dataflow graph. The dataflow graph generator 402 may be configured to identify the subgraph by identifying a portion of the dataflow graph as a subgraph. In some embodiments, the dataflow graph generator 402 may be configured to identify the portion of the dataflow graph based on user input that specifies the portion of the dataflow graph (e.g., as described herein with reference to FIG. 3B). For example, the dataflow graph generator 402 may be configured to save the identified subgraph as a cataloged dataflow graph, as described herein with reference to FIG. 3D.

[0136] In some embodiments, the dataset catalog module 404 may be configured to manage the dataset catalog 204 of the data processing system 200. The dataset catalog module 404 may be configured to manage the creation, deletion, and / or modification of entries in the dataset catalog 204. For example, the dataset catalog module 404 may create a new entry for a new dataset added to the data processing system. The dataset catalog module 404 may create a new entry by (1) instantiating a data object of the entry, (2) adding information to the data object (e.g., information described herein with reference to FIGS. 2B-2C), and (3) saving the data object as the new entry. The data catalog module 404 may also be configured to modify an existing entry. For example, the data catalog module 404 may update information for accessing a dataflow graph in an entry associated with a dataflow graph and / or update information about the dataflow graph stored in the entry.

[0137] In some embodiments, the dataset catalog module 404 may be configured to provide access to datasets (e.g., physical datasets and / or dataflow graph datasets). The dataset catalog module 404 may provide access to datasets to software application programs by entries associated with the datasets. For example, the dataset catalog module 404 may generate a dataset catalog UI menu that allows a user to select an entry for incorporating the associated dataset into the dataflow graph (e.g., as described herein with reference to FIG. 3F). In some embodiments, the dataset catalog module 404 may be configured to provide access to datasets by allowing a software application program to reference entries in the dataset catalog 204. For example, executable instructions of a software application program may reference entries in the dataset catalog 204 to incorporate a dataset. In another example, the dataset catalog module 404 may include one or more software application programs that provide information from entries in the dataset catalog to other software application programs.

[0138] In some embodiments, the dataflow graph storage module 406 may be configured to manage storage of the dataflow graph. The dataflow graph storage module 406 may be configured to save information indicative of the dataflow graph. For example, the dataflow graph storage module 406 may save information indicative of the nodes and links of the dataflow graph. The dataflow graph storage module 406 may further be configured to save configuration parameters of the dataflow graph. For example, the dataflow graph storage module 406 may save the name of the dataflow graph, the location of the dataflow graph (e.g., a file path), and / or other configuration parameters. In some embodiments, the dataflow graph generator 402 may be configured to generate a file that saves the dataflow graph. The file may save information indicative of the nodes and links of the dataflow graph. The file may indicate operations at the nodes of the dataflow graph. For example, the file may indicate one or more data processing operations (e.g., filter, join, sort, and / or one or more other operations) to be performed at the nodes of the dataflow graph. The file may further store information indicating one or more nodes, one or more data links, and / or input datasets associated with the data processing operations of one or more nodes. In some embodiments, the input nodes may obtain data from a physical dataset or data output by an executed subgraph (e.g., a cataloged dataflow graph incorporated as a subgraph). In some embodiments, an entry in the dataset catalog 204 may point to a file that stores information about the dataflow graph. The entry may be used to incorporate the dataflow graph into other dataflow graphs (e.g., in other software application programs).

[0139] In some embodiments, the dataflow graph storage module 406 may be configured to store the dataflow graph as a cataloged dataflow graph. The dataflow graph storage module 406 may be configured to generate an entry in the dataset catalog associated with the cataloged dataflow graph (e.g., as described herein with reference to FIG. 2A). The dataflow graph storage module 406 may be configured to catalog the dataflow graph by storing data records (e.g., documents, files, or other types of records) having information about the dataflow graph (e.g., information indicative of the nodes, links, and configuration parameters of the dataflow graph). The dataflow graph storage module 406 may be configured to store both the dataflow graph and its subgraphs as cataloged dataflow graphs. For example, the dataflow graph storage module 406 may store the entire dataflow graph as a cataloged dataflow graph with an associated entry in the dataset catalog. In another example, the dataflow graph storage module 406 may store a subgraph in the dataflow graph as a cataloged dataflow graph with an associated entry in the dataset catalog.

[0140] In some embodiments, the software application development UI 408 may be configured to generate a GUI that allows a user to develop a software application program as a dataflow graph. The GUI allows a user to lay out the nodes and links of the dataflow graph of the software application program. The GUI may allow a user to save the dataflow graph for execution. In some embodiments, the software application development UI module 408 may be configured to allow a user to identify a subgraph within the dataflow graph. For example, the GUI may allow a user to highlight a portion of the dataflow graph and enter a command to generate a subgraph from the highlighted portion. The GUI may further provide a menu through which a user can enter a command to save the subgraph as a cataloged dataflow graph (e.g., as described herein with reference to FIG. 3C). The data processing system 200 may save the subgraph as a cataloged dataflow graph in response to receiving the user input (e.g., as described herein with reference to FIGS. 3D-3E).

[0141] In some embodiments, the dataset catalog UI module 410 may be configured to generate a UI that displays a graphical element that represents an entry in the dataset catalog. In some embodiments, the dataset catalog UI module 410 may be configured to generate the UI as part of a GUI for development of a software application program as a dataflow graph. For example, the dataset catalog UI module 410 may generate a pane in the GUI that shows a list of entries. The dataset catalog UI module 410 may be configured to allow a user to access datasets through the UI. For example, the UI may allow a user to drag a graphical element that represents an entry to an input data source of the dataflow graph. As another example, the UI may allow a user to right-click a graphical element that represents an entry and select an option to incorporate the dataset associated with the entry into the dataflow graph. An example of a dataset catalog UI and how the dataset catalog UI may be used to incorporate a dataset into a dataflow graph is described herein with reference to FIGS. 3F-3G.

[0142] In some embodiments, the transformation engine 412 may be configured to transform the dataflow graph into a transformed dataflow graph that can be compiled and executed. The transformed dataflow graph may be computationally more efficient to execute. For example, the original data flow graph may, among other things, (1) contain nodes that represent redundant data processing operations, (2) require performing data processing operations whose results are not subsequently used, (3) require unnecessarily sequential processing when parallel processing is possible, (4) may apply data processing operations to more data than necessary to obtain the desired result, (5) may result in computations across multiple nodes that significantly increase the computational cost of performing the computations if the data processing for each data flow graph node is performed by a dedicated thread within a computer program, a dedicated computer program (e.g., a process within an operating system), or a dedicated computing device, (6) may require performing stronger types of data processing operations that require more computations (e.g., sort operations, rollup operations, etc.) when weaker types of data processing operations that require less computations (e.g., sort-within-groups operations, rollup-within-groups operations, etc.) would suffice, (7) may require duplication of processing work, or (8) may not contain operations or other transformations that are useful or necessary for processing the data, or combinations thereof.

[0143] Thus, the transformation engine 412 may be configured to apply one or more of the following transformations to the dataflow graph that are necessary to process the data according to the operations specified in the dataflow graph and / or that improve the processing of the data according to the operations specified in the dataflow graph compared to processing the data without the transformation: For example, a user may create a dataflow graph without having to specify low-level implementation details such as sorting and partitioning operations. However, these operations may be useful or required in the transformed dataflow graph to process the data according to the operations specified in the dataflow graph, may improve the processing of the data according to the operations specified in the dataflow graph (e.g., by increasing processing speed and / or reducing consumption of computing resources), or both. Thus, the transformation engine 412 may add one or more operations, such as a sorting operation, a data type operation, a join operation using a specified key, a partition operation, an automatic parallelism operation, or an operation that specifies metadata, among others, to the transformed dataflow graph to optimize or implement the operations specified in the dataflow graph. In some cases, at least one of the operations added to the transformed dataflow graph may not be present or represented in the original dataflow graph. The transformation engine 412 may remove certain operations (eg, redundant operations).

[0144] In some embodiments, the transformation engine 412 may be configured to add operations by inserting one or more nodes representing the added operations into the dataflow graph used to generate the transformed dataflow graph. In some embodiments, the transformation engine 412 may be configured to insert the added operations directly into the transformed dataflow graph without modifying the nodes of the dataflow graph. The transformation engine 412 may add these operations to all dataflow graphs when generating the corresponding transformed dataflow graph, may add these operations based on operations contained in the dataflow graph (which may be identified using pattern matching techniques, as described below), or may add these operations based on some other optimization rule.

[0145] In some embodiments, the transformation engine 412 may be configured to transform the dataflow graph by applying one or more dataflow graph optimization rules to the dataflow graph to improve the computational efficiency of the transformed dataflow graph, for example, by removing dead or redundant components (e.g., by removing one or more nodes corresponding to the dead or redundant components), by moving filtering steps earlier in the dataflow (e.g., by moving one or more nodes corresponding to the filtering components), or by narrowing down records, among other things. In this manner, the transformation engine 412 transforms the dataflow graph into an optimized, transformed dataflow graph prior to compilation.

[0146] In some embodiments, the transformation engine 412 may be configured to identify two adjacent nodes in the dataflow graph that represent respective operations, where a second operation duplicates or negates the effect of a first operation such that one of the operations is redundant. Thus, the transformation engine 412 may be configured to transform the dataflow graph by removing one or more nodes that represent redundant operations (e.g., nodes that represent duplicated or negated operations) in generating the transformed dataflow graph. For example, the transformation engine 412 may identify two adjacent nodes that have the same operation. Since two adjacent nodes that perform the same operation are generally redundant, one of the two adjacent nodes may be removed without the need to perform both of these operations. As another example, the transformation engine 412 may identify two adjacent nodes having a first node that represents a repartition operation (that partitions data for parallel processing on different computing devices) followed by a node that represents a serialization operation (that operates to consolidate all data for sequential processing by a single computing device). Because the effects of repartitioning are nullified by the subsequent serialization operation, there is no need to perform a repartitioning operation (e.g., the repartitioning operation is redundant) and the repartitioning operation can be removed by the conversion engine 412 during the conversion process.

[0147] In some embodiments, the transformation engine 412 may be configured to identify a first node representing a first operation to be exchanged with one or more other nodes representing other operations. When the first node is exchanged with one or more other nodes, the transformation engine 412 may transform the dataflow graph by reordering the first node with at least one of the one or more other nodes (e.g., by rearranging the order of the nodes). In this manner, the transformation engine 412 may transform the dataflow graph by ordering the nodes and corresponding operations in a manner that improves processing efficiency or speed, or in a manner that optimizes processing through the dataflow graph without changing the results. Furthermore, by exchanging nodes in this manner, the transformation engine 412 may be configured to apply other transformations. For example, the transformation engine 412 may reorder the first node with at least one of the one or more other nodes such that the first node representing the first sorting operation is positioned adjacent to the second node representing the second sorting operation. As a result, the first and second sorting operations become redundant, and the transformation engine 412 may transform the dataflow graph by removing either the first sorting operation or the second sorting operation (e.g., by removing the corresponding node from the dataflow graph when generating the transformed dataflow graph).

[0148] In some embodiments, the transformation engine 412 may be configured to identify and remove "dead" nodes that represent unused or unnecessary operations. For example, the transformation engine 412 may identify one or more nodes that represent operations whose results are not referenced or used (e.g., a sorting operation that is not referenced because the order produced by the sorting operation is not needed or relied upon in subsequent processing). Thus, the transformation engine 412 may be configured to transform the dataflow graph by removing the dead or unused operations (e.g., by removing the corresponding nodes when generating the transformed dataflow graph).

[0149] In some embodiments, the transformation engine 412 may be configured to perform a less-strength transformation on one or more nodes. For example, the transformation engine 412 may identify a first node representing a first operation of a first type (e.g., a first sort operation on a primary key, a first rollup operation on a primary key, etc.), followed by a second node representing a second operation of a second, weaker type (e.g., a second operation on a secondary key, a sort within group operation, a second rollup operation on a secondary key, a grouped rollup operation, etc.). Because processing data with the first operation may require more computing resources than processing data with the second, weaker operation, the transformation engine 412 may perform a less-strength transformation to replace the first operation with the second operation.

[0150] In some embodiments, the transformation engine 412 may be configured to transform the dataflow graph by aggregating two or more nodes. For example, the transformation engine 412 may identify separate nodes that represent operations that may be performed by different processes executing on one or more computing devices, and may transform the dataflow graph by aggregating the separate nodes and their respective operations into a single node such that all operations are performed by a single process executing on a single computing device, which may reduce inter-process (and potentially inter-device) communication overhead. The transformation engine 412 may further be configured to identify other nodes that may be merged, such as two or more separate join operations that may be merged, or a filtering operation that may be merged with a rollup operation, among other combinations.

[0151] In some embodiments, the transformation engine 412 may be configured to identify nodes configured to perform certain operations that may be more efficient if performed separately. The transformation engine 412 may be configured to perform a sequential-to-parallel transformation of the dataflow graph that separates one or more of the operations into separate nodes for parallel processing (e.g., an automatic parallel processing operation). The operations may then be performed in parallel using different processes running on one or more computing devices. The transformation engine 412 may then add a merge operation to merge the results of the parallel operations. In some embodiments, the transformation engine 412 may be configured to identify points in the dataflow graph that contain large chunks of data (e.g., data corresponding to large tables and indexes) and perform a partitioning transformation of the dataflow graph to separate the data into smaller partitions (e.g., an automatic partitioning operation). The partitions may then be processed sequentially or in parallel (e.g., by integrating the automatic partitioning operation with the automatic parallel processing operation). By reducing the size of the data being processed, or by isolating operations for parallel processing, or both, the transformation engine 412 can significantly improve the efficiency of the transformed dataflow graph.

[0152] In some embodiments, the transformation engine 412 may be configured to perform a width-reducing transformation when generating the transformed dataflow graph. For example, the transformation engine 412 may identify data (e.g., one or more data columns) that should be removed at a particular point in the dataflow graph prior to the performance of a subsequent operation because the data (e.g., the data to be removed) is not used in the subsequent operation and does not need to be propagated as part of the processing. As another example, a node in the dataflow graph may be configured to perform several operations, and the results of some of these operations may not be used. Thus, the transformation engine 412 may perform a width-reducing transformation that removes unused or unnecessary data (e.g., by inserting a node to remove the data at the identified point, by replacing a node configured to perform several operations with another node configured to perform only operations whose results are used, etc.). In this manner, the transformation engine 412 optimizes the dataflow graph by reducing the computational resources required by the dataflow graph to carry the data through subsequent operations (e.g., by reducing the network, memory, and processing resources utilized).

[0153] In some embodiments, transforming the dataflow graph may include transforming the cataloged dataflow graph that is incorporated into the dataflow graph as an input (e.g., by associating a data catalog entry associated with the cataloged dataflow graph with an input node of the dataflow graph). Prior to the dataflow graph undergoing the transformation, the cataloged dataflow graph may be incorporated into the dataflow graph. For example, the cataloged dataflow graph may be accessed using an entry associated with an input node of the subgraph and copied into the dataflow graph. The output links of the cataloged dataflow graph may further be connected to nodes of the dataflow graph (e.g., to which the output links of the input nodes are connected). The transformation engine 412 may be configured to transform the dataflow graph resulting from the incorporation of the cataloged dataflow graph. The transformation engine 412 may be configured to transform the incorporated cataloged dataflow graph as part of the transformation of the dataflow graph. Thus, the transformed cataloged dataflow graph may be optimized for the dataflow graph into which it is incorporated. The transformed cataloged dataflow graph may have a different structure than the cataloged dataflow graph that was originally incorporated into the dataflow graph. Furthermore, a cataloged dataflow graph incorporated into a first dataflow graph may have a different structure after undergoing a transformation than the same cataloged dataflow graph incorporated into a second dataflow graph that also undergoes a transformation.

[0154] To identify portions of a dataflow graph (e.g., cataloged dataflow graphs embedded in a dataflow graph) to which to apply one or more transformations, the transformation engine 412 may be configured to use a dataflow graph pattern matching language. The dataflow subgraph pattern matching language may include one or more expressions for identifying specific nodes or operations in the dataflow graph for optimization, as described in more detail below. For example, the pattern matching language may include expressions for identifying a series of nodes of at least a threshold length (e.g., at least 2, 3, 4, 5, etc.) that represent respective series of computations that can be combined using the combined operation optimization rules and that can be represented by a single node in the dataflow graph. Identifying such patterns may facilitate application of the combined operation optimization rules described above. A preferred, but non-limiting, example of one such expression is "A----+B-----+C----+D", which may be useful for identifying a series of four consecutive data processing operations that can be combined.

[0155] As another example, the pattern matching language may contain expressions for identifying portions of a dataflow graph where nodes of a particular type can be exchanged with other nodes in order to optimize the dataflow graph. This may facilitate the application of multiple different types of optimization rules to a dataflow graph. When the data transformation engine 412 determines that the order of one or more nodes in the dataflow graph can be changed without changing the processing result, this allows the data transformation engine 412 to consider changes to the structure of the dataflow graph (as permitted by the freedom provided by the exchange operations) in order to identify portions to which the optimization rules can be applied. As a result of considering the exchange-based changes, one or more optimization rules may become applicable to portions of the graph where those rules were not otherwise applicable.

[0156] For example, as described above, an optimization rule may involve identifying two adjacent nodes in an initial dataflow graph that represent respective sorting operations with a second sorting operation nullifying the effect of the first operation such that the first operation is redundant. By definition, such optimization rules do not apply to dataflow graphs that do not have adjacent nodes that represent sorting operations. However, if a first node representing the first sorting operation were to be swapped with one or more other nodes, it may be possible to reorder the first node with at least one of the one or more other nodes such that the first node representing the first sorting operation is positioned adjacent to a second node representing the second sorting operation. As a result of swapping nodes in this manner, an optimization rule may be applied to the dataflow graph that removes the redundant first sorting operation.

[0157] Thus, in some embodiments, the pattern matching language may include one or more expressions for identifying subgraphs of a dataflow graph under circumstances where order nodes in the dataflow graph may be changed. As an example, the expression "A---+(...)--+B" (each of A and B may be any suitable data processing operation, such as sorting, merging, etc.) may be used to find a portion of a dataflow graph having node "A" (e.g., a node representing operation "A") and node B (representing operation B), and one or more nodes between node A and node B to be swapped with node A (e.g., when the order of the nodes is swapped, the results of the processing performed by these nodes do not change). Once such a portion is identified, the dataflow graph may be altered by moving node A adjacent to node B to obtain portion "AB". As a specific example, if the dataflow graph has nodes ACDB, and if operation A is swapped with operations C and D, the dataflow graph may be altered to become "CDAB". The data transformation engine 412 may then consider whether an optimization rule applies to portion "AB". For example, if operation A is a sort and operation B is a sort, data transformation engine 412 can attempt to determine whether these two sorts can be replaced with a single sort to optimize the dataflow graph.

[0158] As another example, the expression "A---+(...)-----+B*" may be used to find a portion of a dataflow graph having a node A, a second node B, and one or more nodes between these nodes that are swapped with node B. As a particular example, if a dataflow graph has nodes ACDB, and if operation B is swapped with operations C and D, the dataflow graph may be modified to become "ABCD." The data transformation engine 412 may then consider whether an optimization rule applies to portion "AB."

[0159] As another example, the expression "A---+(...)-----+B**" may be used to find the portion of a dataflow graph having node A, node B, and one or more nodes between node A and node B (e.g., C and D) that are not swapped with node B. In this case, the system may attempt to do a "brute force" swap (nodes C and D are sent to the left of node A, if possible). As a specific example, if a dataflow graph has node ACEDB, and if operation B is swapped with operation E, but not with operations C and D, then the dataflow graph may be modified to become "CDABE" (B is swapped with E, but C and D are sent to the left of A).

[0160] As yet another example, the expression "A**-----+(...)-----+B" may be used to find the portion of a dataflow graph having node A, node B, and one or more nodes between node A and node B (e.g., C and D) that are not swapped with node A. In this case, the system may attempt to do a "brute force" swap (where nodes C and D are sent to the right of node B, if possible). As a specific example, if a dataflow graph has node ACEDB, and if operation A is swapped with operation E, but not with operations C and D, then the dataflow graph may be modified to become "EABCD" (node ​​A is swapped with E, but C and D are sent to the right of B).

[0161] In some embodiments, the transformation engine 412 may be configured to iteratively transform the dataflow graph, with each iteration of optimization or transformation transforming the dataflow graph until testing indicates that further optimization or transformation is not possible, necessary, or desirable. For example, the transformation engine 412 may transform the dataflow graph by (1) selecting a first optimization rule, (2) identifying a first portion of the dataflow graph to which to apply the first optimization rule, and (3) applying the first optimization rule to the first portion of the dataflow graph. Later, the transformation engine 412 may determine whether another one or more additional optimizations are applicable to the dataflow graph or are necessary to generate a transformed dataflow graph that can be compiled and executed. If additional optimizations are applicable, the transformation engine 412 may continue to update the dataflow graph by (1) selecting a second optimization rule different from the first optimization rule, (2) identifying a second portion of the dataflow graph to which to apply the second optimization rule, and (3) applying the second optimization rule to the second portion of the dataflow graph. In the absence of further optimizations or transformations, the transformation engine 412 may be configured to output the transformed data flow graph for storage, and / or to output the transformed data flow graph to a compiler module 414, which compiles the transformed data flow graph into an executable software application program (which may be executable, for example, by the execution engine 416).

[0162] Techniques for optimizing data flow graphs that may be used by the transformation engine 412 are described in U.S. Patent Application Publication No. 2021 / 0232579, entitled “Editor for Generating Computational Graphs,” the entirety of which is incorporated by reference herein.

[0163] In some embodiments, the transformed dataflow graph generated by the transformation engine 412 may be stored in the data storage 202 (e.g., by the dataflow graph storage module 406). In some embodiments, the transformation engine 412 may be configured to transform the dataflow graph and store the dataflow graph for later compilation (e.g., by the compiler module 414). In some embodiments, the transformation engine 412 may be configured to transform the dataflow graph as part of compiling the dataflow graph. For example, the transformation engine 412 may be used when a given dataflow graph is being compiled.

[0164] In some embodiments, compiler module 414 may be configured to compile the dataflow graph (e.g., the transformed dataflow graph) for execution (e.g., by dataflow graph execution engine 416). Compiler module 414 may be configured to compile the dataflow graph into an executable software application program that can be executed by data processing system 200. In some embodiments, compiler module 414 may be configured to store the compiled software application program in data storage of data processing system 200. The stored software application program may then be executed by data processing system 200 at a later time. For example, the software application program may be executed in response to a user command and / or may be executed programmatically.

[0165] In some embodiments, the dataflow graph execution engine 416 may be configured to execute a dataflow graph (e.g., a compiled dataflow graph) of a software application program. In some embodiments, the dataflow graph execution engine 416 may be configured to execute the dataflow graph by (1) generating a set of instructions based on the dataflow graph (e.g., the nodes and links of the dataflow graph) and (2) executing the set of instructions. In some embodiments, the dataflow graph execution engine 416 may be configured to use a software application program that interprets and executes the dataflow graph. For example, the dataflow graph execution engine 416 may invoke a program that interprets the dataflow graph and generates computer-executable instructions based on the dataflow graph. Techniques for executing computations encoded by dataflow graphs are described in U.S. Pat. No. 5,966,072, entitled "Executing Computations Expressed as Graphs," and U.S. Pat. No. 7,716,630, entitled "Managing Parameters for Graph-Based Computations," each of which is incorporated herein by reference in its entirety.

[0166] In some embodiments, the dataflow graph execution engine 416 may be configured to generate output data obtained as a result of the execution of the dataflow graph. The dataflow graph execution engine 416 may be configured to execute the dataflow graph of the dataflow graph dataset to generate output data (e.g., as part of the execution of a software application program). The output data may then be used by the software application program for subsequent data processing. For example, a software application program may be developed as a first dataflow graph, and output data generated by executing a dataflow graph from the dataflow graph dataset may be used to perform one or more data processing operations in the first dataflow graph.

[0167] 4C is a diagram illustrating interactions between the system modules of FIG. 4B in accordance with some embodiments of the techniques described herein. As shown in FIG. 4C, system module 400 performs graph generation 420 and graph execution 430.

[0168] The software application development user interface module 408 may enable a user (e.g., of the device 210) to develop a software application program as a dataflow graph (e.g., in a graphical development environment). The dataflow graph generator 402 may generate a dataflow graph based on a user definition in the GUI. The GUI may further enable a user to save a subgraph of the dataflow graph as a cataloged dataflow graph. The dataset catalog module 404 may register the subgraph in the dataset catalog 204 (e.g., by generating an entry in the dataset catalog 204 corresponding to the dataflow graph dataset stored in the data storage 202). The dataset catalog UI module 410 may display an entry corresponding to the cataloged dataflow graph in a dataset catalog UI that may be displayed in the software application program development GUI (e.g., as shown in FIG. 3F).

[0169] 4C, the dataflow graph generated by the dataflow graph generator 402 may be stored in the data storage 202 of the data processing system 200 by the dataflow graph storage module 406. In some embodiments, the transformation engine 412 may be configured to transform the dataflow graph prior to storage. Thus, the dataflow graph storage module 406 may be configured to store the transformed dataflow graph in the data storage 202. In some embodiments, the generated dataflow graph may be stored as it was originally generated (e.g., without transformation by the transformation engine 412).

[0170] A dataflow graph may be transformed by a transformation engine 412 to obtain a transformed dataflow graph (e.g., an optimized version of another dataflow graph). A compiler 414 may compile the transformed dataflow graph generated by the transformation engine 412 to obtain a compiled software application program (e.g., an executable program). An execution engine 416 may then execute the compiled software application program.

[0171] 4D is a diagram illustrating a dataflow graph 450 that includes a subgraph 452 that is incorporated into the dataflow graph 450, according to some embodiments of the techniques described herein. In the example of FIG. 4D, the subgraph 452 is a cataloged dataflow graph 202A that is incorporated into the dataflow graph 450 using an entry 204A in a data set catalog 204 of a data processing system 200. As shown in FIG. 4D, the subgraph 452 includes two input nodes 452A, 452E from which the subgraph 452 receives input data. The input data from node 452A is provided to node 452B, where a filter operation is performed, and the output of the filter operation is provided to node 452C, where a sort operation is performed. The input data from node 452E is provided to node 452F, where a sort operation is performed. The outputs of nodes 452C, 452F are provided as inputs to node 452D, where a join operation is performed. The outputs of subgraph 452 are then provided as inputs to nodes of dataflow graph 450 where a combine operation is performed with the results of other operations performed in dataflow graph 450 .

[0172] FIG. 4E is a diagram illustrating the transformation of dataflow graph 450 by transformation engine 412, according to some embodiments of the techniques described herein. Transformation engine 412 may be configured to identify a portion of dataflow graph 450 to transform. In the example of FIG. 4E, transformation engine 412 has determined that a portion of subgraph 452 can be transformed to optimize the execution of dataflow graph 450. Transformation engine 412 has determined that data from input node 452E is not needed in dataflow graph 450. For example, transformation engine 412 may determine that data from input node 452E is not used in downstream operations of dataflow graph 450. As such, operations involving this data may be removed from dataflow graph 450 because they do not need to be executed. Additionally, because data from input node 452E is not needed, transformation engine 412 determines that a join operation at node 452D to combine the output of node 452C with the output of node 452F is no longer needed. Therefore, transformation engine 412 removes nodes 452E, 452F, and 452D from subgraph 452.

[0173] FIG. 4F illustrates a transformed dataflow graph 460 obtained after the transformation of FIG. 4E according to some embodiments of the techniques described herein. As shown in FIG. 4F, the transformed dataflow graph 460 includes a transformed subgraph 462 obtained by applying a transformation to the subgraph 452 that was originally incorporated into the dataflow graph 450. The transformed subgraph 462 includes only nodes 452A, 452B, 452C of the original subgraph 452. Thus, the transformed subgraph 462 includes fewer execution operations compared to the subgraph 452. Because the transformed subgraph 462 includes fewer operations than the subgraph 452, the transformed subgraph 462 may be executed more efficiently than the subgraph 452. The transformed dataflow graph 460 may be executed more efficiently than the original dataflow graph 450.

[0174] FIG. 4G illustrates a diagram showing an example dataset catalog entry 440 of the dataset catalog 204 of FIG. 2A in accordance with some embodiments of the techniques described herein. As shown in FIG. 4G, the entry 440 includes information 442 for accessing the dataset. In some embodiments, the information 442 for accessing the dataset may include a program for converting a physical dataset into a logical dataset. For example, the program may be a stored data flow graph that, when executed, accesses and converts the physical dataset 202C. In some embodiments, the information 442 for accessing the dataset may be a reference for accessing the dataset. For example, the information 442 may include a uniform resource locator (URL) for accessing the dataset. In another example, the information 442 may include a location path (e.g., a file location path) for accessing the dataset.

[0175] The entry 440 further includes a logical identifier 444 that uniquely identifies the entry 440 among other entries in the dataset catalog 204. For example, the logical identifier 444 may be an alphanumeric value that is unique for the entry 440.

[0176] As shown in FIG. 4G, in some embodiments, the dataset catalog entry 440 may include data store information 446. The data store information 446 may include a reference to the location of the dataset in the data storage 202. For example, the reference to the location may be an identifier of the dataset in the data storage 202. The data store information 446 may include an indication of the type of data store. For example, the type of data store may be a SQL Server database dataset, an ORACLE database dataset, a TERADATA database dataset, a flat file, a multifile data store, a HADOOP database dataset, a DB2 data store, a Microsoft SQL SERVER dataset, an INFORMIX dataset, a table, a collection of tables, or other types of data stores. The data store information 446 may include information about the record format or schema of the physical dataset. The data store information 446 may include a record name. For example, the dataset may be a table and the record name may be the name of the table. In another example, the dataset may be a data flow graph and the record name may be a file name to save the data flow graph.

[0177] As shown in FIG. 4G , in some embodiments, the dataset catalog entry 440 may include other information 448, such as security information, access information, and / or other user parameters. For example, the security information may include a password, key, or other value for accessing the dataset. The other information 448 may include access information for accessing the dataset from the data storage 202. For example, the access information may include an address of a storage system in which the dataset is stored, or other access information. The other information 448 may include other parameters associated with the dataset, such as statistical information, a data steward, a version identifier, and / or other parameters. In some embodiments, the other information 448 may include an indication that the dataset catalog entry is associated with a physical dataset (as opposed to a dataflow graph dataset). For example, the entry 440 may include a parameter indicating that the dataset is a physical dataset.

[0178] In some embodiments, information 442 includes information for accessing the cataloged dataflow graph. In some embodiments, information 442 may include a location of the stored dataflow graph in data storage 202. For example, information 442 may include a location of a file that stores the dataflow graph. In another example, information 442 may include a path (e.g., a URL) to the location of the dataflow graph. In some embodiments, entry 440 may include information indicating that the entry is associated with a cataloged dataflow graph (e.g., as opposed to a physical dataset). For example, entry 440 may include a dataset type field that may take a first value indicating that the entry corresponds to a physical dataset and a second value indicating that the entry corresponds to a cataloged dataflow graph.

[0179] In some embodiments, if entry 440 corresponds to a cataloged dataflow graph, entry 440 may be used to incorporate the cataloged dataflow graph as a subgraph into another dataflow graph. For example, entry 440 may be associated with an input node of the dataflow graph (e.g., by storing the entry or its logical identifier 444 in the configuration information of the input node). Entry 440 may be used to access the cataloged dataflow graph when the incorporating dataflow graph is being compiled, transformed, and / or executed. For example, entry 440 may be used to access the cataloged dataflow graph from data storage 202 and incorporated as a subgraph. The subgraph may then be connected to the dataflow graph (e.g., by connecting an output link of the subgraph to a node of the dataflow graph). The resulting dataflow graph may then be compiled into a software application program, or may be transformed (e.g., for optimization) and then compiled into a software application program.

[0180] 6 is a flowchart of an exemplary process 600 for configuring a software application program developed as a dataflow graph to receive as input output data dynamically generated by the dataflow graph, according to some embodiments of the techniques described herein. In some embodiments, process 600 may be performed by data processing system 200 described herein with reference to Figures 2A-4D. For example, process 600 may be performed using one or more computer hardware processors of data processing system 200.

[0181] Process 600 begins at block 602, where the system provides a UI that allows a user to identify one or more entries in a dataset catalog that are associated with one or more respective cataloged dataflow graphs. The cataloged dataflow graphs may be dataflow graph datasets (e.g., subgraph datasets). The dataflow graphs (e.g., subgraphs) may be cataloged as described in process 700 described herein with reference to FIG. 7.

[0182] The UI may be configured to allow a user to identify entries in one or more ways. In some embodiments, the UI may include a menu with a list of entries in the dataset catalog (e.g., dataset catalog UI 324 described herein with reference to FIG. 3F). The menu may be configured to receive a selection of an entry by user input (e.g., click, drag, tap, and / or other type of user input). In some embodiments, the UI may be part of a software application development UI. For example, the UI may be a pane adjacent to a display of a dataflow graph being developed by a user. In some embodiments, the UI may provide a search bar in which a user can enter a query to search for entries. For example, a user can enter keywords, which may be used by the system to identify matching entries.

[0183] Process 600 then proceeds to block 604, where the system receives, via the UI, an identification of a first entry associated with the first cataloged graph. The system may be configured to receive user input indicating a selection of the first entry associated with the first cataloged graph. For example, the UI may receive user input indicating a selection of the first entry via a menu of the UI.

[0184] In some embodiments, the system may be configured to receive the identification of the first entry by receiving user input indicating a command to associate the first entry with a node of the dataflow graph. For example, the system may receive the user input in response to a user in a software application program development UI dragging a graphical element representing the first entry onto a node of the dataflow graph. In another example, the system may receive the user input when a user places a graphical element representing the first entry within the dataflow graph (e.g., by dragging the graphical element onto an input node of the dataflow graph).

[0185] Process 600 then proceeds to block 606, where the system configures the dataflow graph of the software application to receive as input the output data generated when the first cataloged dataflow graph associated with the first entry is executed. In some embodiments, the system may be configured to associate a node of the dataflow graph with the first cataloged dataflow graph. For example, the system may associate a node of the dataflow graph with the first cataloged dataflow graph by including in the node of the dataflow graph a reference (e.g., a location path) to the first cataloged dataflow graph indicated by the first entry. In this example, the reference may be used by the software application upon execution to execute the first cataloged dataflow graph and obtain the generated output data. In another example, the system may associate a node of the dataflow graph with the first cataloged dataflow graph by copying the dataflow graph to the node such that the dataflow graph is executed at the node. In another example, the system may associate a node of the dataflow graph with the first cataloged dataflow graph by embedding in the node a command to execute the first cataloged dataflow graph.

[0186] In some embodiments, after the dataflow graph is configured to receive as input the output data produced when the first cataloged dataflow graph was executed in block 606, the dataflow graph may obtain the output data when the dataflow graph is executed. For example, the first cataloged dataflow graph or a transformation thereof may be executed automatically as part of executing the dataflow graph or a transformation thereof.

[0187] 7 is a flowchart of an exemplary process 700 for providing a software application program with access to output data dynamically generated by a dataflow graph, according to some embodiments of the techniques described herein. In some embodiments, process 700 may be performed by data processing system 200 described herein with reference to FIGS. 2A-4D. For example, process 700 may be performed using one or more computer hardware processors of data processing system 200. In some embodiments, process 700 may be performed to catalog dataflow graphs so that they may be used in software application programs (e.g., as is performed in process 600 described herein with reference to FIG. 6).

[0188] Process 700 begins at block 702 where the system identifies a subgraph. In some embodiments, the system may be configured to identify a subgraph of a dataflow graph by (1) receiving user input indicating a portion of the dataflow graph (e.g., in a software application development UI as described herein with reference to FIG. 3B) and (2) identifying the portion of the dataflow graph as a subgraph. A subgraph may encompass one or more nodes of the dataflow graph. For example, one or more nodes of the dataflow graph may obtain input data and perform one or more data processing operations using the input data to generate output data. In some embodiments, the system may be configured to identify a subgraph by receiving user input indicating a previously created dataflow graph that was saved in data storage as a subgraph. For example, the dataflow graph may be saved to a previously created file. The system may receive user input, which may be a selection of a file.

[0189] Process 700 then proceeds to block 704, where the system creates a new entry in the dataset catalog associated with the identified subgraph. In some embodiments, the system may be configured to instantiate a new data object (e.g., a data object described in FIG. 4G) for storage in the dataset catalog. The data object may contain information for accessing the stored subgraph. For example, the data object may contain a location path where the subgraph can be accessed. In another example, the data object may contain an index, identifier, key, or other information that can be used to access the subgraph. This information may be used to configure a software application program (e.g., a data flow graph of the software application program) to obtain output data generated by the subgraph (e.g., as described herein with reference to FIG. 6).

[0190] In some embodiments, the system may be configured to store information indicative of the subgraph in a data storage (e.g., data storage 202). For example, the system may store a file having information about the nodes, links, and configuration parameters of the subgraph. The created entry may include a reference (e.g., an index, location path, identifier, key, and / or other reference) to the file. In another example, the system may generate one or more entries in a database that stores information about the subgraph. The created entry may include a reference (e.g., an index) to one or more entries for use in configuring a software application program.

[0191] Process 700 then proceeds to block 706, where the system configures the dataset catalog to enable access to the new entry in the dataset catalog associated with the subgraph. In some embodiments, the system may be configured to update the UI to include the new entry. For example, the system may update a list of entries in the UI to include the new entry. A device displaying the UI may include an updated UI that includes the new entry. In some embodiments, the system may be configured to configure a dataset catalog module (e.g., dataset catalog module 404 described herein with reference to FIGS. 4B-4C) to enable access to the new entry. For example, the system may update the dataset catalog module to enable the dataset catalog module to respond to a request for access to the new entry.

[0192] After configuring the dataset catalog at block 706, the new entry may be usable by a software application program to incorporate output data generated by the sub-graph. For example, a dataflow graph may be configured to use as input the output data generated by executing the sub-graph. As another example, a software application program that requires metadata about information stored in data storage may use information from the new entry about the sub-graph dataset.

[0193] 8 is a screenshot of a software application program development UI 800 in accordance with some embodiments of the techniques described herein. The UI 800 contains a space 802 in which a user can lay out a dataflow graph. The UI 800 provides options for data sets 804 to include in the dataflow graph, and data processing operations, such as data transformations 806 and statistical calculations 808 to include in the dataflow graph.

[0194] 9 is a screenshot of the UI of FIG. 8 including a dataflow graph 902 in accordance with some embodiments of the techniques described herein. The dataflow graph 902 uses a dataset "customer_info" 904 as an input to a filter operation 906. For example, the dataset "customer_info" 904 may be a physical dataset stored by a data processing system. The output of the filter operation 906 is provided as an input to a calculate operation 908.

[0195] Figure 10 is a screenshot of a UI 1000 in which a portion of the dataflow graph of Figure 9 has been configured as a subgraph 1002, in accordance with some embodiments of the techniques described herein. The subgraph 1002 encompasses an input data set "customer_info" 904 and a filter operation 906 from the dataflow graph 902. The subgraph 1002 may have been generated by identifying a portion of the dataflow graph 902 (e.g., based on user input) and generating a subgraph to be the identified portion.

[0196] Figure 11 is a screenshot of a UI 2100 having a menu 2102 for saving the subgraph 1002 of Figure 10 as a dataset accessible via a dataset catalog, in accordance with some embodiments of the techniques described herein. As shown in Figure 11, the UI 2100 generates (e.g., in response to user input) the menu 2102 that provides an option to save the subgraph as a dataset. In the example of Figure 11, the option is labeled "Create Data Source from Subgraph."

[0197] FIG. 12 is a screenshot of a UI 1200 for configuring details of a dataflow graph dataset in accordance with some embodiments of the techniques described herein. The UI 1200 includes a "Name" field 1202 in which a user can enter a name for the dataflow graph dataset. In the example of FIG. 12, the name of the dataflow graph dataset is "dsny_customers." The UI 1200 also includes a "Cart" field 1204 for specifying the location of the dataflow graph dataset. In the example of FIG. 12, the "Cart" field is "DATAPROJ_Data."

[0198] 13 is a screenshot of a UI 1300 for cataloging a dataflow graph, according to some embodiments of the techniques described herein. As shown in FIG. 13, in some embodiments, the UI 1300 may include a display of information about the dataflow graph being cataloged, such as fields in the output data generated by the dataflow graph.

[0199] Figure 14 is a screenshot of a UI 1400 having a dataflow graph 1402 incorporating the cataloged dataflow graph of Figure 13 in accordance with some embodiments of the techniques described herein. As shown in Figure 14, the cataloged dataflow graph "dsny_customers" is associated with node 1404 of the dataflow graph 1402 and is used as an input to node 1406 having a computation operation. The UI 1400 includes a dataset catalog menu 1408 that includes an entry 1408A for the cataloged dataflow graph "dsny_customers." Output data generated by executing the cataloged dataflow graph at node 1404 is provided as input to the computation operation at node 1406.

[0200] Figure 15 is a screenshot of a UI 1500 displaying information about a cataloged dataflow graph, according to some embodiments of the techniques described herein. In the example of Figure 15, the UI 1500 displays fields 1502 that contain information about the name, the schema of the database in which the cataloged dataflow graph is saved, the database name in which it is saved, and fields 1504 that are included in the output data generated by the cataloged dataflow graph. The UI 1500 further includes an "Application" field that may indicate the application for which the cataloged dataflow graph is developed. The UI 1500 includes a "Description" field for a textual description of the cataloged dataflow graph.

[0201] 16 is a screenshot of a UI 1600 displaying output data 1602 generated when a cataloged dataflow graph was executed, according to some embodiments of the techniques described herein. The UI 1600 shows a preview of the output data generated when the cataloged dataflow graph was executed. In the example of FIG. 16, the UI 1600 includes the data columns generated when the cataloged dataflow graph was executed.

[0202] 17 is a screenshot of a UI 1700 displaying information for a dataset catalog entry associated with a physical dataset, according to some embodiments of the techniques described herein. As shown in FIG. 17, the information includes a uniform record locator (URL) for accessing the physical dataset. It also includes an indication 1704 of the type of the physical dataset. The UI 1700 includes other information about the dataset that may be populated.

[0203] 18 is a screenshot of a UI 1800 displaying information for a dataset catalog entry associated with a dataflow graph dataset, according to some embodiments of the techniques described herein. As shown in FIG. 18, the UI 1800 includes a path 1802 for the dataflow graph path. The UI 1800 further includes an indication 1804 that the dataset is a dataflow graph dataset. In the example of FIG. 18, the UI 1800 shows a physical dataset type of "subgraph," indicating that the dataset is a dataflow graph dataset.

[0204] Exemplary Computer System 19 illustrates an example of a suitable computing system environment 1900 on which the technology described herein may be implemented. The computing system environment 1900 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the technology described herein. Neither should the computing environment 1900 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 1900.

[0205] The technology described herein can be used with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computers, distributed computing environments that incorporate any of the above systems or devices, and the like.

[0206] A computing environment can execute computer-executable instructions such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The techniques described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0207] With reference to FIG. 19, an exemplary system for implementing the techniques described herein includes a general purpose computing device in the form of a computer 1110. Components of the computer 1910 may include, but are not limited to, a processing unit 1920, a system memory 1930, and a system bus 1921 that couples various system components including the system memory to the processing unit 1920. The system bus 1921 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, also known as Mezzanine bus.

[0208] Computer 1910 typically includes a variety of computer readable media. Computer readable media may be any available media that can be accessed by computer 1910, and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store the desired information and that can be accessed by computer 1910. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable media.

[0209] The system memory 1930 encompasses computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 1931 and random access memory (RAM) 1932. A basic input / output system 1933 (BIOS), containing the basic routines that help to transfer information between elements within the computer 1910, such as during start-up, is typically stored in ROM 1931. RAM 1932 typically contains data and / or program modules that are immediately available and / or presently being operated on by the processing unit 1920. By way of example, and not limitation, FIG. 19 illustrates operating system 1934, application programs 1935, other program modules 1036, and program data 1937.

[0210] Computer 1910 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, Figure 19 illustrates a hard disk drive 1941 which reads from or writes to non-removable, non-volatile magnetic media, a flash drive 1951 which reads from or writes to removable, non-volatile memory 1952, such as a flash memory, and an optical disk drive 1955 which reads from or writes to a removable, non-volatile optical disk 1956, such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 1941 is typically connected to the system bus 1921 through a non-removable memory interface, such as interface 1940, and the magnetic disk drive 1951 and optical disk drive 1955 are typically connected to the system bus 1921 by a removable memory interface, such as interface 1950.

[0211] The drives and their associated computer storage media, discussed above and illustrated in FIG. 19, provide storage of computer readable instructions, data structures, program modules, and other data for the computer 1910. In FIG. 19, for example, hard disk drive 1941 is illustrated as storing operating system 1944, application programs 1945, other program modules 1946, and program data 1947. Note that these components may be the same as or different from the operating system 1934, application programs 1935, other program modules 1936, and program data 1937. The operating system 1944, application programs 1945, other program modules 1946, and program data 1947 are numbered differently here to illustrate that, at a minimum, they are different copies. Actors can enter commands and information into the computer 1910 through input devices such as a keyboard 1962 and pointing device 1961, commonly referred to as a mouse, trackball, or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, etc. These and other input devices are often connected to the processing unit 1920 by a user input interface 1960 that is coupled to the system bus, but may also be connected by other interface and bus structures, such as a parallel port, game port, or universal serial bus (USB). A monitor 1991 or other type of display device is also connected to the system bus 1921 via an interface, such as a video interface 1990. In addition to the monitor, computers may also include other peripheral output devices such as speakers 1997 and printer 1996, which may be connected through an output peripheral interface 1995.

[0212] The computer 1910 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 1980. The remote computer 1980 may be a personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer 1910, although only a memory storage device 1981 is illustrated in FIG. 19. The logical connections depicted in FIG. 19 include a local area network (LAN) 1981 and a wide area network (WAN) 1983, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0213] When used in a LAN networking environment, the computer 1910 is connected to the LAN 1971 through a network interface or adapter 1970. When used in a WAN networking environment, the computer 1910 typically includes a modem 1972 or other means for establishing communications over the WAN 1973, such as the Internet. The modem 1972, which may be internal or external, may be connected to the system bus 1921 via the actor input interface 1960 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 1910, or portions thereof, may be stored in a remote memory storage device. By way of example, and not limitation, FIG. 19 illustrates remote application programs 1985 as residing on memory device 1981. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0214] Having thus described several aspects of at least one embodiment of the technology described herein, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0215] Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Furthermore, advantages of the technology described herein are shown, but it is to be understood that not all embodiments of the technology described herein include all described advantages. Some embodiments may not implement any of the features described herein as advantageous, and in some cases, one or more of the described features may be implemented to obtain further embodiments. Thus, the above description and drawings are merely examples.

[0216] The above embodiments of the techniques described herein may be implemented in any of a number of ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in integrated circuit components, including commercially available integrated circuit components known in the industry by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processors may be implemented in custom circuits, such as ASICs, or semi-custom circuits resulting from the configuration of programmable logic devices. As yet a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of the multiple cores may constitute a processor. However, the processors may be implemented using circuits of any suitable format.

[0217] Further, it should be understood that the computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. Additionally, the computer may be incorporated into devices not generally considered computers, but equipped with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other suitable portable or fixed electronic device.

[0218] A computer may also have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for a visual representation of the output, and a speaker or other sound generating device for an audible representation of the output. Examples of input devices that may be used in a user interface include keyboards and pointing devices such as mice, touch pads, and digitizer tablets. As another example, a computer may receive input information by voice recognition or in other audible formats.

[0219] Such computers may be interconnected by one or more networks of any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0220] Also, the various methods or processes outlined herein may be encoded as software executable for one or more processors using any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine code or intermediate code that runs on a framework or virtual machine.

[0221] In this regard, aspects of the technology described herein may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods for implementing the various embodiments described hereinabove. As is evident from the above examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such one or more computer-readable storage media may be portable such that one or more programs stored thereon may be loaded into one or more different computers or other processors to implement various aspects of the technology as described above. As used herein, the term "computer-readable storage medium" encompasses only non-transitory computer-readable media that may be considered to be an article (i.e., an article of manufacture) or a machine. Alternatively or additionally, aspects of the technology described herein may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.

[0222] The terms "program" or "software" are used herein generically to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to perform various aspects of the technology as described above. Additionally, in accordance with certain aspects of the present embodiments, it is to be understood that one or more computer programs that, when executed, perform the methods of the technology described herein need not reside on a single computer or processor, but may be distributed in a modular manner among a number of different computers or processors to perform various aspects of the technology described herein.

[0223] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0224] Also, the data structures may be stored in the computer-readable medium in any suitable form. For ease of illustration, the data structures may be shown with fields related by location within the data structure. Such relationships may likewise be achieved by assigning locations within the computer-readable medium that convey the relationship between the fields to the storage of the fields. However, any suitable mechanism may be used to establish relationships between information in the fields of the data structure, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0225] Various aspects of the technology described herein may be used alone, in combination, or in various arrangements not specifically set forth in the embodiments described above, and therefore are not limited in their application to the details and arrangements of components set forth in the above description or illustrated in the drawings. For example, aspects described in one embodiment can be combined in any manner with aspects described in other embodiments.

[0226] The techniques described herein may also be embodied as methods, examples of which are provided in connection with Figures 3 and 7 herein. Acts performed as part of any method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that shown (which may include performing some acts simultaneously, even though in the illustrative embodiments they are shown as sequential acts).

[0227] Additionally, some actions are described as being performed by an "actor" or a "user." It is to be understood that an "actor" or "user" need not be a single individual, and that in some embodiments, actions attributed to an "actor" or "user" may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms.

[0228] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, of itself, imply a priority, precedence, or ordering of one claim element over another, or the chronological order in which acts of a method are performed, but rather is used merely as a label to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms) that distinguishes between the claim elements.

[0229] Also, the phraseology and terminology used herein are for purposes of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items.

Claims

1. 1. A method performed by a data processing system for enabling efficient development of software application programs in a dynamic environment having multiple datasets by using entries in a dataset catalog to provide software application programs developed as data flow graphs with access to output data dynamically generated by one or more other data flow graphs, said entries being associated with each software application program developed as a data flow graph having nodes representing data processing operations and links representing flow of data, said method comprising: Using at least one computer hardware processor, providing a user interface that allows a user to identify one or more entries in a dataset catalog that are associated with one or more respective cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph; the first cataloged data flow graph having one or more nodes representing one or more respective data sources and one or more nodes representing one or more respective data processing operations; when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more respective data sources; To provide and receiving, via the user interface, an identification of the first entry associated with the first cataloged dataflow graph; configuring the data flow graph of the software application program to receive as input the output data produced when the first cataloged data flow graph is executed, said configuring including associating one of the input nodes of the data flow graph with the first cataloged data flow graph; A method comprising:

2. 2. The method of claim 1 , wherein receiving the identification of the first entry associated with the first cataloged dataflow graph comprises receiving a selection of the first entry via the user interface.

3. providing the user interface includes generating a graphical user interface having a searchable menu of the one or more entries in the dataset catalog; 3. The method of claim 1 or claim 2, wherein receiving the identification of the first entry associated with the first cataloged dataflow graph comprises receiving user input via the user interface indicating a selection of the first entry in the searchable menu.

4. 10. The method of claim 1 or any other preceding claim, further comprising executing the constructed data flow graph of the software application program.

5. executing the constructed data flow graph of the software application program, executing the first cataloged dataflow graph to generate the output data; providing the generated output data as input to the data flow graph of the software application to perform at least one of the one or more data processing operations using the output data; 10. The method of claim 4 or any other preceding claim, comprising:

6. 5. The method of claim 4 or any other preceding claim, wherein executing the constructed data flow graph results in execution of the first cataloged data flow graph.

7. 10. The method of claim 6 or any other preceding claim, wherein the output data is generated by the first cataloged data flow graph during execution of the constructed data flow graph.

8. 10. The method of claim 1 or any other preceding claim, wherein the dataset catalog includes a plurality of entries associated with each cataloged dataflow graph and a plurality of entries associated with each dataset previously stored in memory.

9. 10. The method of claim 1 or any other preceding claim, wherein the user interface enables the user to identify at least one entry in the dataset catalog associated with at least one respective cataloged physical dataset previously stored in memory, the at least one entry including a second entry associated with a physical dataset stored in memory.

10. the one or more input nodes include a plurality of input nodes, and the method further comprises: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; configuring the data flow graph of the software application program to receive data from the physical dataset as an input, said configuring including associating another one of the plurality of input nodes in the data flow graph with the data from the physical dataset; 10. The method of claim 9 or any other preceding claim, further comprising:

11. transforming the data flow graph that includes the input node associated with the first cataloged data flow graph to obtain a transformed data flow graph; compiling the transformed data flow graph into a software application program; executing the software application program; 10. The method of claim 1 or any other preceding claim, further comprising:

12. transforming the data flow graph containing the input node associated with the first cataloged data flow graph to obtain the transformed data flow graph; incorporating the first cataloged dataflow graph as a first subgraph into the dataflow graph at the input node associated with the first cataloged dataflow graph; transforming the first subgraph to obtain a second subgraph that is different from the first subgraph; 12. The method of claim 11 or any other preceding claim, comprising:

13. Transforming the first subgraph to obtain the second subgraph, 13. The method of claim 12 or any other preceding claim, comprising transforming the first sub-graph based at least in part on at least one operation represented by at least one node downstream of the input node in the dataflow graph.

14. Transforming the first subgraph to obtain the second subgraph, 13. The method of claim 12 or any other preceding claim, comprising applying at least one optimization to the first subgraph to obtain the second subgraph.

15. The at least one optimization comprises: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; reordering the nodes of the first subgraph; Merging a plurality of nodes of the first subgraph; Parallelizing the processing of at least one operation represented by at least one node of said first subgraph; or removing data of at least one node of the first subgraph so that the data is not used in subsequent operations represented by nodes downstream of the at least one node of the first subgraph; 15. The method of claim 14 or any other preceding claim, comprising at least one of:

16. The converting step comprises: identifying at least one portion of the dataflow graph to transform, the at least one portion comprising the first cataloged dataflow graph associated with the input node; transforming the at least one portion of the dataflow graph to obtain the transformed dataflow graph; 12. The method of claim 11 or any other preceding claim, comprising:

17. 10. The method of claim 1 or any other preceding claim, wherein the first cataloged data flow graph is generated from a subgraph embedded in another data flow graph, the other data flow graph having nodes representing data processing operations and links representing flows of data between the nodes, the other data flow graph being separate from the data flow graph of the software application.

18. displaying, in a UI, a graphical representation of the other data flow graph; receiving user input via the UI indicating that the subgraph within the dataflow graph is to be cataloged; In response to receiving the user input indicating that the subgraph within the dataflow graph is to be cataloged, saving the subgraph as the first cataloged dataflow graph; 18. The method of claim 17 or any other preceding claim, further comprising:

19. 10. The method of claim 1 or any other preceding claim, wherein the first cataloged dataflow graph has only a single output link representing data output by the first cataloged dataflow graph by applying the one or more data processing operations to data obtained from the one or more data sources.

20. 10. The method of claim 1 or any other preceding claim, wherein the first cataloged dataflow graph is stored in a data storage of the data processing system, and the first entry stores a reference to a location of the first cataloged dataflow graph within the data storage.

21. 10. The method of claim 1 or any other preceding claim, wherein the first entry stores a reference to a file that stores information indicating nodes of the first cataloged data flow graph and / or configuration parameters of the first cataloged data flow graph.

22. configuring the data flow graph of the software application program to receive as input the output data produced when the first cataloged data flow graph is executed; receiving, via the user interface, an association of the first entry with the input node in the dataflow graph; in response to receiving the user input associating the first entry with the input node in the dataflow graph; configuring the data flow graph to receive at the input nodes data output via output links of the first cataloged data flow graph as a result of execution of the first cataloged data flow graph; 10. The method of claim 1 or any other preceding claim, comprising:

23. receiving the association between the first entry and the input node in the dataflow graph; 10. The method of claim 1 or any other preceding claim, comprising receiving user input via the user interface indicating an association between a first graphical element representing the first entry and a second graphical element representing the input node in the dataflow graph.

24. 24. The method of claim 23 or any other preceding claim, wherein the user input indicating the association between the first graphical element representing the first entry and the second graphical element representing the input node comprises dragging the first graphical element to the second graphical element in the user interface.

25. A data processing system configured to perform a method according to any one of claims 1 to 24.

26. A computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform a method according to any one of claims 1 to 24.

Images