Data flow graph data set

Abstract

Described herein is a technique performed by a data processing system for providing a software application with access to output data dynamically generated by a dataflow graph by generating entries in a dataset directory, thereby enabling efficient development of the software application in a dynamic environment having multiple datasets, these entries are associated with respective software applications developed as dataflow diagrams. The techniques include identifying a sub-graph that, when executed, generates output data by applying one or more data processing operations to data obtained from one or more data sources; creating a new entry associated with the identified sub-graph in the dataset directory, the new entry associated with information indicative of nodes, links, and configuration parameters of the identified sub-graph; and configuring the dataset directory to allow access to a new entry in the dataset directory associated with the identified sub-graph.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 63 / 338,855, filed on May 5, 2022, and entitled “DATAFLOW GRAPH DATASETS,” and U.S. Provisional Patent Application No. 63 / 432,615, filed on December 14, 2022, and entitled “DATAFLOW GRAPH DATASETS,” each of which is hereby incorporated by reference in its entirety. Technical Field

[0003] Various aspects of the present disclosure relate to techniques for enabling efficient data analysis in a dynamic environment with multiple data sets, wherein software applications are developed to access data flow graphs of data sets through a data set directory. This technique allows subgraphs of a data flow graph to be stored as data sets that can be accessed by other data flow graphs through the data set directory. Background Art

[0004] Modern data processing systems manage large amounts of data (e.g., millions, billions, or trillions of data records) and manage how that data is accessed (e.g., created, updated, read, or deleted). Large organizations (e.g., multinational banks, global technology companies, etc.) may have millions of data sets. For example, a data set may store transaction records, documents, tables, files, or any other suitable type of data. As another example, a data set may store "metadata," which is data that contains information about other data (e.g., stored in the same data processing system and / or another data processing system) and / or processes (e.g., in the same data processing system and / or another data processing system). For example, a data processing system may store metadata about credit card transaction data stored in a table of a credit card company's database. Non-limiting examples of such metadata include information indicating: the size of the table in memory; when the table was created; when the table was last updated; the number of rows and / or columns in the table; where the table is stored; who has the authority to read, update, delete, and / or perform any other suitable action(s) on the table.

[0005] Data processing systems may execute software applications to support a variety of functions. Software applications may be used to provide functions that support various processes of an organization. As part of performing such functions, software applications may perform operations on data sets. For example, a bank may develop software applications to support various aspects of its business, such as programs for generating credit reports, bank account histories, transaction reports, and / or other data. Software applications may also be used to extract information from data sets. Summary of the invention

[0006] Some embodiments provide a method performed by a data processing system for providing a software application developed as a dataflow graph with access to output data dynamically generated by one or more other dataflow graphs, thereby enabling efficient development of software applications in a dynamic environment with multiple data sets, by using entries in a data set directory, the entries being associated with corresponding software applications developed as dataflow graphs having nodes representing data processing operations and links representing data flows. The method includes: using at least one computer hardware processor to perform the following operations: providing a user interface through which a user is able to identify one or more entries associated with one or more corresponding cataloged dataflow graphs in a data set catalog, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged dataflow graph has one or more nodes representing one or more corresponding data sources and one or more nodes representing one or more corresponding data processing operations, wherein when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receiving an identification of the first entry associated with the first cataloged dataflow graph via the user interface; and configuring the dataflow graph of the software application to receive as input the output data generated when the first cataloged dataflow graph is executed, the configuration comprising associating one of the input nodes in the dataflow graph with the first cataloged dataflow graph.

[0007] In some embodiments, receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving a selection of the first entry via the user interface.

[0008] In some embodiments, providing the user interface includes: generating a graphical user interface having a searchable menu of the one or more entries in the data set directory; and receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving user input via the user interface indicating a selection of the first entry in the searchable menu.

[0009] In some embodiments, the method includes: executing a configured dataflow graph of the software application. In some embodiments, executing the configured dataflow graph of the software application includes: executing the first cataloged dataflow graph to generate the output data; and providing the generated output data as input to the dataflow graph of the software application to perform at least one of the one or more data processing operations using the output data. In some embodiments, executing the configured dataflow graph results in executing the first cataloged dataflow graph. In some embodiments, the output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

[0010] In some embodiments, the data set directory includes a plurality of entries associated with corresponding cataloged data flow graphs and a plurality of entries associated with corresponding data sets previously stored in memory.

[0011] In some embodiments, the user interface allows the user to identify at least one entry in the data set catalog associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory. In some embodiments, the one or more input nodes comprise a plurality of input nodes, the method further comprising: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and configuring a data flow graph of the software application to receive data from the physical data set as input, the configuring comprising associating another one of the plurality of input nodes in the data flow graph with the data from the physical data set.

[0012] In some embodiments, the method further comprises: transforming the dataflow graph including the input node associated with the first cataloged dataflow graph to obtain a transformed dataflow graph; compiling the transformed dataflow graph into a software application; and executing the software application. In some embodiments, transforming the dataflow graph including the input node associated with the first cataloged dataflow graph to obtain the transformed dataflow graph comprises: incorporating the first cataloged dataflow graph into the dataflow graph as a first subgraph at the input node associated with the first cataloged dataflow graph; and transforming the first subgraph to obtain a second subgraph different from the first subgraph.

[0013] In some embodiments, transforming the first subgraph to obtain the second subgraph includes: transforming the first subgraph based at least in part on at least one operation represented by at least one node downstream of the input node in the data flow graph. In some embodiments, transforming the first subgraph to obtain the second subgraph includes: applying at least one optimization to the first subgraph to obtain the second subgraph. In some embodiments, the at least one optimization includes at least one of the following: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; changing the order of multiple nodes of the first subgraph; combining multiple nodes of the first subgraph; processing at least one operation represented by at least one node of the first subgraph in parallel; or deleting data in at least one node in the first subgraph so that the data is not used in a subsequent operation represented by a node downstream of at least one node in the first subgraph. In some embodiments, the transformation includes: identifying at least a portion of the data flow graph to be transformed, the at least a portion including the first cataloged data flow graph associated with the input node; and transforming the at least a portion of the data flow graph to obtain the transformed data flow graph.

[0014] In some embodiments, the first cataloged dataflow graph is generated from a subgraph embedded in another dataflow graph having nodes representing data processing operations and links representing data flows between the nodes, wherein the other dataflow graph is separate from the dataflow graph of the software application. In some embodiments, the method further comprises: displaying a graphical representation of the other dataflow graph in a UI; and receiving, through the UI, a user input indicating that a subgraph within the dataflow graph is to be cataloged; and in response to receiving the user input indicating that a subgraph within the dataflow graph is to be cataloged, storing the subgraph as the first cataloged dataflow graph.

[0015] In some embodiments, the first cataloged dataflow graph has only a single output link, the output link representing data output by the first cataloged dataflow graph by applying the one or more data processing operations to data obtained from the one or more data sources. In some embodiments, the first cataloged dataflow graph is stored in a data storage device of the data processing system, and the first entry stores a reference to the location of the first cataloged dataflow graph in the data storage device.

[0016] In some embodiments, the first entry stores a reference to a file that stores information indicating nodes of the first cataloged dataflow graph and / or configuration parameters of the first cataloged dataflow graph.

[0017] In some embodiments, configuring the dataflow graph of the software application to receive as input output data generated when executing the first cataloged dataflow graph includes: receiving, via the user interface, an association of the first entry with the input node in the dataflow graph; and in response to receiving the user input associating the first entry with the input node in the dataflow graph: configuring the dataflow graph to receive at the input node data output through an output link of the first cataloged dataflow graph as a result of executing the first cataloged dataflow graph.

[0018] In some embodiments, receiving an association of the first entry with the input node in the data flow graph includes: receiving user input via the user interface, the user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node in the data flow graph. In some embodiments, the user input indicating an association of the first graphical element representing the first entry with the second graphical element representing the input node includes dragging the first graphical element to the second graphical element in the user interface.

[0019] Some embodiments provide a data processing system for enabling efficient development of software applications in a dynamic environment with multiple data sets by providing software applications developed as data flow graphs with access to output data dynamically generated by one or more other data flow graphs using entries in a data set directory associated with corresponding software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows. The system includes: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform the following operations: provide a user interface through which a user can identify one or more entries associated with one or more corresponding cataloged dataflow graphs in a data set catalog, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged dataflow graph has one or more nodes representing one or more corresponding data sources and one or more nodes representing one or more corresponding data processing operations, wherein when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receive, via the user interface, an identification of the first entry associated with the first cataloged dataflow graph; and configure the dataflow graph of the software application to receive as input the output data generated when the first cataloged dataflow graph is executed, the configuration comprising associating one of the input nodes in the dataflow graph with the first cataloged dataflow graph.

[0020] In some embodiments, receiving identification of the first entry associated with the first cataloged dataflow graph comprises receiving, via the user interface, a selection of the first entry. In some embodiments, providing the user interface comprises: generating a graphical user interface having a searchable menu of the one or more entries in the dataset catalog; and receiving identification of the first entry associated with the first cataloged dataflow graph comprises receiving, via the user interface, a user input indicating a selection of the first entry in the searchable menu.

[0021] In some embodiments, the instructions further cause the at least one computer hardware processor to execute a dataflow graph for executing the software application. In some embodiments, executing the dataflow graph for the software application comprises: executing the first cataloged dataflow graph to generate the output data; and providing the generated output data as input to the dataflow graph for the software application to perform at least one of the one or more data processing operations using the output data. In some embodiments, executing the configured dataflow graph results in executing the first cataloged dataflow graph. In some embodiments, the output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

[0022] In some embodiments, the data set directory includes a plurality of entries associated with corresponding cataloged data flow graphs and a plurality of entries associated with corresponding physical data sets previously stored in memory.

[0023] In some embodiments, the user interface allows the user to identify at least one entry in the data set catalog associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory. In some embodiments, the one or more input nodes comprise a plurality of input nodes, and the instructions further cause the at least one computer hardware processor to perform the following operations: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and configuring a data flow graph of the software application to receive data from the physical data set as input, the configuring comprising associating another of the plurality of input nodes in the data flow graph with the data from the physical data set.

[0024] In some embodiments, the data set catalog includes a plurality of entries associated with respective cataloged data flow graphs and a plurality of entries associated with respective physical data sets previously stored in the memory. In some embodiments, the user interface allows the user to identify at least one entry in the data set catalog associated with at least one respective cataloged physical data set previously stored in the memory, the at least one entry including a second entry associated with a physical data set stored in the memory.

[0025] In some embodiments, the one or more input nodes include a plurality of input nodes, and the instructions further cause the at least one computer hardware processor to: receive, via the user interface, an identification of the second entry associated with a physical data set stored in the memory; and configure a data flow graph of the software application to receive data from the physical data set as input, the configuration comprising associating another of the plurality of input nodes in the data flow graph with the data from the physical data set.

[0026] In some embodiments, the instructions further cause the at least one computer hardware processor to perform the following operations: transforming the dataflow graph including the input node associated with the first cataloged dataflow graph to obtain a transformed dataflow graph; compiling the transformed dataflow graph into a software application; and executing the software application. In some embodiments, transforming the dataflow graph including the input node associated with the first cataloged dataflow graph to obtain the transformed dataflow graph includes: incorporating the first cataloged dataflow graph into the dataflow graph as a first subgraph at the input node associated with the first cataloged dataflow graph; and transforming the first subgraph to obtain a second subgraph different from the first subgraph.

[0027] In some embodiments, transforming the first subgraph to obtain the second subgraph includes: transforming the first subgraph based at least in part on at least one operation represented by at least one node downstream of the input node in the data flow graph. In some embodiments, transforming the first subgraph to obtain the second subgraph includes: applying at least one optimization to the first subgraph. In some embodiments, the at least one optimization includes at least one of the following: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; changing the order of multiple nodes of the first subgraph; combining multiple nodes of the first subgraph; processing at least one operation represented by at least one node of the first subgraph in parallel; or deleting data in at least one node in the first subgraph so that the data is not used in a subsequent operation represented by a node downstream of at least one node in the first subgraph. In some embodiments, the transformation includes: identifying at least a portion of the data flow graph to be transformed, the at least a portion including the first cataloged data flow graph associated with the input node; and transforming the at least a portion of the data flow graph to obtain the transformed data flow graph.

[0028] In some embodiments, the first cataloged dataflow graph is generated from a subgraph embedded in another dataflow graph having nodes representing data processing operations and links representing data flow between the nodes, wherein the other dataflow graph is separate from the dataflow graph of the software application. In some embodiments, the instructions further cause the at least one computer hardware processor to perform the following operations: display a graphical representation of the other dataflow graph in a UI; and receive user input indicating a subgraph within the dataflow graph through the UI.

[0029] In some embodiments, the first cataloged dataflow graph has a single output link that represents data output by the first cataloged dataflow graph by applying the one or more data processing operations to data obtained from the one or more data sources.

[0030] In some embodiments, the first entry stores a reference to a file that stores information indicating nodes of the first cataloged dataflow graph and / or configuration parameters of the first cataloged dataflow graph.

[0031] In some embodiments, configuring the dataflow graph of the software application to receive as input output data generated when executing the first cataloged dataflow graph includes: receiving, via the user interface, an association of the first entry with the input node in the dataflow graph; and in response to receiving the user input associating the first entry with the input node in the dataflow graph: configuring the dataflow graph to receive at the input node data output through an output link of the first cataloged dataflow graph as a result of executing the first cataloged dataflow graph.

[0032] In some embodiments, receiving an association of the first entry with the input node in the data flow graph includes: receiving user input via the user interface, the user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node in the data flow graph. In some embodiments, the user input indicating an association of the first graphical element representing the first entry with the second graphical element representing the input node includes dragging the first graphical element to the second graphical element in the user interface.

[0033] Some embodiments provide at least one non-transitory computer-readable storage medium storing instructions that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing a software application developed as a dataflow graph with access to output data dynamically generated by one or more other dataflow graphs by using entries in a data set catalog associated with respective software applications developed as dataflow graphs having nodes representing data processing operations and links representing data flows, thereby enabling efficient development of software applications in a dynamic environment having multiple data sets. The method includes: providing a user interface through which a user is able to identify one or more entries associated with one or more corresponding cataloged dataflow graphs in a data set catalog, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged dataflow graph has one or more nodes representing one or more corresponding data sources and one or more nodes representing one or more corresponding data processing operations, wherein when the first cataloged dataflow graph is executed, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receiving an identification of the first entry associated with the first cataloged dataflow graph via the user interface; and configuring the dataflow graph of the software application to receive as input the output data generated when the first cataloged dataflow graph is executed, the configuration comprising associating one of the input nodes in the dataflow graph with the first cataloged dataflow graph.

[0034] Some embodiments provide a method performed by a data processing system for providing a software application with access to output data dynamically generated by a data flow graph by generating entries in a data set directory, thereby enabling efficient development of software applications in a dynamic environment with multiple data sets, the entries being associated with corresponding software applications developed as data flow graphs, the data flow graphs having nodes representing data processing operations and links representing data flows. The method includes: using at least one computer hardware processor to perform the following operations: identifying a subgraph, wherein the subgraph has one or more nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and output links representing data output by the subgraph, wherein when the subgraph is executed, the subgraph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry associated with the identified subgraph in the data set directory, the new entry being associated with information indicating nodes, links, and configuration parameters of the identified subgraph; and configuring the data set directory to allow access to the new entry associated with the identified subgraph in the data set directory to incorporate the subgraph into the software application.

[0035] In some embodiments, identifying the subgraph includes: displaying a graphical representation of the data flow graph in a user interface; and receiving, via the user interface, a first user input indicating a subgraph within the data flow graph. In some embodiments, the method further includes: receiving, via the user interface, a second user input commanding creation of the new entry associated with the indicated subgraph; wherein creating the new entry associated with the identified subgraph is performed in response to receiving the second user input.

[0036] In some embodiments, the data processing system includes a data storage device storing a previously created data flow graph, and identifying the subgraph includes: receiving user input identifying the previously created data flow graph as the subgraph via a user interface. In some embodiments, the new entry includes information indicating nodes, links, and configuration parameters of the identified subgraph. In some embodiments, configuring the data set directory to allow access to the new entry associated with the identified subgraph includes providing a user interface through which a user can identify the new entry associated with the identified subgraph in the data set directory.

[0037] In some embodiments, the one or more corresponding data sources include a physical data set previously stored in the memory, and the data set directory includes an entry associated with the physical data set. In some embodiments, the data processing system includes a data storage device, and the method further includes: storing the subgraph in the data storage device of the data processing system; and wherein creating the new entry includes storing in the new entry a reference to a location of the stored subgraph in the data storage device of the data processing system.

[0038] Some embodiments provide a data processing system for providing software applications with access to output data dynamically generated by a data flow graph by generating entries in a data set directory, thereby enabling efficient development of software applications in a dynamic environment with multiple data sets, the entries being associated with corresponding software applications developed as data flow graphs, the data flow graphs having nodes representing data processing operations and links representing data flows. The system includes: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing instructions, which instructions, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform the following operations: identifying a subgraph of a data flow graph, wherein the subgraph has one or more nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and output links representing data output by the subgraph, wherein when the subgraph is executed, the subgraph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry associated with the identified subgraph in the data set directory, the new entry being associated with information indicating nodes, links, and configuration parameters of the identified subgraph; and configuring the data set directory to allow access to the new entry associated with the identified subgraph in the data set directory.

[0039] In some embodiments, identifying the subgraph includes: displaying a graphical representation of the data flow graph in a user interface; and receiving, via the user interface, a first user input indicating a subgraph within the data flow graph. In some embodiments, the instructions further cause the at least one computer hardware processor to perform the following operations: receiving, via the user interface, a second user input commanding creation of the new entry associated with the indicated subgraph; wherein creating the new entry associated with the identified subgraph is performed in response to receiving the second user input.

[0040] In some embodiments, the data processing system includes a data storage device storing a previously created data flow graph, and identifying the subgraph includes: receiving user input identifying the previously created data flow graph as the subgraph via a user interface. In some embodiments, the new entry includes information indicating nodes, links, and configuration parameters of the identified subgraph. In some embodiments, configuring the data set directory to allow access to the new entry associated with the identified subgraph includes providing a user interface through which a user can identify the new entry associated with the identified subgraph in the data set directory.

[0041] In some embodiments, the one or more corresponding data sources include a physical data set previously stored in the memory, and the data set directory includes an entry associated with the physical data set. In some embodiments, the data processing system includes a data storage device, and the instructions further cause the at least one computer hardware processor to perform the following operations: store the subgraph in the data storage device of the data processing system; and wherein creating the new entry includes storing in the new entry a reference to a location of the stored subgraph in the data storage device of the data processing system.

[0042] Some embodiments provide at least one non-transitory computer-readable storage medium storing instructions. These instructions, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing software applications with access to output data dynamically generated by a data flow graph by generating entries in a data set directory, thereby enabling efficient development of software applications in a dynamic environment with multiple data sets, these entries are associated with corresponding software applications developed as data flow graphs, and the data flow graphs have nodes representing data processing operations and links representing data flows. The method includes: identifying a subgraph of a data flow graph, wherein the subgraph has one or more nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and output links representing data output by the subgraph, wherein when the subgraph is executed, the subgraph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry associated with the identified subgraph in the data set directory, the new entry being associated with information indicating nodes, links, and configuration parameters of the identified subgraph; and configuring the data set directory to allow access to the new entry associated with the identified subgraph in the data set directory.

[0043] In some embodiments, identifying the sub-graph includes: displaying a graphical representation of the dataflow graph in a user interface; and receiving, via the user interface, a first user input indicating a sub-graph within the dataflow graph.

[0044] In some embodiments, the method further includes: receiving a second user input via the user interface, the second user input commanding creation of the new entry associated with the indicated sub-graph; wherein creating the new entry associated with the identified sub-graph is performed in response to receiving the second user input.

[0045] Some embodiments provide a method performed by a data processing system for providing a software application developed as a dataflow graph having nodes representing data processing operations and links representing data flow between the nodes with access to output data dynamically generated by one or more other dataflow graphs. The method includes: using at least one computer hardware processor to perform the following operations: providing a user interface configured to receive from a user an identification of one or more entries in a data set catalog associated with one or more corresponding cataloged data flow graphs having nodes representing data processing operations and links representing data flows between the nodes, the one or more entries including a first entry associated with a first cataloged data flow graph, wherein the first cataloged data flow graph has one or more input nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and an output link representing data output from the first cataloged data flow graph, wherein when the first cataloged data flow graph is executed, the first cataloged data flow graph generates output data on the output link by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, an identification of the first entry associated with the first cataloged data flow graph; and configuring the data flow graph of the software application to receive as input the output data dynamically generated when the first cataloged data flow graph is executed by using the first entry, the configuration comprising associating one of the input nodes in the data flow graph of the software application with the first cataloged data flow graph.

[0046] In some embodiments, receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving a selection of the first entry via the user interface.

[0047] In some embodiments, providing the user interface includes: generating a graphical user interface having a searchable menu of the one or more entries in the data set directory; and receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving user input via the user interface indicating a selection of the first entry in the searchable menu.

[0048] In some embodiments, the method further comprises executing the configured dataflow graph of the software application. In some embodiments, executing the configured dataflow graph of the software application comprises: executing the first cataloged dataflow graph to generate the output data; and providing the generated output data as input to the dataflow graph of the software application to perform at least one of the one or more data processing operations using the output data.

[0049] In some embodiments, executing the configured dataflow graph results in executing the first cataloged dataflow graph. In some embodiments, the output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph. In some embodiments, executing the configured dataflow graph of the software application comprises: compiling the configured dataflow graph of the software application to obtain a compiled software application; and executing the compiled software application.

[0050] In some embodiments, the method further comprises maintaining the data set directory, wherein the data set directory comprises a plurality of entries associated with corresponding cataloged data flow graphs and preferably a plurality of entries associated with corresponding physical data sets previously stored in the memory.

[0051] In some embodiments, the user interface allows the user to identify at least one entry in the data set catalog associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory. In some embodiments, the data flow graph comprises a plurality of input nodes, including the first input node, the method further comprising: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and configuring the data flow graph of the software application to receive data from the physical data set as input, the configuring comprising associating a second input node of the plurality of input nodes in the data flow graph with the data from the physical data set.

[0052] In some embodiments, the method further comprises: transforming the data flow graph of the software application including the first input node associated with the first cataloged data flow graph to obtain a transformed data flow graph. In some embodiments, the transforming comprises: incorporating the first cataloged data flow graph into the data flow graph at the first input node associated with the first cataloged data flow graph; identifying at least a portion of the data flow graph to be transformed, the at least a portion comprising the first cataloged data flow graph incorporated at the first input node; and transforming the first cataloged data flow graph incorporated at the first input node to obtain the transformed data flow graph. In some embodiments, the transforming comprises: incorporating the first cataloged data flow graph into the data flow graph as a first subgraph at the first input node associated with the first cataloged data flow graph; and transforming the first subgraph to obtain a second subgraph different from the first subgraph. In some embodiments, transforming the first subgraph to obtain the second subgraph comprises: transforming the first subgraph based at least in part on at least one operation represented by at least one node downstream of the first input node in the data flow graph. In some embodiments, transforming the first sub-graph to obtain the second sub-graph comprises: applying at least one optimization to the first sub-graph to obtain the second sub-graph.

[0053] In some embodiments, at least one optimization includes at least one of the following: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; changing the order of multiple nodes of the first subgraph; combining multiple nodes of the first subgraph; processing at least one operation represented by at least one node of the first subgraph in parallel; or deleting data in at least one node in the first subgraph so that the data is not used in subsequent operations represented by downstream nodes of at least one node in the first subgraph.

[0054] In some embodiments, the method further comprises: compiling the transformed data flow graph into a software application; and executing the software application.

[0055] In some embodiments, the first cataloged dataflow graph is generated from a subgraph embedded in another dataflow graph having nodes representing data processing operations and links representing data flows between the nodes, wherein the other dataflow graph is separate from the dataflow graph of the software application. In some embodiments, the method further comprises: displaying a graphical representation of the other dataflow graph in a UI; and receiving user input through the UI indicating a subgraph within the dataflow graph. In some embodiments, the output link is the only output link of the first cataloged dataflow graph.

[0056] In some embodiments, the first cataloged dataflow graph is stored in a data storage device of the data processing system, and the first entry stores a reference to a location of the first cataloged dataflow graph in the data storage device. In some embodiments, the first entry stores a reference to a file that stores information identifying nodes of the first cataloged dataflow graph and / or operations at nodes of the first cataloged dataflow graph.

[0057] In some embodiments, configuring the dataflow graph of the software application to receive as input output data generated when executing the first cataloged dataflow graph includes: receiving, via the user interface, an association of the first entry with the first input node in the dataflow graph of the software application; and in response to receiving the user input associating the first entry with the first input node in the dataflow graph of the software application: configuring the dataflow graph of the software application to receive at the first input node data output through an output link of the first cataloged dataflow graph as a result of executing the first cataloged dataflow graph.

[0058] In some embodiments, receiving an association of the first entry with the first input node in the dataflow graph of the software application includes: receiving user input via the user interface, the user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the first input node in the dataflow graph of the software application. In some embodiments, the user input indicating an association of the first graphical element representing the first entry with the second graphical element representing the first input node includes dragging the first graphical element to the second graphical element in the user interface.

[0059] Some embodiments provide a data processing system for providing a software application developed as a data flow graph having nodes representing data processing operations and links representing data flows between the nodes with access to output data dynamically generated by one or more other data flow graphs. The data processing system includes: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform the following operations: providing a user interface configured to receive from a user an identification of one or more entries in a data set catalog associated with one or more corresponding cataloged data flow graphs having nodes representing data processing operations and links representing data flows between the nodes, the one or more entries including a first entry associated with a first cataloged data flow graph, wherein the first cataloged data flow graph has one or more input nodes representing one or more corresponding data sources, a first node representing a first data source, and a second node representing a second data source. or one or more nodes of a plurality of corresponding data processing operations, and an output link representing data output from the first cataloged data flow graph, wherein, when the first cataloged data flow graph is executed, the first cataloged data flow graph generates output data on the output link by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, an identification of the first entry associated with the first cataloged data flow graph; and configuring the data flow graph of the software application by using the first entry to receive as input the output data dynamically generated when the first cataloged data flow graph is executed, the configuration comprising associating one of the input nodes in the data flow graph of the software application with the first cataloged data flow graph.

[0060] Some embodiments provide at least one non-transitory computer-readable storage medium storing instructions that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing a software application developed as a data flow graph having nodes representing data processing operations and links representing data flow between the nodes with access to output data dynamically generated by one or more other data flow graphs. The method includes: providing a user interface configured to receive from a user an identification of one or more entries in a data set catalog associated with one or more corresponding cataloged data flow graphs having nodes representing data processing operations and links representing data flows between the nodes, the one or more entries including a first entry associated with a first cataloged data flow graph, wherein the first cataloged data flow graph has one or more input nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and an output link representing data output from the first cataloged data flow graph, wherein when the first cataloged data flow graph is executed, the first cataloged data flow graph generates output data on the output link by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, an identification of the first entry associated with the first cataloged data flow graph; and configuring the data flow graph of the software application by using the first entry to receive the output data dynamically generated when the first cataloged data flow graph is executed as input, the configuration comprising associating one of the input nodes in the data flow graph of the software application with the first cataloged data flow graph.

[0061] The above is a non-limiting summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Various aspects and embodiments will be described with reference to the following drawings. It should be understood that the drawings are not necessarily drawn to scale. Items appearing in multiple drawings are indicated by the same or similar reference numerals in all drawings in which they appear.

[0063] Figure 1 is a diagram of a data processing system.

[0064] Figure 2A is a diagram illustrating cataloging of subgraphs as dataflow graph data sets in a data processing system according to some embodiments of the techniques described herein.

[0065] Figure 2B is a diagram illustrating some embodiments of the technology described herein. Figure 2A A cataloged subgraph of a graph that is incorporated into a software application.

[0066] Figure 3Ais a software application development user interface (UI) for a data processing system according to some embodiments of the techniques described herein.

[0067] Figure 3B Some embodiments of the technology described herein are illustrated. Figure 3A Identification of subgraphs within a data flow graph for software application development UI.

[0068] Figure 3C Some embodiments of the technology described herein are shown. Figure 3B The subgraph UI menu associated with the subgraph identified in .

[0069] Figure 3D In some embodiments according to the techniques described herein, Figure 3C The sub-image UI menu receives user input and will Figure 3B The subgraph catalog is the data set directory of the data processing system.

[0070] Figure 3E Some embodiments of the technology described herein are shown. Figure 3D The data set directory and data storage device 202 of the data processing system are updated by cataloging the sub-graphs in the data processing system.

[0071] Figure 3F A software application development UI is shown with another data flow diagram for another software application in accordance with some embodiments of the techniques described herein.

[0072] Figure 3G According to some embodiments of the technology described herein Figure 3F A data flow graph in which a catalog subgraph is associated with an input node.

[0073] Figure 4A is a block diagram illustrating a data set directory of a data processing system according to some embodiments of the techniques described herein.

[0074] Figure 4B is a block diagram of system modules in a data processing system according to some embodiments of the techniques described herein.

[0075] Figure 4C is a diagram illustrating some embodiments of the technology described herein Figure 4B A diagram of the interactions between system modules.

[0076] Figure 4D is a diagram illustrating a dataflow graph including a subgraph incorporated as input into the dataflow graph according to some embodiments of the techniques described herein.

[0077] Figure 4Eis a diagram illustrating some embodiments of the technology described herein Figure 4D A diagram of the transformation of a data flow graph.

[0078] Figure 4F is a diagram illustrating some embodiments of the technology described herein. Figure 4D Figure 3 is a diagram of the transformed data flow graph obtained after the transformation of .

[0079] Figure 4G is a diagram illustrating some embodiments of the technology described herein Figure 4A A diagram of a dataset directory entry that is associated with a physical dataset.

[0080] Figure 5 is an example data flow diagram for a software application according to some embodiments of the techniques described herein.

[0081] Figure 6 is a flow diagram of an example process for configuring a software application developed as a dataflow graph to receive as input output data dynamically generated by the dataflow graph in accordance with some embodiments of the techniques described herein.

[0082] Figure 7 is a flow diagram of an example process for providing a software application with access to output data dynamically generated by a dataflow graph in accordance with some embodiments of the techniques described herein.

[0083] Figure 8 is a screenshot of a software application development UI according to some embodiments of the techniques described herein.

[0084] Fig. 9 is a data flow graph according to some embodiments of the technology described herein Figure 8 A screenshot of the UI.

[0085] Fig.10 According to some embodiments of the technology described herein, Fig. 9 Screen shot of the UI with a portion of a data flow graph configured as a subgraph.

[0086] Fig.11 is a method for converting a Fig.10 The subgraphs are stored as UI screenshots of the dataset's menus accessible through the dataset catalog.

[0087] Fig.12 is a UI screenshot of details for configuring a dataflow graph dataset according to some embodiments of the techniques described herein.

[0088] Fig.13is a screenshot of a UI for cataloging a dataflow graph according to some embodiments of the techniques described herein.

[0089] Fig.14 is a method according to some embodiments of the technology described herein that incorporates Fig.13 Screen shot of the Data Flow Diagram UI for cataloging data flow diagrams.

[0090] Fig.15 is a screenshot of a UI showing information about a cataloged data flow graph according to some embodiments of the techniques described herein

[0091] Fig.16 is a UI screenshot showing output data generated when executing a catalog dataflow graph according to some embodiments of the techniques described herein.

[0092] Fig.17 is a UI screenshot showing information in a data set catalog entry associated with a physical data set according to some embodiments of the techniques described herein.

[0093] Fig.18 is a UI screenshot showing information in a dataset catalog entry associated with a dataflow graph dataset according to some embodiments of the techniques described herein.

[0094] Fig.19 is a block diagram of an illustrative computing system that can be used to implement some embodiments of the techniques described herein. DETAILED DESCRIPTION

[0095] A data processing system may use one or more software applications to process data. One or more software applications utilized by a data processing system may be developed as a data flow graph. A data flow graph may include: (1) components, referred to as "nodes" or "vertices," representing data processing operations to be performed on input data; and (2) "links" or "edges" between these components, representing the flow of data. The nodes of a data flow graph may include: one or more nodes representing corresponding input data sources; one or more output nodes representing corresponding output data sets; and / or one or more nodes representing data processing operations to be performed on the data.

[0096] To illustrate, Figure 55 is an example data flow graph 500 of a software application according to some embodiments of the technology described herein. The data flow graph 500 receives a data set as input at nodes 502A, 502B. The data set input at node 502A is provided to a filter operation at node 504, as shown in link 504A. Then, the output of the filter operation at node 504 is provided as an input to a sort operation at node 506, as shown in link 506A. The data set input at node 502B is provided to a filter operation at node 508, as shown in link 508A. Then, the output of the sort operation at node 506 and the filter operation at node 508 are provided as inputs to a join operation at node 550, as shown in links 550A, 550B. The output of the join operation 550 is then provided to an output data receiver 552, as shown in link 552A.

[0097] The inventors have developed a technique that allows software applications and / or portions thereof developed as data flow graphs to be stored and then accessed by other software applications (e.g., developed as other data flow graphs) through a data directory as a source of dynamic input data in the other software applications. The stored data flow graphs can be incorporated (e.g., as input) into one or more other software applications (e.g., programs developed as data flow graphs or other types of software programs) by using a data directory. This allows data dynamically generated by executing cataloged data flow graphs to be used as input to other (multiple) software applications. The directory maintains entries corresponding to the data flow graphs. These entries allow the data flow graphs to be reused in other software applications without the need to redevelop and / or copy the code in the software applications. This allows for more efficient utilization of computing resources, thereby efficiently performing data operations in a dynamic environment with multiple data sets.

[0098] The term "dataflow graph" may refer to a dataflow graph of a software application or a portion of a given dataflow graph. A stored dataflow graph accessible through a data directory may be referred to herein as a "cataloged dataflow graph" or a "dataflow graph dataset". A stored dataflow graph accessible through a data directory has a single link (e.g., in some embodiments, only a single link) to represent data output by the dataflow graph. A dataflow graph that has only a single link to represent data output by a dataflow graph may also be referred to herein as a "subgraph". A cataloged dataflow graph may be incorporated into one or more other dataflow graphs. Output data provided by an output link of a cataloged dataflow graph may be used for downstream operations of a dataflow graph into which the cataloged dataflow graph is incorporated.

[0099] Figure 1 is a diagram of data processing system 100. Figure 1As shown, data processing system 100 includes data storage device 102 for storing data sets including data sets 102A and 102B. Data processing system 100 includes data set directory 104, in which entries correspond to data sets stored in data storage device 102. These entries include entry 104A corresponding to data set 102A and entry 104B corresponding to data set 102B. A user of device 110 can develop software applications in data processing system 100, which use data portions of one or more data sets stored in data storage device 102 or data portions derived from these data sets to perform operations. Software applications can be developed as data flow graphs. Figure 1 Shown are software applications 106A, 106B, 106C that have been developed by a user.The software applications 106A, 106B, 106C operate on input data sets.

[0100] The entries 104A, 104B in the data set directory 104 can be used to incorporate the data sets into a software application. A user of the device 110 can use the entries 104A, 104B to associate the data sets 102A, 102B with input receivers in a data flow graph. Figure 1 As shown in the example of , data set 102A has been included as input in each of software applications 106A, 106B, 106C that use entry 104A of data set directory 104. Each of software applications 106A, 106B, 106C performs one or more operations using data from data set 102A. When data processing system 100 executes one of software applications 106A, 106B, 106C, data processing system 100 may perform a set of operations indicated by the data flow graph of the software application using data from data set 102A. Data processing system 100 may generate output data as a result of executing the software application.

[0101] The data processing system 100 may have a large number (e.g., hundreds or thousands) of software applications developed as data flow graphs to perform data processing using data sets managed by the data processing system. The data sets may change over time (e.g., due to operations performed on the data stored in the data sets). Further, users may frequently develop new data flow graphs for new software applications. For example, the data processing system 100 may be used to manage data sets for a multinational bank. The multinational bank may develop thousands of data flow graphs to process customer data associated with millions of bank accounts. In another example, the data processing system 100 may manage data sets for a credit card company. The credit card company may develop thousands of data flow graphs to process transaction data generated by millions of credit card transactions that occur every day.

[0102] The inventors have recognized that a given data flow graph (e.g., an independent graph or a subgraph of a larger data flow graph) can be used as part of one or more other software applications. Figure 5 The output of the sort operation at node 506 in may be useful in one or more other data flow graphs of one or more other corresponding software applications. Traditionally, a user needs to recreate a given data flow graph in each other data flow graph that will use the given data flow graph. This will require the user to recreate the same data flow graph multiple times to incorporate it into other data flow graphs. For example, a user may recreate the same subgraph dozens, hundreds, or thousands of times for different software applications. Further, if the data flow graph needs to be updated, the user needs to update the data flow graph in each of these different software applications. Since recreating the data flow graph requires repeated efforts, this traditional method is inefficient.

[0103] One possible solution is to store the output data generated when executing the data flow graph so that it can be used as input for other data flow graphs later. However, the data sets managed by the data processing system may be frequently updated, deleted, or otherwise modified. For example, in the case where the data processing system 100 manages data sets that store multinational bank information (e.g., customer information, account information, transaction information), these data sets may be frequently updated during the bank's operations. As another example, in the case where the data processing system 100 manages data sets that store credit card company data (e.g., transaction history, account balance, credit usage, etc.), these data sets may be frequently updated based on millions of credit card transactions that occur every day. In addition, after executing the data flow graph, it may be necessary to modify the data flow graph so that subsequent executions of the data flow graph will result in output data that is different from the first execution. In view of the dynamic nature of data and data flow graphs, the output data generated when a data flow graph is executed at a certain time may be different from the output data generated when the data flow graph is executed at another time. Therefore, the previously generated output data obtained when executing the data flow graph will lose its validity or expire. Furthermore, storing the output of performing large numbers (e.g., thousands or millions) of data flow graph generation would require significant additional storage resources, making this approach untenable.

[0104] Therefore, the inventors have developed a data processing system that stores data flow graphs and allows software applications to access these data flow graphs through a data set directory of the data processing system. Software applications (e.g., developed as data flow graphs) can access data flow graph datasets through the dataset directory to incorporate output data generated when the data flow graphs are executed. Unlike previous methods that can only register datasets in a dataset directory for access by software applications, the inventors have developed a system in which data flow graphs themselves can be registered in a dataset directory and subsequently accessed by software applications via the dataset directory. This allows the cataloged data flow graphs to be efficiently incorporated into multiple software applications using the dataset directory.

[0105] The techniques described herein allow previously developed software to be easily incorporated into several applications, thereby increasing the efficiency of developing software applications. The cataloged dataflow graph incorporated into the software application becomes a subgraph embedded in the software application and compiled and executed with the software application. The execution of the subgraph generates data that can be used in downstream operations of the software application. Thus, the cataloged dataflow graph allows the software application to incorporate data generated at runtime into the operation of the software application.

[0106] The data flow graph may be compiled into an executable software application and then executed. In some embodiments, the data processing system may transform the data flow graph prior to compilation to improve the execution efficiency of the software application. When a cataloged data flow graph (e.g., as a subgraph) is incorporated into a data flow graph, transforming the data flow graph may include transforming the subgraph. The data processing system may transform the subgraph in a manner customized for the software application (e.g., based on operations in the data flow graph downstream of the subgraph) and produce a more efficient software application.

[0107] Figure 2A 2 illustrates a data processing system 200 in which a data flow graph may be registered in a data set directory 204 of the data processing system 200 according to some embodiments of the techniques described herein. Figure 2A As shown in the example of FIG. 2 , data processing system 200 stores subgraph 208 identified within the data flow graph of software application 206A in data storage device 202 of data processing system 200. Data processing system 200 also generates entry 204A associated with cataloged data flow graph 202A in data set directory 204. Entry 204A can then be used by other software applications (e.g., developed as data flow graphs) to incorporate cataloged data flow graph 202A. Figure 1 Compared to a data processing system 100 (whose data directory 104 includes only entries for physical data sets), Figure 2AThe data processing system 200 may include entries of physical data sets and data flow graph data sets that may be incorporated into software applications.

[0108] like Figure 2A As shown, the system developed by the inventors allows a user to identify a data flow graph (e.g., in a storage device) or a subgraph of a larger data flow graph, and stores the identified data flow graph as a data flow graph dataset that can be accessed through a data directory. For example, a user may determine that a specific subgraph in a data flow graph may be useful in one or more other data flow graphs. The system allows a user to identify a specific subgraph and then stores the subgraph as a data flow graph dataset that can be accessed by a software application through a data directory. In some embodiments, the system may allow a user to highlight a portion of a data flow graph to indicate a subgraph, and then provide input through a UI menu to indicate a command for storing the highlighted portion as a data flow graph dataset. The stored data flow graph dataset can then be used for other software applications (e.g., developed as a data flow graph) like an input dataset. In some embodiments, the system may allow a user to provide input to indicate a command for making a previously stored subgraph into a data flow graph dataset that can be accessed by a software application through a data directory.

[0109] Figure 2B Illustration of how to use the Figure 2A The entry 204A associated with the cataloged data flow graph 202A in the incorporation of the cataloged data flow graph 202A into other software applications 206B, 206C. Figure 2B As shown in the example of , the cataloged data flow graph 202A is incorporated into the software applications 206B, 206C by associating the entry 204A with the input nodes 208A, 208B of the data flow graph of the software applications 206B, 206C. For example, the entry 204A may include a reference to the data flow graph 202A. The system 200 may use the entry 204 to configure the input nodes 208A, 208B to access the data flow graph 202A, respectively (e.g., before the data flow graph is compiled into an executable software application). The incorporated data flow graph may be compiled and executed as part of the software application. The data output by the incorporated data flow graph is obtained by performing the operations indicated by the incorporated data flow graph as part of executing the software application. Therefore, the data processing system 200 allows the cataloged data flow graph to be used as a source of dynamic data generated at runtime by the software application.

[0110] like Figure 2BAs shown, a software application (e.g., developed as a data flow graph) can access a data flow graph dataset through a dataset directory to incorporate output data generated by a subgraph. Therefore, the system allows subgraphs to be reused in multiple different software applications without the need for the user to recreate the subgraph. In addition, when the subgraph is executed as part of a software application, the output data generated by the subgraph will be the latest data when the subgraph is executed. The cataloged subgraph can be further updated after its creation. When executed as part of a software application, the cataloged subgraph can be executed in its latest state. Therefore, the cataloged subgraph allows the software application to incorporate the latest output data generated by the current version of the cataloged subgraph.

[0111] In some embodiments, the system can optimize a data flow graph generated by incorporating a catalog subgraph into a given data flow graph. The system can analyze the structure of the generated data flow graph (e.g., by analyzing the structure of the catalog subgraph, other data flow graphs, and how they are linked) to identify transformations that can be applied to the generated data flow graph so that a software application obtained by compiling the data flow graph can be executed more efficiently by a data processing system. As part of optimizing the generated data flow graph, the system can apply transformations to the subgraphs incorporated into the generated data flow graph. These transformations can be specific to the given data flow graph into which the catalog subgraph is incorporated. After these transformations, the generated subgraph may have a different structure than the catalog subgraph that was originally incorporated into the given data flow graph. The generated subgraph can be optimized so that it is executed as part of a given data flow graph.

[0112] For example, as part of optimizing a given data flow graph that incorporates a cataloged subgraph, the system may transform the incorporated subgraph by removing redundant operations in the subgraph, reducing the amount of data read from an input data set, and / or processing operations in the subgraph in parallel. The transformed subgraph may have a different structure than the cataloged subgraph. The system may then compile the given data flow graph including the transformed subgraph to obtain an executable software application. The software application may execute more efficiently than a software application obtained by compiling the given data flow graph without transforming the incorporated subgraph.

[0113] In some embodiments, a data processing system may have a data set directory to provide a data flow graph of a software application with access to data sets managed by the data processing system. The data set directory may have entries associated with individual data sets. A software application developed as a data flow graph may access the entries of the data set directory to incorporate the data sets as input into the data flow graph. In some embodiments, the system may store a subgraph as a data set by: (1) identifying the subgraph (e.g., within the data flow graph or in a storage device); (2) creating an entry associated with the identified subgraph in the data set directory, the entry being associated with information indicating the subgraph; and (3) configuring the data set directory to allow access to the entry associated with the subgraph. The cataloged subgraph may then be used as a data set in other data flow graphs by accessing the entry associated with the cataloged subgraph in the data set directory.

[0114] In some embodiments, a subgraph may have a single link representing data output by the subgraph. This characteristic of a subgraph enables the subgraph to be registered in a data directory as a subgraph data set that can be used by a software application. Output data generated when executing a cataloged subgraph may be provided at an input node associated with an entry associated with the subgraph. When a data flow graph having input nodes associated with the subgraph is compiled into an executable software application, the output link of the subgraph may be connected to a downstream node to which the input node is connected. The data generated by executing the subgraph will be provided as input to the data flow graph in the compiled software application.

[0115] In some embodiments, identifying a subgraph of a dataflow graph may include: (1) displaying a graphical representation of the dataflow graph in a UI (e.g., via graphical elements representing input data sets and operations); and (2) receiving, via the UI, a first user input indicating a subgraph within the dataflow graph. The system may receive, via the UI, user input commanding creation of an entry associated with the indicated subgraph. The entry may include information indicating information about the subgraph, such as nodes, links, configuration parameters, and / or other information.

[0116] In some embodiments, the system can provide a UI (e.g., a dataset catalog UI) through which a user can identify an entry in the dataset catalog associated with the identified subgraph (e.g., by selecting a graphical element representing the entry from a list of elements representing entries of the dataset catalog). Dataflow graphs of other software applications can access output data dynamically generated by the subgraph. The system can: (1) receive, via the UI, an identification of an entry associated with the cataloged subgraph; and (2) configure the dataflow graph of another software application to receive as input the output data generated when executing the cataloged subgraph (e.g., by associating one of the input nodes in the dataflow graph with the cataloged subgraph).

[0117] In some embodiments, the system can provide a UI having a searchable menu of entries in the dataset catalog. The system can receive identification of an entry associated with the cataloging subgraph by receiving, via the UI, user input indicating a selection of the entry from the searchable menu (e.g., from a list of search results). In some embodiments, the system can execute a dataflow graph of another software application by: (1) executing the cataloging subgraph to generate an output dataset; and (2) providing the generated output dataset as input to the dataflow graph to perform at least one of the one or more data processing operations using data in the output dataset.

[0118] In some embodiments, the system can be configured to optimize the data flow graph of other software applications before execution. The system can be configured to optimize the data flow graph by transforming the data flow graph to obtain an optimized data flow graph. The system can be configured to transform the data flow graph to obtain a transformed data flow graph that is more efficient to execute. For example, the system can remove redundant operations in the data flow graph, process one or more operations in the data flow graph in parallel, partition the data in the data flow graph, and / or limit the reading of input data to one or more fields used downstream in the data flow graph. Transforming the data flow graph can include transforming the catalog subgraph incorporated into the data flow graph (for example, because the catalog subgraph is associated with the input node of the data flow graph). For example, the system can identify various parts that can be optimized in the data flow graph. These parts can include at least a portion of the catalog subgraph incorporated into the data flow graph (for example, by associating the data directory entry with the input node of the data flow graph). As part of the data flow graph transformation, the system can transform a portion of the catalog subgraph.

[0119] In some embodiments, a data set may include data and information about the data. In some embodiments, the information about the data may be stored in attribute-value pairs. For example, a data set may include one or more attributes having values, and the information about the data in the data set may include the values ​​of the attributes. The data set may be stored by a data processing system in any suitable format and / or using any suitable (multiple) data structures, as aspects of the technology described herein are not limited in this respect.

[0120] In some embodiments, a data processing system may manage data sets for an organization (e.g., a multinational corporation (e.g., a financial institution, a utility company, an automobile company, an electronics company, etc.) or other enterprise or organization). Large organizations may have a large number of data sets, and thus, in some embodiments, a data processing system may be used to manage a large number (e.g., millions, billions, or trillions) of data sets for the organization. For example, in some embodiments, a data processing system may be configured to manage millions or billions of data sets. In some such embodiments, a data processing system may be used for metadata management in an enterprise environment, whereby a data set stores information about other data sets (e.g., tables, transactions, documents, data records, etc.) stored on a globally distributed information technology (IT) infrastructure (including many databases, data warehouses, data lakes, etc.). In this case, a data set may store information about a corresponding object, such as, for example, when the object was created, where it is stored, its size, the identity of the user(s) allowed to edit the object, information identifying which applications use the object, information identifying the sensitivity level of the data, and the like. Since large organizations (e.g., financial institutions such as banks or credit card companies, utilities such as telephone or electric companies) typically manage millions or billions of such data sets, there may be millions or billions of data sets storing information about such data sets that will be managed by the data processing system. Since in this example application, the data processing system will store information about other data (sometimes referred to as "metadata"), this example application may be referred to as "metadata management." However, it should be understood that the techniques described herein are not limited to data processing systems for metadata management, and may be applied to any data processing system that uses data sets to manage data, regardless of whether the data being managed is metadata or any other type of data.

[0121] The techniques described herein can be implemented in any of a variety of ways, as the techniques are not limited to any particular implementation. Examples of implementation details are provided herein for illustrative purposes only. In addition, the techniques disclosed herein can be used alone or in any suitable combination, as aspects of the techniques described herein are not limited to use with any particular technique or combination of techniques.

[0122] Figure 2A is a diagram illustrating cataloging of subgraphs as dataflow graph data sets in data processing system 200 according to some embodiments of the techniques described herein. Figure 2A It is illustrated that a subgraph is identified in software application 206A and stored in data processing system 200 as a cataloged dataflow graph. Figure 2B The cataloged data flow graph is subsequently incorporated into other software applications. Data processing system 200 includes data storage 202 and data set directory 204. Data processing system 200 is in communication with client device 210.

[0123] The client device 210 may be any suitable type of computing device. For example, the client device 210 may be a desktop, a laptop, a smart phone, or a tablet computer. The data processing system 200 may provide a canvas through which a user may create and / or edit a data flow graph. The data processing system 200 may be configured to provide a canvas as part of a UI, which a user of the client device 210 may use to develop a software application. The UI may include a display of a data flow graph of a software application. The UI may allow a user to modify a data flow graph, catalog a subgraph of a data flow graph, and / or incorporate a cataloged data flow graph into a data flow graph. The UI may include an interface for displaying a data set directory entry. The UI may allow a user to use the interface to incorporate a cataloged data set (e.g., a physical data set and / or a data flow graph data set). A user may select an entry from the interface to incorporate a data set associated with the entry into the data flow graph shown. For example, a user may drag an entry from the interface to an input node of a data flow graph to associate a cataloged data set with the input node.

[0124] exist Figure 2A In the example of FIG. 2 , a user of device 210 has developed software application 206A in data processing system 200. Software application 206A is developed as a data flow graph. The data flow graph of software application 206A includes subgraph 208. In some embodiments, data processing system 200 may be configured to identify subgraph 208 based on user input. For example, data processing system 200 may receive input through a GUI, wherein a user has identified subgraph 208 embedded in the data flow graph of software application 206A.

[0125] like Figure 2A As shown in the example embodiment of FIG. 2 , subgraph 208 has a single link representing an output of subgraph 208. Data outputted via the output link of subgraph 208 may subsequently be used in downstream operations. For example, data outputted by subgraph 208 may be used in one or more downstream operations in a data flow graph. The single output link of subgraph 208 may allow subgraph 208 to be registered as a data flow graph dataset in dataset directory 204 and subsequently incorporated into other software applications (e.g., to provide input data to the software application).

[0126] like Figure 2AAs shown in the example embodiment of , data processing system 200 stores subgraph 208 as cataloged data flow graph 202A in data storage device 202 of data processing system 200. Data flow graph 202A can be executed as part of a software application to obtain data. This is in contrast to a physical data set consisting of previously stored data. In some embodiments, data processing system 200 can be configured to generate a data record storing data flow graph 202A. For example, data processing system 200 can use a file configured to store data flow graphs. In this example, data processing system 200 can generate a file storing data flow graph 202A. The file can then be used to execute data flow graph 202A (e.g., as part of executing software application 206A or another software application that contains data flow graph 202A).

[0127] In some embodiments, data processing system 200 may be configured to store sub-graph 208 in data storage device 202 in response to a user command. Data processing system 200 may be configured to receive input via a GUI to indicate a command to store sub-graph 208 as a data set. Figures 3A to 3G An example of how the data processing system 200 can store a subgraph as a data set is described. In some embodiments, the data processing system 200 can be configured to allow updating of the data flow graph 202A stored in the data storage device 202. For example, a user can modify the data flow graph 202A in the data storage device 202 to modify the operations performed by the data flow graph 202A.

[0128] like Figure 2A As shown, data processing system 200 registers data flow graph 202A in data set directory 204. In some embodiments, data processing system 200 may be configured to register data flow graph 202A in data set directory 204 by generating entry 204A corresponding to data flow graph 202A. Entry 204A may be used to incorporate data flow graph 202A into another software application (e.g., Figure 2B ). In some embodiments, entry 204A may provide a reference to data flow graph 202A, which may be used to incorporate data flow graph 202A into another software application. For example, entry 204A may be used to associate data flow graph 202A with an input node of the data flow graph.

[0129] Figure 2B 2 is a diagram illustrating the use of data flow graph 202A in software applications 206B, 206C of data processing system 200 according to some embodiments of the techniques described herein. Figure 2BAs shown, data processing system 200 incorporates data flow graph 202A (which is a subgraph 208 of software application 206A) into software applications 206B, 206C using entry 204A. Data processing system 200 associates entry 204A with: (1) input node 208A of software application 206B; and (2) input node 208B of software application 206C. Data processing system 200 can be configured to execute data flow graph 202A to generate data at input nodes 208A, 208B, which is used by the corresponding software applications 206B, 206C.

[0130] For example, when software application 206B is executed, data flow graph 202A may be executed to obtain data for input node 208A. Data flow graph 202A is a subgraph 208 from software application 206A. Thus, data at input node 208A may be output generated by executing data flow graph 202A when software application 206B is executed. Similarly, when software application 206C is executed, data flow graph 202A may be executed to obtain data for input node 208B. Thus, data at input node 208B may be output generated by executing subgraph 208 when software application 206C is executed.

[0131] As reference Figure 2A As described, in some embodiments, the data flow graph 202A can be updated. For example, the data flow graph 202A can initially be a copy of the subgraph 208 from which the data flow graph 202A was created. The user can then modify the data flow graph 202A in the data storage device 202 (e.g., modify the operations performed by the data flow graph 202A). Therefore, when the data flow graph 202A is incorporated into the software application as a data set, the data flow graph 202A can be executed in its current form when the software application is executed.

[0132] In some embodiments, the data processing system 200 may be configured to compile a data flow graph and execute the compiled data flow graph. In some cases, the data processing system 200 may be configured to transform the data flow graph before compiling. The data processing system 200 may be configured to transform the data flow graph to optimize the execution of the data flow graph. The data processing system 200 may apply a series of one or more transformations to the data flow graph to obtain an optimized data flow graph. Then, the data processing system 200 may compile and execute the optimized data flow graph instead of the original data flow graph. Compared with compiling the original data flow graph, the data processing system 200 may execute the compiled optimized data flow graph more efficiently. For example, the optimized data flow graph may read less data than the original data flow graph, process some operations of the original data flow graph in parallel, and / or eliminate operations in the original data flow graph.

[0133] As an illustrative example, data processing system 200 may transform the data flow graph of software application 206B and / or the data flow graph of software application 206C to obtain an optimized data flow graph. Data processing system 200 may be configured to transform data flow graph 202A as part of transforming the data flow graphs of software applications 206B, 206C (e.g., because data flow graph 202A is executed together with the execution of software applications 206B, 206C). In some embodiments, data processing system 200 may be configured to transform data flow graph 202A of software applications based on the operation of the software applications. For example, data processing system 200 may transform data flow graph 202A as part of transforming the data flow graph of software application 206B. In this example, data flow graph 202A may be optimized for software application 206B. As another example, data processing system 200 may transform data flow graph 202A as part of transforming the data flow graph of software application 206C. In this example, the data flow graph 202A may be optimized for the software application 206C. Each of the software applications 206B, 206C may be compiled using a corresponding transformed version of the data flow graph 202A. In some cases, the transformation of the data flow graph 202A for one software application may be different than the transformation of the data flow graph 202B for another software application. In other cases, the transformation of the data flow graphs 202A for both software applications may be the same.

[0134] The transformation performed by the data processing system 200 may include optimizations for processing data according to one or more operations specified in the data flow graph 202A (relative to data processing without optimization, or transformation, or both). For example, the data processing system 200 adds one or more sort operations, data type operations, join operations (including join operations based on keys specified in the data flow graph 202A), partition operations, automatic parallel operations, or operations specifying metadata, etc., to produce a transformed data flow graph having the desired functionality of the data flow graph 202A. In some embodiments, the data flow graph is transformed into an optimized data flow graph by applying one or more data flow graph optimization rules to the data flow graph to improve the computational efficiency of the transformed data flow graph (relative to the computational efficiency of the data flow graph before the optimization is applied). The data flow graph optimization rules may include, for example, eliminating dead or redundant components, early screening, or narrowing the scope of records, etc., as described herein with reference to the transformation engine 412 of the data processing system 200.

[0135] FIG. 3A to FIG. 3D An example sequence of steps for storing a subgraph of dataflow graph 306 as a cataloged dataflow graph in data processing system 200 is illustrated. Figure 3E The data processing system 200 is illustrated after storing the subgraph as a cataloged dataflow graph. Figure 3F to Figure 3G Illustrated is an example sequence of steps for merging a cataloging dataflow graph into another dataflow graph.

[0136] Figure 3A 2 is a diagram of a software application development UI 220 of data processing system 200 shown on a display of device 210 that interacts with data processing system 200, according to some embodiments of the techniques described herein. Software application development UI 220 displays a data flow graph 306 of a software application. Figure 3A As shown, the software application receives data from a plurality of input nodes 310A, 310B, 310C, 310D and performs various data processing operations (e.g., filtering, sorting, joining, etc.) indicated at the nodes of the data flow graph 306 to generate data at the output node 312. A user of the device 210 may use the software application development UI 220 to generate the data flow graph 306. In some embodiments, the data flow graph 306 may be automatically generated by the data processing system 200. For example, the data flow graph 306 may be generated by the data processing system 200 based on a query input by a user.

[0137] Figure 3B Some embodiments of the technology described herein are illustrated. Figure 3A 308 within a data flow graph 306 in the software application development UI 220 of the present invention. In some embodiments, the data processing system 200 may be configured to identify the sub-graph 308 based on user input. For example, the data processing system 200 may identify the sub-graph 308 based on user input highlighting a portion of the data flow graph 306 (e.g., by clicking and dragging a box around the portion). In another example, the data processing system 200 may identify the sub-graph 308 based on user input clicking a node to be included in the sub-graph 308. In another example, the data processing system 200 may identify the sub-graph 308 based on a user selection of a link that marks an output of a portion of the data flow graph 306. Figure 3B In the example of , the identified subgraph 308 is a portion of the dataflow graph 306 that performs data processing operations on data received at input nodes 310A, 310B to generate output data that is provided to a sort operation at node 314.

[0138] Figure 3C Some embodiments of the technology described herein are shown. Figure 3B 308. The sub-picture menu 320 may include one or more options associated with the functionality associated with the sub-picture 308. Figure 3CIn the example of , the subgraph UI menu 320 includes an option 320A for copying the subgraph, an option 320B for deleting the subgraph, and an option 320C for saving the subgraph as a cataloged dataflow graph. The user can select option 320C to store the identified subgraph 308 as a cataloged dataflow graph (e.g., in the data storage device 202). For example, the user can select option 320C by clicking, tapping, or providing another form of input to indicate the selection of option 320C. In some embodiments, the user can enter a command to save the subgraph 308 as a cataloged dataflow graph without using the subgraph menu 320. For example, the user can enter a shortcut key, or select an icon in the software application development UI, to trigger the storage of the identified subgraph 308 as a cataloged dataflow graph.

[0139] In some embodiments, data processing system 200 may be configured to save sub-graph 308 in response to the user selecting option 320C. For example, data processing system 200 may be configured to save sub-graph 308 in a file stored in data storage device 202 of data processing system 202. Data processing system 200 may further determine a reference to the saved file (e.g., a URL, file path, or other type of reference). Data processing system 200 may include the reference in an entry registered for the sub-graph dataset in data directory 204.

[0140] In some embodiments, the data processing system 200 may organize the data directory 204 into multiple applications and / or projects. The data processing system 200 may be configured to determine the application or project in which the cataloged data flow graph is located. For example, the data processing system 200 may: (1) provide a list of applications and / or projects in a GUI from which a user can specify an application and / or project; and (2) determine the application and / or project based on a user selection from the list. In some embodiments, the data processing system 200 may be configured to determine a storage schema for data output by the cataloged data flow graph. The schema may be an organizational container (e.g., a directory) that stores data sources. In some embodiments, the data processing system 200 may be configured to: (1) provide a list of schemas in a GUI from which a user can select a schema for the cataloged data flow graph; and (2) determine a schema based on user input indicating a selection of one of the schemas.

[0141] In some embodiments, the data processing system 200 can be configured to determine a name for the cataloged data flow graph. For example, the data processing system 200 can generate a GUI that includes a field for entering a name for a subgraph data set. The data processing system 200 can name the cataloged data flow graph as indicated in the field. The data processing system 200 can be configured to display the determined name in a data directory entry list interface.

[0142] In some embodiments, the data set catalog 204 may organize the data into a plurality of collections (e.g., a shopping cart). The data processing system 200 may be configured to determine the collection to which the cataloged data flow graph belongs. The data processing system 200 may be configured to provide a GUI through which a user may specify the collection. For example, the data processing system 200 may: (1) provide a collection list in the GUI; and (2) determine the collection to catalog the data flow graph based on the collection selected by the user from the collection list.

[0143] In some embodiments, the data processing system 200 may be configured to determine a record format for data output when executing a cataloging data flow graph. A record format may describe the data output by the cataloging data flow graph. For example, a record format may indicate fields of data, formats of data in fields, primary keys, foreign keys, and / or other information. In some embodiments, the data processing system 200 may be configured to determine a record format inherited from an upstream data set (e.g., an input data set). If the data processing system 200 does not recognize an inherited record format, the data processing system 200 may allow a user to specify a record format. For example, the data processing system 200 may determine the record format using a user-specified file indicating a record format. In some embodiments, the data processing system 200 may be configured to: (1) provide a GUI through which a user can define a record format; and (2) determine a record format based on a user definition of a record format obtained through the GUI.

[0144] In some embodiments, the data processing system 200 can be configured to determine one or more keys (e.g., primary keys and / or foreign keys) of the data. In some embodiments, the data processing system 200 can be configured to determine keys inherited from upstream data. For example, the data processing system 200 can use keys from one or more input data sets used by the subgraph 208. In some embodiments, the data processing system 200 can be configured to obtain user-specified keys when the data processing system 200 does not recognize inherited keys. For example, the data processing system 200 can provide an interface through which a user can specify a key.

[0145] Figure 3D In some embodiments according to the techniques described herein, Figure 3C The sub-picture UI menu 320 receives the user input and Figure 3BThe subgraph 308 of the identified data flow graph 202B is cataloged as a data set directory 204 of the data processing system 200. As indicated by the arrow from the identified subgraph 308 to the data set directory 220, the data processing system 200 may be configured to store the subgraph 308 as a cataloged data flow graph 202B in the data storage device 202. The data processing system 200 may be configured to generate an entry 204B associated with the data flow graph 202B in the data set directory 204. The entry 204B may then be used to incorporate the data flow graph 202B into other software applications (e.g., by associating the entry 204B with an input node of the data flow graph).

[0146] Figure 3E 2 shows data set catalog 204 and data storage device 202 of data processing system 200 updated after cataloging sub-graph 308 according to some embodiments of the techniques described herein. Figure 3E As shown, data storage device 202 now includes cataloged dataflow graph 202B, including subgraph 308 identified in dataflow graph 306. Data set catalog 204 includes additional entry 204B associated with cataloged dataflow graph 202B. Entry 204B may be used to incorporate cataloged dataflow graph 202B into a software application.

[0147] Figure 3F 2 shows a software application development UI 220 in which another data flow diagram 326 on a display of device 210 interacts with data processing system 200 according to some embodiments of the techniques described herein. Figure 3F As shown, the software application development UI 220 includes a data set catalog UI 324, which displays a representation of Figure 3E The graphical elements of entries 204A, 204B, 204C, 204D, 204E of the updated dataset catalog 204. A user of device 300 may use dataset catalog UI 324 to incorporate datasets of data processing system 200 into data flow graph 326 shown in software application development UI 220.

[0148] In some embodiments, the software application development UI 220 may allow a user to associate a cataloged data set with an input node of a data flow graph 326. Figure 3F , the user has selected entry 204B to associate with input node 328. For example, software application development UI 220 may allow the user to drag a graphical element representing entry 204B to input node 328 to associate catalog dataflow graph 202B with input node 328. In another example, UI 302 may allow the user to select input node 328 and then select a graphical element representing entry 204B to associate catalog dataflow graph 202B with input node 328.

[0149] Figure 3G According to some embodiments of the technology described herein Figure 3F The data flow graph 326 of FIG. 10 is a data flow graph of FIG. 10 , wherein the catalog data flow graph 202B is associated with the input node 328. The catalog data flow graph 202B is incorporated into the data flow graph 326 by associating the entry 204B with the input node 328. FIG. 3A to FIG. 3E As described, the cataloged data flow graph 202B is a subgraph 308 identified from the data flow graph 306. Therefore, the data at the input node 328 can be output data generated when the cataloged data flow graph 202B is executed. In some embodiments, the data processing system can be configured to compile and execute the cataloged data flow graph 202B or its transformation when executing the data flow graph 326. For example, the data processing system 200 can compile and execute the cataloged data flow graph or its transformation when compiling and executing the software application. In some embodiments, the data processing system can independently compile and execute the cataloged data flow graph 202B. For example, the data processing system 200 can compile and execute the cataloged data flow graph 202B locally in response to a user command. Therefore, the data received at the input node 328 can be data output by the cataloged data flow graph 202B.

[0150] In some embodiments, the cataloged dataflow graph may be compiled and executed in its current state at the time the dataflow graph 326 is updated. In some cases, the cataloged dataflow graph 202B may be different from when it was originally saved as a data set. For example, the input data used in the dataflow graph 202B may be different from when the subgraph 308 was originally saved as the dataflow graph 202B. As another example, one or more operations of the dataflow graph 202B may have been modified since the dataflow graph 202B was created. Therefore, when the dataflow graph 326 is compiled and executed, the cataloged dataflow graph 202B may be in the latest version.

[0151] Figure 4A 2 is a block diagram illustrating a data processing system 200 according to some embodiments of the techniques described herein. Figure 2A As shown, data processing system 200 includes data set directory 204, data storage device 202, software application 206 and system module 400. Figure 4A As shown, the software application 206 can access entries in the data set catalog 204 to incorporate the cataloged data flow graphs and / or data sets into the software application 206.

[0152] Data set directory 204 includes entries for accessing corresponding data sets (e.g., data flow graph data sets and stored data sets) managed by data processing system 200. Figure 2AIn the example of , entries 202A, 202B, 202C, 202D, 202E can be used to access data sets 202A, 202B, 202C, 202D, 202E from the data storage device 202 of the data processing system 200 (for example, by associating the data set with an input node). In some embodiments, the data set directory 204 may include entries for accessing one or more data sets in a data storage device outside the data processing system 200. These entries may include entries associated with physical data sets. For example, entries 204C, 204D of the data set directory 204 are associated with corresponding physical data sets 202C, 202D. These entries may include entries associated with cataloged data flow graphs. The cataloged data flow graphs may each include a data flow graph that generates output data when executed. For example, entries 204A, 204B, 204C are associated with corresponding cataloged data flow graphs 202A, 202B, 202E. Reference is made to Figure 4G An example of an entry of the data set directory 204 is described.

[0153] In some embodiments, the data set directory 204 may be any suitable data structure for storing a collection of entries. For example, the data set directory 204 may be an array, where each element of the array is an entry of the data set directory 204. In another example, the data set directory 204 may be a hash table, where each entry includes a set of hash information for accessing a data set and is associated with a key that can be used to find the entry.

[0154] In some embodiments, data set catalog 204 may be searchable. Data processing system 200 may allow a user of data processing system 200 to search through entries in data set catalog 204. For example, data processing system 200 may provide a UI via which a user may enter a search query (e.g., a keyword) that data processing system 200 uses to identify one or more entries in data set catalog 204. In some embodiments, data processing system 200 may be configured to search the contents of entries of data catalog 202 and / or the contents of data sets in data storage device 202 to determine matches to the search query. For example, data processing system 200 may perform a keyword search on the contents of the entries and / or data sets.

[0155] In some embodiments, data set catalog 204 may provide software applications 206 of data processing system 200 with access to data sets as input data for use by the software applications. Figure 4AIn an example of the present invention, an entry of the data set directory 204 can be used to incorporate a physical data set from the data storage device 202 or output data generated when executing the cataloged data flow graph as an input in a software application. For example, to include data from the physical data set 202C as input in a software application (e.g., software application 206A), the software application can access the entry 204C of the data set directory 204. The entry 204C can provide information for obtaining the data set 202C to the software application 206A. The entry 204C can include a reference to the data set 202C in the data storage device 202, which can be used by the software application 206A to access the data set 202C. In another example, the entry 204D can include a reference to the data set 202D stored in the data storage device 202, which can be used by the software application 206A to access the data flow graph. In some embodiments, the entries of the data set directory 204 can be used to associate the cataloged data flow graph with the input node of the software application. When the software application is executed, it may use the information from the entry to obtain data (eg, by reading from a physical data set or executing a data flow graph or its transformations to generate output data).

[0156] Data storage device 202 may be any suitable storage device for data processing system 200. In some embodiments, data storage device 202 may include storage hardware for storing data sets, data flow graph information, and / or other information. For example, the storage hardware may include one or more hard disk drives (e.g., magnetic disk drives, solid state drives, and / or other hard disk drives). In some embodiments, data storage device 202 may include a distributed database. A distributed database may include data storage device resources in multiple geographic regions. A distributed database may include one or more data centers that store data sets and data flow graph information. Although in Figure 4A In the example of FIG. 2 , the data set directory 204 is shown as being separate from the data storage device 202 , but in some embodiments, the data set directory 204 may also be stored within the data storage device 202 .

[0157] like Figure 4AAs shown, the data storage device 202 can store physical data sets (such as data sets 202C, 202D) and cataloged data flow diagrams (such as data flow diagrams 202A, 202B, 202E). The physical data set can be a previously generated data set. For example, the physical data set can be a SQL table, an ORACLE database data set, a TERADATA database data set, a flat file, a multi-file data storage, a HADOOP database data set, a DB2 data storage, a Microsoft SQL SERVER data set, an INFORMIX data set, a table, or other types of data sets. In another example, the physical data set can be a document storing values ​​in one or more fields (for example, a structured document based on a pattern, or other unstructured documents). In another example, the physical data set can be an XML file, a JSON file, or other types of storage files. In another example, the physical data set can be an array, a linked list, a queue, or any other data structure suitable for storing data.

[0158] like Figure 4A As shown, data processing system 200 includes software application 206. Software application 206D is developed as a dataflow graph. The dataflow graph of software application 206D includes an input node (also called a "read component") configured to receive a data set via data set catalog 204. The dataflow graph is configured to receive the following as input: (1) data set 202C using entry 204C; and (2) output data generated when cataloged dataflow graph 202B is executed using entry 204B. The dataflow graph includes nodes at which operations are performed on the input data to generate an output data set. Software application 206E is another software application developed as a dataflow graph. The dataflow graph is configured to receive as input the output data generated when dataflow graph 202E is executed using entry 204E. Although Figure 4A The example of 206D and 206E shows software applications 206D and 206E, but data processing system 200 may include any number of software applications. For example, data processing system 200 may include hundreds or thousands of such software applications.

[0159] In some embodiments, the software application developed as a data flow graph can be manually created by a user. For example, data processing system 200 can provide a UI (e.g., UI 220) that a user can use to develop a data flow graph. In some embodiments, the software application developed as a data flow graph can be generated programmatically. For example, data processing system 200 can include a software application that generates a data flow graph.

[0160] In some embodiments, a software application can access a physical data set by using the information in an entry of the data set directory 204, thereby using the entry to access the physical data set. For example, a software application can access a physical data set using a reference to a physical location stored in an entry (e.g., a URL, a key, or other type of reference). In some embodiments, an entry associated with a physical data set may include a reference to a software application for accessing the physical data set. For example, a software application can convert a physical data set into a logical data set that the software application uses as input. In some embodiments, a software application for accessing a physical data set can be developed as a data flow graph. For example, a software application developed as a data flow graph can perform various data processing operations to change the data format from a physical data set to a logical data set for use by other software applications. An example of an application that accesses a physical data set using an entry of a data set directory is described in U.S. Patent Application Publication No. 2022 / 0245125 (titled "Dataset Multiplexer for Data Processing System"), which is incorporated herein by reference in its entirety.

[0161] In some embodiments, a software application can use information in an entry to incorporate a cataloged dataflow graph. The entry can be used to configure the software application to: (1) execute the cataloged dataflow graph; and (2) receive as input output data generated when executing the cataloged dataflow graph. For example, the output data generated when executing the cataloged dataflow graph can be used in one or more operations performed by the software application.

[0162] In some embodiments, the data processing system 200 may be configured to use an entry to configure a data flow graph of a software application to receive output data generated by a cataloged data flow graph. The data processing system 200 may be configured to configure the data flow graph by associating the cataloged data flow graph with an input node of the data flow graph. For example, the data processing system 200 may use a reference to information indicating the cataloged data flow graph in the data storage device 202 to associate the cataloged data flow graph with the input node. In another example, the entry may store the cataloged data flow graph, and the data processing system 200 may associate the cataloged data flow graph from the entry with the input node. In some embodiments, the data processing system 200 may be configured to associate the cataloged data flow graph with the input node by including a reference to the cataloged data flow graph at the input node. When the cataloged data flow graph is associated with an input node of an incorporated data flow graph, the cataloged data flow graph may be executed as part of the incorporated data flow graph. The execution of the cataloged data flow graph may generate output data, and the incorporated data flow graph may use these output data to perform subsequent operations.

[0163] although Figure 4A 2. Data processing system 200 may also include software applications that are not developed as data flow graphs. In some embodiments, entries of data set directory 204 may be used by software applications that are developed as data flow graphs (e.g., software applications 206D, 206E) as well as software applications that are not developed as data flow graphs. For example, a software application may perform operations using metadata about a data set stored in data storage device 202. In this example, the software application may access information about the data set from an entry of data set directory 204. In another example, the software application may be developed as software code and use the relevant entry of data set directory 204 to access the data set.

[0164] Figure 4B A block diagram of a system module 400 of a data processing system 200 according to some embodiments of the techniques described herein is shown. As shown in FIG4 , the system module 400 includes a data flow graph generator 402, a data set directory module 404, a data flow graph storage module 406, a software application development UI module 408, a data set directory UI module 410, a transformation engine 412, a compiler 414, and a data flow graph execution engine 416.

[0165] The data flow graph generator 402 may be configured to generate a data flow graph for a software application. In some embodiments, the data flow graph generator 402 may be configured to obtain user input indicating a data flow graph through a graphical UI to generate a data flow graph. A user may layout nodes and links representing input data sources, data processing operations, outputs, and / or data flows in the graphical UI. In some embodiments, the data flow graph generator 402 may be configured to automatically generate a data flow graph. For example, the data flow graph generator 402 may be configured to generate a data flow graph based on a query. The data flow graph generator 402 may automatically layout nodes and links to execute a query.

[0166] In some embodiments, data flow graph generator 402 may be configured to generate a data flow graph. In some embodiments, data flow graph generator 402 may be configured to generate a data flow graph for an application by: (1) obtaining a user definition of a data flow graph (e.g., in a software application development UI); and (2) generating a data flow graph for the application based on the user definition. In some embodiments, data flow graph generator 402 may be configured to save the user-defined data flow graph as a software application in data processing system 200. Data processing system 200 may access and execute the software application (e.g., to analyze data or perform processing as part of a task). In some embodiments, data flow graph generator 402 may be configured to compile the data flow graph into a software application.

[0167] In some embodiments, the data flow graph generator 402 may be configured to identify a subgraph within a data flow graph. The data flow graph generator 402 may be configured to identify a subgraph by identifying a portion of the data flow graph as a subgraph. In some embodiments, the data flow graph generator 402 may be configured to identify a portion of the data flow graph based on user input specifying a portion of the data flow graph (e.g., as described herein with reference to Figure 3B For example, the data flow graph generator 402 may be configured to store the identified subgraphs as a cataloged data flow graph, as described herein with reference to Figure 3D described.

[0168] In some embodiments, data set directory module 404 may be configured to manage data set directory 204 of data processing system 200. Data set directory module 404 may be configured to manage the creation, deletion, and / or modification of entries of data set directory 204. For example, data set directory module 404 may create a new entry for a new data set added to the data processing system. Data set directory module 404 may create a new entry by: (1) instantiating a data object for the entry; (2) adding information to the data object (e.g., Figure 2B 2C); and (3) storing the data object as a new entry. The data directory module 404 can be further configured to modify an existing entry. For example, the data directory module 404 can update information for accessing the data flow graph in the entry associated with the data flow graph and / or update information about the data flow graph stored in the entry.

[0169] In some embodiments, the data set catalog module 404 can be configured to provide access to data sets (e.g., physical data sets and / or data flow diagram data sets). The data set catalog module 404 can provide software applications with access to data sets through entries associated with the data sets. For example, the data set catalog module 404 can generate a data set catalog UI menu, allowing a user to select an entry for incorporating a related data set into a data flow diagram (e.g., as described herein with reference to FIG. 1 ). Figure 3F In some embodiments, the data set directory module 404 can be configured to provide access to data sets by allowing software applications to reference entries of the data set directory 204. For example, executable instructions of a software application can reference entries of the data set directory 204 to incorporate a data set. In another example, the data set directory module 404 can include one or more software applications that provide information from entries of the data set directory to other software applications.

[0170] In some embodiments, the data flow graph storage module 406 may be configured to manage the storage of the data flow graph. The data flow graph storage module 406 may be configured to indicate information of the data flow graph. For example, the data flow graph storage module 406 may store information indicating the nodes and links of the data flow graph. The data flow graph storage module 406 may be further configured to store configuration parameters of the data flow graph. For example, the data flow graph storage module 406 may store the name, location (e.g., file path) of the data flow graph and / or other configuration parameters of the data flow graph. In some embodiments, the data flow graph generator 402 may be configured to generate a file storing the data flow graph. The file may store information indicating the nodes and links of the data flow graph. The file may indicate operations at nodes in the data flow graph. For example, the file may indicate one or more data processing operations (e.g., filtering, connecting, sorting and / or other operations) to be performed at nodes in the data flow graph. The file may further store information indicating input data sets associated with one or more nodes, one or more data links and / or data processing operations of one or more nodes. In some embodiments, the input node can obtain data from a physical data set or data output by an executed subgraph (e.g., a cataloged dataflow graph incorporated as a subgraph). In some embodiments, an entry in the data set directory 204 can refer to a file that stores information about a dataflow graph. The entry can be used to incorporate the dataflow graph into other dataflow graphs (e.g., dataflow graphs of other software applications).

[0171] In some embodiments, the data flow graph storage module 406 may be configured to store the data flow graph as a cataloged data flow graph. The data flow graph storage module 406 may be configured to generate an entry of a data set directory associated with the cataloged data flow graph (e.g., as described herein with reference to Figure 2A The data flow graph storage module 406 may be configured to catalog the data flow graph by storing a data record (e.g., a document, file, or other type of record) having information about the data flow graph (e.g., information indicating nodes, links, and configuration parameters of the data flow graph). The data flow graph storage module 406 may be configured to store the data flow graph and its subgraphs as cataloged data flow graphs. For example, the data flow graph storage module 406 may store the entire data flow graph as a cataloged data flow graph with associated entries in a data set directory. In another example, the data flow graph storage module 406 may store a subgraph within the data flow graph as a cataloged data flow graph with associated entries in the data set directory.

[0172] In some embodiments, the software application development UI 408 can be configured to generate a GUI that allows a user to develop a software application as a data flow graph. The GUI allows the user to lay out the nodes and links of the data flow graph of the software application. The GUI can allow the user to save the data flow graph for execution. In some embodiments, the software application development UI module 408 can be configured to allow the user to identify a subgraph in the data flow graph. For example, the GUI can allow the user to highlight a portion of the data flow graph and enter a command to generate a subgraph from the highlighted portion. The GUI can also provide a menu through which the user can enter a command to store the subgraph as a cataloged data flow graph (e.g., as described herein with reference to Figure 3C The data processing system 200 may store the subgraph as a cataloged dataflow graph (e.g., as described herein with reference to FIG. 3D to FIG. 3E described above).

[0173] In some embodiments, the dataset directory UI module 410 may be configured to generate a UI for displaying graphical elements representing entries of a dataset directory. In some embodiments, the dataset directory UI module 410 may be configured to generate a UI as part of a GUI for developing a software application as a dataflow diagram. For example, the dataset directory UI module 410 may generate a pane within the GUI for showing a list of entries. The dataset directory UI module 410 may be configured to allow a user to access datasets through the UI. For example, the UI may allow a user to drag a graphical element representing an entry to an input data source of a dataflow diagram. As another example, the UI may allow a user to right-click on a graphical element representing an entry and select an option to incorporate the dataset associated with the entry into the dataflow diagram. References herein Figure 3F to Figure 3G Describes an example of a dataset catalog UI and how it can be used to incorporate datasets into a dataflow graph.

[0174] In some embodiments, the transformation engine 412 may be configured to transform the data flow graph into a transformed data flow graph that can be compiled and executed. The execution efficiency of the transformed data flow graph may be higher. For example, the original data flow graph may: (1) include nodes representing redundant data processing operations; (2) require the performance of data processing operations whose results are not subsequently used; (3) require unnecessarily performing serial processing when parallel processing is possible; (4) apply data processing operations to more data than is necessary to obtain the desired results; (5) decompose the computation to be performed on multiple nodes, where the data processing at each data flow graph node is performed by a dedicated thread in a computer program, a dedicated computer program (e.g., a process in an operating system), or a dedicated computing device, which significantly increases the computational cost of performing the computation; (6) require the performance of stronger types of data processing operations (e.g., sorting operations, roll-up operations, etc.) that require more computation when weaker types of data processing operations (e.g., intra-group sorting operations or intra-group roll-up operations, etc.) that require less computation are sufficient; (7) require repeated processing work; or (8) do not include operations or other transformations that are useful or necessary for processing the data, or a combination thereof.

[0175] Therefore, the transformation engine 412 can be configured to apply the following one or more transformations necessary to process data according to the operations specified in the data flow graph to the data flow graph, and / or to improve data processing according to the operations specified in the data flow graph relative to data processing without using the transformation. For example, a user can create a data flow graph without specifying low-level implementation details, such as sorting and partitioning operations. However, these operations may be useful or necessary for processing data according to the operations specified in the data flow graph in the transformed data flow graph, and the processing of data can be improved according to the operations specified in the data flow graph (for example, by increasing the processing speed and / or reducing the consumption of computing resources, etc.), or both. Therefore, the transformation engine 412 can add one or more operations to the transformed data flow graph, such as sorting operations, data type operations, connection operations through specified keys, partition operations, automatic parallel operations, or operations specifying metadata, etc., to optimize or implement the operations specified in the data flow graph. In some cases, at least one operation added to the transformed data flow graph may not exist or be not represented in the original data flow graph. The transformation engine 412 can remove certain operations (for example, redundant operations).

[0176] In some embodiments, the transformation engine 412 can be configured to add operations by inserting one or more nodes representing the added operations into the data flow graph used to generate the transformed data flow graph. In some embodiments, the transformation engine 412 can be configured to directly insert the added operations into the transformed data flow graph without modifying the nodes of the data flow graph. The transformation engine 412 can add these operations to all data flow graphs when generating its corresponding transformed data flow graph, can add these operations based on the operations included in the data flow graph (these operations can be identified using pattern matching techniques, as described below), or can add these operations based on some other optimization rules.

[0177] In some embodiments, the transformation engine 412 may be configured to transform the data flow graph by applying one or more data flow graph optimization rules to the data flow graph to improve the computational efficiency of the transformed data flow graph, such as by removing dead or redundant components (e.g., by removing one or more nodes corresponding to the dead or redundant components), moving the filtering step to an earlier position in the data flow (e.g., by moving one or more nodes corresponding to the filtering component), or reducing the scope of the records, etc. In this way, the transformation engine 412 transforms the data flow graph into an optimized transformed data flow graph before compilation.

[0178] In some embodiments, the transformation engine 412 may be configured to identify two adjacent nodes in a data flow graph that represent corresponding operations, wherein the second operation repeats the effect of the first operation or invalidates it, so that one of the operations is redundant. Therefore, the transformation engine 412 may be configured to transform the data flow graph by removing nodes representing redundant operations (e.g., nodes representing repeated or invalid operations) when generating the transformed data flow graph. For example, the transformation engine 412 may identify two adjacent nodes with the same operation. Since two adjacent nodes performing the same operation are usually redundant, it is not necessary to perform both operations at the same time, and one of the two adjacent nodes may be removed. As another example, the transformation engine 412 may identify two adjacent nodes, wherein the first node represents a repartitioning operation (which partitions the data for parallel processing on different computing devices), and the latter node represents a serialization operation (whose operation is used to combine all data for serial processing by a single computing device). Since the effect of the repartitioning will be offset by the subsequent serialization operation, it is not necessary to perform the repartitioning operation (e.g., the repartitioning operation is redundant), and the transformation engine 412 may remove the repartitioning operation during the transformation process.

[0179] In some embodiments, the transformation engine 412 may be configured to identify a first node that represents a first operation that is exchanged with one or more other nodes that represent other operations. If the first node is exchanged with one or more other nodes, the transformation engine 412 may transform the data flow graph by changing the order of the first node and at least one of the one or more other nodes (e.g., by rearranging the order of the nodes). In this way, the transformation engine 412 may transform the data flow graph by sorting the nodes and the corresponding operations to improve processing efficiency, speed, or otherwise optimize the processing of the data flow graph without changing the result. Further, by exchanging nodes in this manner, the transformation engine 412 may be configured to apply other transformations. For example, the transformation engine 412 may change the order of the first node and at least one of the one or more other nodes so that the first node representing the first sorting operation is placed adjacent to the second node representing the second sorting operation. As a result, the first sorting operation and the second sorting operation become redundant, and the transformation engine 412 may transform the data flow graph by removing one of the first sorting operation or the second sorting operation (e.g., by removing the corresponding node from the data flow graph when generating the transformed data flow graph).

[0180] In some embodiments, the transformation engine 412 can be configured to identify and remove "dead" nodes representing unused or otherwise unnecessary operations. For example, the transformation engine 412 can identify one or more nodes representing operations whose results are not referenced or used (e.g., a sort operation that is not referenced because the order produced by the sort operation is not needed or relied upon in subsequent processing). Therefore, the transformation engine 412 can be configured to transform the data flow graph by removing dead or unused operations (e.g., by removing corresponding nodes when generating the transformed data flow graph).

[0181] In some embodiments, the transformation engine 412 can be configured to perform a strength reduction transformation on one or more nodes. For example, the transformation engine 412 can identify a first node representing a first operation of a first type (e.g., a first sort operation on a primary key, a first rollup operation on a primary key, etc.), followed by a second node representing a second operation of a second, weaker type (e.g., a second operation on a secondary key, a sort operation within a group, a second rollup operation on a secondary key, a group rollup operation, etc.). Because processing data through the first operation may require more computing resources than processing data through the second, weaker operation, the transformation engine 412 can perform a strength reduction transformation to replace the first operation with the second operation.

[0182] In some embodiments, the transformation engine 412 can be configured to transform the data flow graph by combining two or more nodes. For example, the transformation engine 412 can identify individual nodes representing operations that can be performed by different processes running on one or more computing devices, and can transform the data flow graph by combining these individual nodes and their respective operations into a single node so that all operations are performed by a single process executing on a single computing device, which can reduce the overhead of inter-process (and potentially inter-device) communication. The transformation engine 412 can be further configured to identify other nodes that can be combined, such as two or more individual join operations that can be combined, or a filter operation that can be combined with a rollup operation, and many other combinations.

[0183] In some embodiments, the transformation engine 412 may be configured to identify nodes configured to perform several operations that may be more efficient when performed separately. The transformation engine 412 may be configured to perform a serial-to-parallel transformation on a data flow graph, which decomposes one or more of the several operations into separate nodes for parallel processing (e.g., automatic parallel operation). These operations can then be performed in parallel using different processes running on one or more computing devices. The transformation engine 412 can then add a merge operation to merge the results of the parallel operation. In some embodiments, the transformation engine 412 may be configured to identify points in the data flow graph that contain large blocks of data (e.g., data corresponding to larger tables and indexes), and perform a partition transformation on the data flow graph to divide the data into smaller partitions (e.g., automatic partition operation). These partitions can then be processed serially or in parallel (e.g., by combining the automatic partition operation with the automatic parallel operation). The transformation engine 412 can significantly improve the efficiency of the transformed data flow graph by reducing the size of the data to be processed, or by separating operations for parallel processing, or both.

[0184] In some embodiments, the transformation engine 412 can be configured to perform a width reduction transformation when generating a transformed data flow graph. For example, the transformation engine 412 can identify data (e.g., one or more columns of data) to be deleted at a certain point in the data flow graph before performing subsequent operations, because the data (e.g., the data to be deleted) will not be used in subsequent operations and does not need to be propagated as part of the processing. As another example, the nodes in the data flow graph can be configured to perform several operations, and the results of some of the calculations may not be used. Therefore, the transformation engine 412 can perform a width reduction transformation to remove unused or other unnecessary data (e.g., by inserting nodes to delete data at the identified points, by replacing nodes configured to perform several operations with another node configured to perform only operations whose results are used, etc.). In this way, the transformation engine 412 optimizes the data flow graph by reducing the computing resources required for the data flow graph to transfer data in subsequent operations (e.g., by reducing the network, memory, and processing resources utilized).

[0185] In some embodiments, transforming a data flow graph may include transforming a cataloged data flow graph that is incorporated as an input into the data flow graph (e.g., by associating a data directory entry associated with the cataloged data flow graph with an input node of the data flow graph). Before the data flow graph is transformed, the cataloged data flow graph may be integrated into the data flow graph. For example, the cataloged data flow graph may be accessed using an entry associated with an input node of a subgraph and copied into the data flow graph. The output link of the cataloged data flow graph may be further connected to a node of the data flow graph (e.g., a node to which the output link of an input node is connected). The transformation engine 412 may be configured to transform the data flow graph resulting from the integration of the cataloged data flow graph. The transformation engine 412 may be configured to transform the integrated cataloged data flow graph as part of a data flow graph transformation. Therefore, the transformed cataloged data flow graph may be optimized for the data flow graph into which the transformed cataloged data flow graph is incorporated. The transformed cataloged data flow graph may have a different structure than the cataloged data flow graph that was initially incorporated into the data flow graph. Furthermore, a cataloged dataflow graph integrated into a first dataflow graph may have a different structure after undergoing a transformation than the same cataloged dataflow graph undergoing a transformation incorporated into a second dataflow graph.

[0186] In order to identify the parts of the data flow graph to which one or more transformations are to be applied (e.g., a cataloged data flow graph incorporated into a data flow graph), the transformation engine 412 is configured to employ a data flow graph pattern matching language. The data flow subgraph pattern matching language may include one or more expressions for identifying specific nodes or operations in the data flow graph for optimization, as described in detail below. For example, the pattern matching language may include an expression for identifying a series of nodes of at least a threshold length (e.g., at least two, three, four, five, etc.), which represent a series of corresponding calculations that can be combined and represented by a single node in the data flow graph using a combined operation optimization rule. Identifying such a pattern can facilitate the application of the combined operation optimization rule as described above. A preferred but non-limiting example of such an expression is "A-----+B-----+C---+D", which can help identify a series of four consecutive data processing operations that can be combined.

[0187] As another example, the pattern matching language may include expressions for identifying data flow graphs in which certain types of nodes can be exchanged with other nodes to optimize portions of the data flow graph. This can facilitate the application of multiple different types of optimization rules to the data flow graph. When the transformation engine 412 determines that the order of one or more nodes in the data flow graph can be changed without changing the processing results, this allows the transformation engine 412 to consider changes to the structure of the data flow graph (as allowed by the degrees of freedom available through the exchange operation) in order to identify portions to which the optimization rules can be applied. As a result of considering changes based on the exchange, one or more optimization rules may become applicable to a portion of the graph, while the original (multiple) rules do not apply to this portion.

[0188] For example, as described above, an optimization rule may involve identifying two adjacent nodes in an initial data flow graph that represent corresponding sorting operations, wherein the second sorting operation would invalidate the effect of the first operation, making the first operation redundant. By definition, such an optimization rule will not be applied to a data flow graph that does not have adjacent nodes representing sorting operations. However, if a first node representing a first sorting operation is to be exchanged with one or more other nodes, the order of the first node and at least one of the one or more other nodes can be changed so that the first node representing the first sorting operation is placed adjacent to the second node representing the second sorting operation. As a result of exchanging nodes in this manner, an optimization rule that removes redundant first sorting operations can be applied to the data flow graph.

[0189] Therefore, in some embodiments, the pattern matching language may include one or more expressions for identifying a subgraph of a data flow graph that conforms to a situation in which the order of nodes in the data flow graph can be changed. As an example, the expression "A---+(...)--+B" (wherein each of A and B can be any suitable data processing operation, such as sorting, merging, etc.) can be used to find a portion of a data flow graph having the following items: a node "A" (e.g., a node representing operation "A") and a node "B" (representing operation "B"), and one or more nodes that are exchanged with node "A" between node "A" and node "B" (e.g., when the order of the nodes is changed, the result of the processing performed by these nodes will not change). If such a portion is identified, the data flow graph can be changed or optimized by moving node "A" adjacent to node "B" to obtain a portion "AB". As a specific example, if the data flow graph has a node "ACDB" and operation "A" is exchanged with operations "C" and "D", the data flow graph can be changed to "CDAB". Furthermore, the transformation engine 412 can consider whether the optimization rule applies to the portion "AB". For example, if operation "A" is a sort and operation "B" is a sort, the transformation engine 412 may attempt to determine whether both sorts can be replaced with a single sort to optimize the dataflow graph.

[0190] As another example, the expression "A---+(...)-----+B*" can be used to find a portion of a data flow graph that has the following: a node "A", a second node "B", and one or more nodes between the two nodes that are swapped with node "B". As a specific example, if the data flow graph has a node "ACDB", and operation B is swapped with operations "C" and "D", the data flow graph can be changed or optimized to "ABCD". In turn, the transformation engine 412 can consider whether the optimization rule applies to the portion "AB".

[0191] As another example, the expression "A---+(...)-----+B**" can be used to find a portion of a data flow graph that has the following: node "A", node "B", and one or more nodes (e.g., "C" and "D") between node "A" and node "B" that are not swapped with node "B". In this case, the system may attempt to perform a "forced" swap, where nodes "C" and "D" are pushed to the left of node "A" if possible. As a specific example, if the data flow graph has node "ACEDB", and operation "B" swaps with operation "E" but not with operations "C" and "D", the data flow graph can be changed to "CDABE" - swapping B with E and pushing "C" and "D" to the left of "A".

[0192] As yet another example, the expression "A**-----+(...)-----+B" can be used to find a portion of a data flow graph that has the following: node "A", node "B", and one or more nodes (e.g., "C" and "D") between node "A" and node "B" that are not swapped with node "A". In this case, the system may attempt to perform a "forced" swap, where nodes "C" and "D" are pushed to the right of node "B" if possible. As a specific example, if the data flow graph has node "ACEDB", and operation "A" swaps with operation "E" but not with operations "C" and "D", the data flow graph can be changed to "EABCD" - swapping node "A" with "E" and pushing "C" and "D" to the right of "B".

[0193] In some embodiments, the transformation engine 412 can be configured to iteratively transform the dataflow graph, wherein each iteration of the optimization or transformation transforms the dataflow graph until testing indicates that further optimization or transformation is not possible, necessary, or desired. For example, the transformation engine 412 can transform the dataflow graph by: (1) selecting a first optimization rule; (2) identifying a first portion of the dataflow graph to which the first optimization rule is to be applied; and (3) applying the first optimization rule to the first portion of the dataflow graph. Subsequently, the transformation engine 412 can determine whether one or more additional optimizations can be applied to the dataflow graph, or whether it is necessary to generate a transformed dataflow graph that can be compiled and executed. If additional optimizations are applicable, the transformation engine 412 can continue to update the dataflow graph by: (1) selecting a second optimization rule different from the first optimization rule; (2) identifying a second portion of the dataflow graph to which the second optimization rule is to be applied; and (3) applying the second optimization rule to the second portion of the dataflow graph. Without further optimization or transformation, the transformation engine 412 can be configured to output the transformed data flow graph to store the transformed data flow graph and / or output the transformed data flow graph to a compiler module 414, which compiles the transformed data flow graph into an executable software application (e.g., which can be executed by the execution engine 416).

[0194] The dataflow graph optimization techniques that can be used by the transformation engine 412 are described in U.S. patent application publication No. 2021 / 0232579, entitled “Editor for Generating Computational Graphs,” which is incorporated herein by reference in its entirety.

[0195] In some embodiments, the transformed data flow graph generated by the transformation engine 412 can be stored in the data storage device 202 (e.g., by the data flow graph storage module 406). In some embodiments, the transformation engine 412 can be configured to transform the data flow graph and store the data flow graph for subsequent compilation (e.g., by the compiler module 414). In some embodiments, the transformation engine 412 can be configured to transform the data flow graph as part of compiling the data flow graph. For example, when compiling a given data flow graph, the transformation engine 412 can be used.

[0196] In some embodiments, compiler module 414 may be configured to compile a data flow graph (e.g., a transformed data flow graph) for execution (e.g., by data flow graph execution engine 416). Compiler module 414 may be configured to compile the data flow graph into an executable software application that may be executed by data processing system 200. In some embodiments, compiler module 414 may be configured to store the compiled software application in a data storage device of data processing system 200. Data processing system 200 may then execute the stored software application at a subsequent time. For example, the software application may be executed in response to a user command and / or executed programmatically.

[0197] In some embodiments, the data flow graph execution engine 416 can be configured to execute a data flow graph (e.g., a compiled data flow graph) of a software application. In some embodiments, the data flow graph execution engine 416 can be configured to execute the data flow graph by: (1) generating a set of instructions based on the data flow graph (e.g., the nodes and links of the data flow graph); and (2) executing the set of instructions. In some embodiments, the data flow graph execution engine 416 can be configured to use a software application that interprets and executes the data flow graph. For example, the data flow graph execution engine 416 can call a program that interprets the data flow graph and generates computer executable instructions based on the data flow graph. Techniques for executing computations encoded by data flow graphs are described in U.S. Patent Nos. 5,966,072, entitled "Executing Computations Expressed as Graphs," and 7,716,630, entitled "Managing Parameters for Graph-Based Computations," each of which is incorporated herein by reference in its entirety.

[0198] In some embodiments, the data flow graph execution engine 416 can be configured to generate output data obtained as a result of executing the data flow graph. The data flow graph execution engine 416 can be configured to execute the data flow graph of the data flow graph data set to generate the output data (e.g., as part of executing a software application). The software application can then use the output data for subsequent data processing. For example, the software application can be developed as a first data flow graph, and the output data generated by executing the data flow graph from the data flow graph data set can be used to perform one or more data processing operations in the first data flow graph.

[0199] Figure 4C is a diagram illustrating some embodiments of the technology described herein Figure 4B A diagram of the interactions between system modules. Figure 4C As shown, the system module 400 performs graph generation 420 and graph execution 430 .

[0200] The software application development user interface module 408 can allow a user (e.g., of the device 210) to develop a software application as a dataflow graph (e.g., in a graphical development environment). The dataflow graph generator 402 can generate a dataflow graph based on user definitions in the GUI. The GUI can further allow the user to store a subgraph of the dataflow graph as a cataloged dataflow graph. The dataset directory module 404 can register the subgraph in the dataset directory 204 (e.g., by generating an entry in the dataset directory 204 corresponding to the dataflow graph dataset stored in the data storage device 202). The dataset directory UI module 410 can display the entry corresponding to the cataloged dataflow graph in the dataset directory UI, which can be displayed in the software application development GUI (e.g., as Figure 3F shown).

[0201] like Figure 4C As shown, the data flow graph generated by the data flow graph generator 402 can be stored in the data storage device 202 of the data processing system 200 by the data flow graph storage module 406. In some embodiments, the transformation engine 412 can be configured to transform the data flow graph before storage. Therefore, the data flow graph storage module 406 can be configured to store the transformed data flow graph in the data storage device 202. In some embodiments, the generated data flow graph can be stored in the manner in which it was originally generated (for example, without being transformed by the transformation engine 412).

[0202] The data flow graph may be transformed by the transformation engine 412 to obtain a transformed data flow graph (e.g., an optimized version of another data flow graph). The compiler 414 may compile the transformed data flow graph generated by the transformation engine 412 to obtain a compiled software application (e.g., an executable program). Then, the execution engine 416 may execute the compiled software application.

[0203] Figure 4D 4 is a diagram illustrating a data flow graph 450 including a sub-graph 452 incorporated into the data flow graph 450, according to some embodiments of the techniques described herein. Figure 4D In the example of FIG. 4 , subgraph 452 is a cataloged dataflow graph 202A that is incorporated into dataflow graph 450 using entry 204A of data set catalog 204 of data processing system 200. Figure 4D As shown, subgraph 452 includes two input nodes 452A, 452E, and subgraph 452 receives input data at these two input nodes. The input data from node 452A is provided to node 452B that performs a filtering operation, and the output of the filtering operation is provided to node 452C that performs a sorting operation. The input data from node 452E is provided to node 452F that performs a sorting operation. The outputs of nodes 452C and 452F are provided as inputs to node 452D that performs a join operation. The output of subgraph 452 is then provided as input to a node of data flow graph 450, where a join operation with the results of other operations performed in data flow graph 450 is performed.

[0204] Figure 4E 4 is a diagram illustrating a transformation performed by transformation engine 412 on data flow graph 450 according to some embodiments of the techniques described herein. Transformation engine 412 may be configured to identify portions of data flow graph 450 to be transformed. Figure 4E In the example of , the transformation engine 412 has determined that a portion of the subgraph 452 can be transformed to optimize the execution of the data flow graph 450. The transformation engine 412 has determined that the data from the input node 452E is not needed in the data flow graph 450. For example, the transformation engine 412 can determine that the data from the input node 452E will not be used in the downstream operations of the data flow graph 450. Therefore, the operations involving the data can be removed from the data flow graph 450 because these operations do not need to be performed. Further, because the data from the input node 452E is not needed, the transformation engine 412 determines that the connection operation at the node 452D is no longer needed to connect the output of the node 452C with the output of the node 452F. Therefore, the transformation engine 412 removes the nodes 452E, 452F, and 452D from the subgraph 452.

[0205] Figure 4F is a diagram illustrating some embodiments of the technology described herein. Figure 4E The transformed data flow graph 460 is obtained after the transformation of Figure 4F As shown, the transformed data flow graph 460 includes a transformed subgraph 462 obtained by applying a transformation to the subgraph 452 originally incorporated into the data flow graph 450. The transformed subgraph 462 includes only the nodes 452A, 452B, 452C of the original subgraph 452. Therefore, relative to the subgraph 452, the transformed subgraph 462 includes fewer operations to be executed. Since the transformed subgraph 462 includes fewer operations than the subgraph 452, the transformed subgraph 462 can be executed more efficiently than the subgraph 452. The transformed data flow graph 460 can be executed more efficiently than the original data flow graph 450.

[0206] Figure 4G FIG. 1 is a diagram illustrating some embodiments of the technology described herein. Figure 2A FIG. 4 is a diagram of an example dataset directory entry 440 of the dataset directory 204. Figure 4G As shown, entry 440 includes information 442 for accessing a data set. In some embodiments, information 442 for accessing a data set may include a program for converting a physical data set into a logical data set. For example, the program may be a stored data flow graph that accesses and converts physical data set 202C when executed. In some embodiments, information 442 for accessing a data set may be a reference for accessing a data set. For example, information 442 may include a uniform resource locator (URL) for accessing a data set. In another example, information 442 may include a location path (e.g., a file location path) for accessing a data set.

[0207] The entry 440 further includes a logical identifier 444 for uniquely identifying the entry 440 among the other entries in the data set directory 204. For example, the logical identifier 444 may be an alphanumeric value unique to the entry 440.

[0208] like Figure 4GAs shown, in some embodiments, the data set directory entry 440 may include data storage information 446. The data storage device information 446 may include a reference to the location of the data set in the data storage device 202. For example, the reference to the location may be an identifier of the data set within the data storage device 202. The data storage information 446 may include an indication of the data storage type. For example, the type of data storage may be a SQL server database data set, an ORACLE database data set, a TERADATA database data set, a flat file, a multi-file data storage, a HADOOP database data set, a DB2 data storage, a Microsoft SQL SERVER data set, an INFORMIX data set, a table, a collection of tables, or other types of data storage. The data storage information 446 may include information about the record format or mode of the physical data set. The data storage information 446 may include a record name. For example, the data set may be a table, and the record name may be the name of the table. In another example, the data set may be a data flow graph, and the record name may be the name of a file storing the data flow graph.

[0209] like Figure 4G As shown, in some embodiments, the data set directory entry 440 may include other information 448, such as security information, access information, and / or other user parameters. For example, security information may include a password, key, or other value for accessing a data set. Other information 448 may include access information for accessing a data set from the data storage device 202. For example, the access information may include the address of a storage system storing the data set or other access information. Other information 448 may include other parameters associated with the data set, such as statistical information, data management specialists, version identifiers, and / or other parameters. In some embodiments, other information 448 may include an indication that the data set directory entry is associated with a physical data set (rather than a data flow graph data set). For example, entry 440 may include a parameter indicating that the data set is a physical data set.

[0210] In some embodiments, information 442 includes information for accessing a cataloged dataflow graph. In some embodiments, information 442 may include the location of a dataflow graph stored in data storage 202. For example, information 442 may include the location of a file storing the dataflow graph. In another example, information 442 may include a path (e.g., a URL) to the location of the dataflow graph. In some embodiments, entry 440 may include information indicating that the entry is associated with a cataloged dataflow graph (e.g., rather than a physical data set). For example, entry 440 may include a data set type field that may take a first value indicating that the entry corresponds to a physical data set and may take a second value indicating that the entry corresponds to a cataloged dataflow graph.

[0211] In some embodiments, when entry 440 corresponds to a cataloged dataflow graph, entry 440 can be used to incorporate the cataloged dataflow graph as a subgraph into another dataflow graph. For example, entry 440 can be associated with an input node of the dataflow graph (e.g., by storing the entry or its logical identifier 444 in the configuration information of the input node). When compiling, transforming and / or executing the incorporated dataflow graph, entry 440 can be used to access the cataloged dataflow graph. For example, entry 440 can be used to access the cataloged dataflow graph from the data storage device 202 and integrate it into a subgraph. The subgraph can then be connected to the dataflow graph (e.g., by connecting the output link of the subgraph to the node of the dataflow graph). The resulting dataflow graph can then be compiled into a software application, or transformed (e.g., optimized) and then compiled into a software application.

[0212] Figure 6 is a flow chart of an example process 600 for configuring a software application developed as a data flow graph to receive output data dynamically generated by the data flow graph as input according to some embodiments of the technology described herein. In some embodiments, process 600 may be implemented by reference to FIG. 2A to FIG. 4D Process 600 may be performed by data processing system 200 as described. For example, one or more computer hardware processors of data processing system 200 may be used to perform process 600.

[0213] Process 600 begins at block 602, where the system provides a UI through which a user can identify one or more entries in a data set catalog associated with one or more corresponding cataloged data flow graphs. The cataloged data flow graphs can be data flow graph data sets (e.g., subgraph data sets). The data flow graphs (e.g., subgraphs) can be as described herein with reference to Figure 7 The process 700 described is catalogued.

[0214] The UI can be configured to allow the user to identify the entry in one or more ways. In some embodiments, the UI may include a menu listing the entries in the data set directory (e.g., as described herein with reference to Figure 3F The menu may be configured to receive a selection of an entry by user input (e.g., a click, drag, tap, and / or other type of user input). In some embodiments, the UI may be part of a software application development UI. For example, the UI may be a pane adjacent to a display of a data flow diagram that a user is developing. In some embodiments, the UI may provide a search bar in which a user may enter a query to search for entries. For example, a user may enter a keyword, which the system may use to identify matching entries.

[0215] Next, process 600 proceeds to block 604, where the system receives, via the UI, an identification of a first entry associated with the first catalog graph. The system can be configured to receive user input indicating a selection of the first entry associated with the first catalog graph. For example, the UI can receive user input indicating a selection of the first entry through a menu of the UI.

[0216] In some embodiments, the system can be configured to receive identification of the first item by receiving user input indicating a command to associate the first item with a node of the data flow graph. For example, the system can receive the user input in response to a user dragging a graphical element representing the first item to a node of the data flow graph in the software application development UI. In another example, the system can receive the user input when the user places the graphical element representing the first item into the data flow graph (e.g., by dragging the graphical element to an input node of the data flow graph).

[0217] Next, process 600 proceeds to block 606, where the system configures the data flow graph of the software application to receive as input the output data generated when executing the first cataloged data flow graph associated with the first entry. In some embodiments, the system can be configured to associate the nodes of the data flow graph with the first cataloged data flow graph. For example, the system can associate the nodes of the data flow graph with the first cataloged data flow graph by including a reference (e.g., a location path) to the first cataloged data flow graph indicated by the first entry at the nodes of the data flow graph. In this example, the software application can use the reference to execute the first cataloged data flow graph and obtain the generated output data when executing. In another example, the system can associate the nodes of the data flow graph with the first cataloged data flow graph by copying the data flow graph into the node so that the data flow graph is executed at the node. In another example, the system can associate the nodes of the data flow graph with the first cataloged data flow graph by embedding a command to execute the first cataloged data flow graph at the node.

[0218] In some embodiments, the dataflow graph may obtain the output data when the dataflow graph is executed, after the dataflow graph is configured to receive as input the output data generated when the first cataloged dataflow graph is executed at block 606. For example, the first cataloged dataflow graph or a transformation thereof may be automatically executed as part of executing the dataflow graph or a transformation thereof.

[0219] Figure 7 is a flow chart of an example process 700 for providing a software application with access to output data dynamically generated by a data flow graph according to some embodiments of the techniques described herein. In some embodiments, process 700 may be implemented by reference to FIG. 2A to FIG. 4DProcess 700 may be performed by the data processing system 200 described herein. For example, process 700 may be performed using one or more computer hardware processors of data processing system 200. In some embodiments, process 700 may be performed to catalog data flow graphs so that they may be used in a software application (e.g., as described in reference to FIG. 1 ). Figure 6 as done in process 600 described herein).

[0220] Process 700 begins at block 702, where the system identifies a subgraph. In some embodiments, the system can be configured to identify a subgraph of a data flow graph by: (1) receiving user input indicating a portion of the data flow graph (e.g., in reference to Figure 3B (a) identifying a portion of the data flow graph as a subgraph. The subgraph may include one or more nodes of the data flow graph. For example, one or more nodes of the data flow graph may obtain input data and perform one or more data processing operations using the input data to generate output data. In some embodiments, the system may be configured to identify the subgraph by receiving user input indicating a previously created data flow graph stored in a data storage device as a subgraph. For example, the data flow graph may be stored in a previously generated file. The system may receive user input, which may be a selection of a file.

[0221] Next, process 700 proceeds to block 704, where the system creates a new entry in the data set directory associated with the identified subgraph. In some embodiments, the system can be configured to instantiate a new data object to be stored in the data set directory (e.g., Figure 4G The data object may include information for accessing the stored subgraph. For example, the data object may include a location path from which the subgraph may be accessed. In another example, the data object may include an index, identifier, key, or other information that may be used to access the subgraph. This information may be used to configure a software application (e.g., a data flow graph of the software application) to obtain output data generated by the subgraph (e.g., as described herein with reference to Figure 6 described above).

[0222] In some embodiments, the system can be configured to store information indicating the subgraph in a data storage device (e.g., data storage device 202). For example, the system can store a file containing information about nodes, links, and configuration parameters of the subgraph. The created entry can include a reference to the file (e.g., an index, a location path, an identifier, a key, and / or other reference). In another example, the system can generate one or more entries in a database that store information about the subgraph. The created entry can include a reference to one or more entries for configuring the software application (e.g., an index).

[0223] Next, process 700 proceeds to block 706, where the system configures the dataset directory to allow access to the new entry associated with the subgraph in the dataset directory. In some embodiments, the system can be configured to update the UI to include the new entry. For example, the system can update a list of entries in the UI to include the new entry. The device displaying the UI can include an updated UI including the new entry. In some embodiments, the system can be configured to configure the dataset directory module (e.g., reference FIG. 4B to FIG. 4C The system may update the dataset directory module 404 described above to allow access to the new entry. For example, the system may update the dataset directory module so that it can respond to requests to access the new entry.

[0224] After configuring the dataset directory in block 706, the new entry may be used by a software application to incorporate output data generated by the subgraph. For example, a dataflow graph may be configured to use output data generated by executing the subgraph as input. As another example, a software application that requires metadata about information stored in a data storage device may use information about the subgraph dataset in the new entry.

[0225] Figure 8 800 is a screen shot of a software application development UI 800 according to some embodiments of the technology described herein. UI 800 includes a space 802 in which a user can lay out a data flow diagram. UI 800 provides options for including data sets 804 in the data flow diagram, as well as options for including data processing operations (such as data transformations 806 and statistical calculations 808) in the data flow diagram.

[0226] Fig. 9 is a data flow graph 902 according to some embodiments of the techniques described herein. Figure 8 The data flow graph 902 uses the data set "customer_info" 904 as the input of the filter operation 906. For example, the data set "customer_info" 904 can be a physical data set stored by the data processing system. The output of the filter operation 906 is provided as the input of the calculation operation 908.

[0227] Fig.10 According to some embodiments of the technology described herein, Fig. 9 1 is a screenshot of UI 1000 with a portion of a dataflow graph configured as a subgraph 1002. Subgraph 1002 includes input data set "customer_info" 904 and filter operation 906 in dataflow graph 902. Subgraph 1002 can be generated by identifying a portion of dataflow graph 902 (e.g., based on user input) and generating the subgraph as the identified portion.

[0228] Fig.11 is a method for converting a Fig.10 The sub-image 1002 of is stored as a screenshot of a UI 2100 of a menu 2102 of datasets accessible through a dataset directory. Fig.11 As shown, UI 2100 generates a menu 2102 (e.g., in response to user input) that provides an option to store the subgraph as a data set. Fig.11 In the example in , this option is labeled "Create data source from subgraph".

[0229] Fig.12 1 is a screenshot of a UI 1200 for configuring details of a dataflow graph dataset according to some embodiments of the technology described herein. The UI 1200 includes a "Name" field 1202 in which a user can enter a name for the dataflow graph dataset. Fig.12 In the example of , the name of the data flow graph dataset is "dsny_customers". UI 1200 also includes a "shopping cart" field 1204 for specifying the location of the data flow graph dataset. Fig.12 In the example shown in the figure, the "Shopping Cart" field is "DATAPROJ_Data".

[0230] Fig.13 is a screenshot of a UI 1300 for cataloging a data flow graph according to some embodiments of the techniques described herein. Fig.13 As shown, in some embodiments, UI 1300 may include a display of information about the cataloged dataflow graph, such as fields to be included in output data generated by the dataflow graph.

[0231] Fig.14 is a method according to some embodiments of the technology described herein that incorporates Fig.13 Screen shot of UI 1400 for cataloging data flow diagrams 1402. Fig.14 As shown, the catalog data flow graph "dsny_customers" is associated with node 1404 of data flow graph 1402 and is used as input to node 1406 having a computation operation. UI 1400 includes a data set catalog menu 1408, which includes an entry 1408A for the "dsny_customers" catalog data flow graph. Output data generated by executing the catalog data flow graph at node 1404 is provided as input to the computation operation at node 1406.

[0232] Fig.15 is a screenshot of a UI 1500 displaying information about a cataloged data flow graph according to some embodiments of the techniques described herein. Fig.15In the example of , UI 1500 displays fields 1502 (including the name, the schema of the database storing the catalog data flow diagram, the name of the database storing the catalog data flow diagram) and information 1504 about the fields that will be included in the output data generated by the catalog data flow diagram. UI 1500 further includes an "application" field, which can indicate the application for which the catalog data flow diagram was developed. UI 1500 includes a "description" field for a textual description of the catalog data flow diagram.

[0233] Fig.16 is a screenshot of a UI 1600 showing output data 1602 generated when cataloging a dataflow graph according to some embodiments of the techniques described herein. The UI 1600 shows a preview of the output data generated when the cataloging dataflow graph is executed. Fig.16 In the example of FIG. 16 , UI 1600 includes columns for data generated when a catalog dataflow graph is executed.

[0234] Fig.17 17 is a screenshot of a UI 1700 that displays information in a data set catalog entry associated with a physical data set according to some embodiments of the techniques described herein. Fig.17 As shown, the information includes a uniform record locator (URL) for accessing the physical data set. It also includes an indication of the type of physical data set 1704. UI 1700 includes other information about the data set that can be populated.

[0235] Fig.18 18 is a screenshot of a UI 1800 that displays information in a data set catalog entry associated with a data flow graph data set according to some embodiments of the techniques described herein. Fig.18 As shown, UI 1800 includes a path 1802 of a data flow graph path. UI 1800 further includes an indication 1804 that the data set is a data flow graph data set. Fig.18 In the example of , UI 1800 indicates that the physical dataset type is "subgraph", indicating that the dataset is a data flow graph dataset.

[0236] Example Computer System

[0237] Fig.19 An example of a suitable computing system environment 1900 on which the techniques described herein may be implemented is shown. The computing system environment 1900 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the techniques described herein. Neither should the computing environment 1900 be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary operating environment 1900.

[0238] The technology described herein operates with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0239] A computing environment can execute computer executable instructions, such as program modules. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The techniques described herein can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0240] Reference Fig.19 , an exemplary system for implementing the techniques described herein includes a general purpose computing device in the form of a computer 1900. Components of the computer 1910 may include, but are not limited to, a processing unit 1920, a system memory 1930, and a system bus 1921 that couples various system components including the system memory to the processing unit 1920. The system bus 1921 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (ELISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (also known as a Mezzanine bus).

[0241] Computer 1910 typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by computer 1910, and includes volatile and non-volatile, removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computer 1910. Communication media typically embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery medium. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a way as to encode the information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0242] The system memory 1930 includes computer storage media in the form of volatile and / or nonvolatile memory, such as read-only memory (ROM) 1931 and random access memory (RAM) 1932. A basic input / output system 1933 (BIOS), containing the basic routines that help transfer information between elements within the computer 1910, such as during startup, is typically stored in ROM 1931. RAM 1932 typically contains data and / or program modules that are immediately accessible and / or currently being operated on by the processing unit 1920. By way of example and not limitation, Fig.19 Operating system 1934 , application programs 1935 , other program modules 1936 , and program data 1937 are shown.

[0243] The computer 1910 may also include other removable / non-removable, volatile / non-volatile computer storage media. For example only, Fig.19A hard disk drive 1941 that reads or writes to non-removable non-volatile magnetic media, a flash drive 1951 that reads or writes to removable non-volatile memory 1952 (such as flash memory), and an optical drive 1955 that reads or writes to a removable non-volatile optical disk 1956 (such as a CD ROM or other optical media) are shown. Other removable / non-removable, volatile / non-volatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic cassettes, flash memory cards, digital versatile disks, digital video tapes, solid-state RAM, solid-state ROM, etc. The hard disk drive 1941 is typically connected to the system bus 1921 via a non-removable memory interface such as interface 1940, and the magnetic disk drive 1951 and optical drive 1955 are typically connected to the system bus 1921 by a removable memory interface such as interface 1950.

[0244] Described above and in Fig.19 The drives and their associated computer storage media shown in FIG. 1 provide storage for computer readable instructions, data structures, program modules and other data for the computer 1910. For example, in FIG. Fig.19 In the figure, hard disk drive 1941 is shown as storing operating system 1944, application program 1945, other program module 1946 and program data 1947. Note that these components can be the same or different from operating system 1934, application program 1935, other program module 1936 and program data 1937. Different reference numerals are given to operating system 1944, application program 1945, other program module 1946 and program data 1947 here to show that at least they are different copies. An actor can enter commands and information into computer 1910 through input devices such as keyboard 1962 and pointing device 1961 (commonly called mouse, trackball or touch pad). Other input devices (not shown) may include microphones, joysticks, game controllers, satellite dishes, scanners, etc. These and other input devices are usually connected to processing unit 1920 through user input interface 1960 coupled to the system bus, but can be connected by other interfaces and bus structures such as parallel port, game port or universal serial bus (USB). A monitor 1991 or other type of display device is also connected to the system bus 1921 via an interface, such as a video interface 1990. In addition to the monitor, computers may also include other peripheral output devices such as speakers 1997 and printer 1996, which may be connected through an output peripheral interface 1995.

[0245] The computer 1910 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 1980. The remote computer 1980 can be a personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above with respect to the computer 1910, although in the example of FIG. Fig.19 Only memory storage device 1981 is shown. Fig.19 The logical connections depicted in the diagram include a local area network (LAN) 1981 and a wide area network (WAN) 1983, but other networks may also be included. Such networking environments are common in offices, enterprise-wide computer networks, intranets and the Internet.

[0246] When used in a LAN networking environment, the computer 1910 is connected to the LAN 1981 through a network interface or adapter 1980. When used in a WAN networking environment, the computer 1910 typically includes a modem 1982 or other means for establishing communications over the WAN 1983, such as the Internet. The modem 1982 may be internal or external and may be connected to the system bus 1921 via the actor input interface 1960 or other appropriate mechanism. In a networking environment, program modules depicted relative to the computer 1910, or portions thereof, may be stored in the remote memory storage device. By way of example and not limitation, Fig.19 Remote application programs 1985 are shown as residing on memory device 1981. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0247] Having thus described several aspects of at least one embodiment of the technology described herein, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0248] Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Further, although the advantages of the technology described herein are indicated, it should be understood that not every embodiment of the technology described herein will include every described advantage. Some embodiments may not implement any feature described as advantageous herein, and in some cases, one or more of the described features may be implemented to achieve further embodiments. Therefore, the foregoing description and accompanying drawings are by way of example only.

[0249] The above-mentioned embodiments of the technology described herein can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or set of processors, whether the software code is provided in a single computer or distributed among multiple computers. Such processors can be implemented as integrated circuits (wherein the integrated circuit assembly has one or more processors), including commercially available integrated circuit assemblies known in the art such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processor can be implemented with a custom circuit system (such as an ASIC) or a semi-custom circuit system generated by configuring a programmable logic device. As another alternative, the processor can be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, so that one or a subset of these cores can constitute a processor. However, the processor can be implemented using a circuit system of any suitable format.

[0250] Further, it should be understood that the computer may be embodied in any of a variety of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. In addition, the computer may be embedded in a device that is not generally considered a computer but has suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other suitable portable or fixed electronic device.

[0251] Moreover, the computer can have one or more input devices and output devices. These devices can be used in particular for presenting a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visually presenting the output, and speakers or other sound generating devices for auditorily presenting the output. Examples of input devices that can be used for a user interface include keyboards and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer can receive input information by speech recognition or in other auditory formats.

[0252] Such computers may be interconnected by one or more networks of any suitable form, including a local area network or a wide area network, such as an enterprise network or the Internet. Such a network may be based on any suitable technology and may operate according to any suitable protocol, and may include a wireless network, a wired network, or a fiber optic network.

[0253] In addition, the various methods or processes summarized herein can be encoded as software executable on one or more processors using any of a variety of operating systems or platforms. In addition, such software can be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code executed on a framework or virtual machine.

[0254] In this regard, various aspects of the technology described herein may be embodied as a computer-readable storage medium (or multiple computer-readable media) encoded with one or more programs (e.g., computer memory, one or more floppy disks, compact discs (CDs), optical disks, digital video discs (DVDs), tapes, flash memory, circuit configurations in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media), which, when executed on one or more computers or other processors, perform methods for implementing the various embodiments described above. As will be clear from the foregoing examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transient form. Such a computer-readable storage medium may be transportable so that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the technology as described above. As used herein, the term "computer-readable storage medium" covers only non-transient computer-readable media that can be considered as an article of manufacture (i.e., an article of manufacture) or a machine. Alternatively or additionally, various aspects of the technology described herein may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0255] The term "program" or "software" is used in a general sense herein to refer to any type of computer code or computer executable instruction set that can be used to program a computer or other processor to implement various aspects of the technology as described above. In addition, it should be understood that according to one aspect of this embodiment, one or more computer programs that perform the methods of the technology described herein when executed do not have to reside on a single computer or processor, but can be distributed in a modular manner among multiple different computers or processors to implement various aspects of the technology described herein.

[0256] Computer executable instructions can be in many forms, such as program modules, that are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functions of program modules can be combined or distributed as needed in different embodiments.

[0257] In addition, the data structure can be stored in a computer-readable medium in any suitable form. To simplify the description, the data structure can be shown as having fields that are related by position in the data structure. Such a relationship can also be achieved by assigning a location in a computer-readable medium that conveys the relationship between the fields for storage of the fields. However, any suitable mechanism can be used to establish the relationship between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish the relationship between data elements.

[0258] The various aspects of the technology described herein may be used alone, in combination, or in a variety of arrangements not specifically described in the embodiments described hereinabove, and are therefore not limited in their application to the details and arrangements of the components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0259] In addition, the technology described in this article can be embodied as a method, and this article provides an example of the method, including reference to FIG. 3 and Figure 7 The acts performed as part of any of these methods may be ordered in any suitable manner. Thus, embodiments may be constructed in which the acts are performed in an order different from that shown, which may include performing some acts simultaneously even though the acts are shown as sequential acts in illustrative embodiments.

[0260] Further, some actions are described as being performed by an “actor” or “user.” It should be understood that an “actor” or “user” need not be a single individual, and that in some embodiments, actions attributable to an “actor” or “user” may be performed by a team of individuals and / or a combination of individuals and computer-assisted tools or other mechanisms.

[0261] The use of ordinal terms such as "first", "second", "third", etc. to modify claim elements in the claims does not itself imply any priority, superiority or order of one claim element over another claim element, or the temporal order of performance of the actions of the method, but is merely used as a label to distinguish one claim element with a specific name from another element with the same name (but using ordinal terms) to distinguish the claim elements.

[0262] Furthermore, the words and terms used in this article are for descriptive purposes and should not be regarded as limiting. The words "including", "comprising", or "having", "containing", "involving" and their variations used in this article are intended to cover the multiple items listed thereafter and their equivalents, as well as additional items.

Claims

1. A method performed by a data processing system for providing a software application developed as a data flow graph with access to output data dynamically generated by one or more other data flow graphs, thereby enabling efficient development of software applications in a dynamic environment having multiple data sets, by using entries in a data set catalog associated with respective software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows, the method comprising: Use at least one computer hardware processor to do the following: providing a user interface by which a user can identify, in a data set catalog, one or more entries associated with one or more corresponding cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged data flow graph has one or more nodes representing one or more corresponding data sources and one or more nodes representing one or more corresponding data processing operations, wherein, when executing the first cataloged dataflow graph, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more corresponding data sources; receiving, via the user interface, identification of the first entry associated with the first cataloged dataflow graph; and The dataflow graph of the software application is configured to receive as input the output data generated when executing the first cataloged dataflow graph, the configuring comprising associating one of the input nodes in the dataflow graph with the first cataloged dataflow graph.

2. The method of claim 1, wherein: Receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving, via the user interface, a selection of the first entry.

3. The method of claim 1 or any other preceding claim, wherein: Providing the user interface includes generating a graphical user interface having a searchable menu of the one or more entries in the data set catalog; as well as Receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving, via the user interface, user input indicating a selection of the first entry in the searchable menu.

4. The method of claim 1 or any other preceding claim, further comprising: A configured data flow diagram for executing the software application.

5. A method as claimed in claim 4 or any other preceding claim, wherein: The configured data flow diagram for executing the software application includes: executing the first cataloged dataflow graph to generate the output data; and The generated output data is provided as input to a data flow graph of the software application to perform at least one of the one or more data processing operations using the output data.

6. The method of claim 4 or any other preceding claim, wherein: Execution of the configured dataflow graph results in execution of the first cataloged dataflow graph.

7. A method as claimed in claim 6 or any other preceding claim, wherein: The output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

8. The method of claim 1 or any other preceding claim, wherein: The data set directory includes a plurality of entries associated with corresponding cataloged data flow graphs and a plurality of entries associated with corresponding data sets previously stored in memory.

9. The method of claim 1 or any other preceding claim, wherein: The user interface allows the user to identify, in the data set catalog, at least one entry associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory.

10. The method of claim 9 or any other preceding claim, wherein: The one or more input nodes include a plurality of input nodes, the method further comprising: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and The dataflow graph of the software application is configured to receive data from the physical data set as input, the configuring comprising associating another one of the plurality of input nodes in the dataflow graph with the data from the physical data set.

11. The method of claim 1 or any other preceding claim, further comprising: transforming the dataflow graph including the input node associated with the first cataloged dataflow graph to obtain a transformed dataflow graph; compiling the transformed data flow graph into a software application; and Execute the software application.

12. A method as claimed in claim 11 or any other preceding claim, wherein: Transforming the data flow graph including the input node associated with the first cataloged data flow graph to obtain the transformed data flow graph includes: Incorporating the first cataloged dataflow graph into the dataflow graph as a first subgraph at the input node associated with the first cataloged dataflow graph; and The first sub-image is transformed to obtain a second sub-image different from the first sub-image.

13. The method of claim 12 or any other preceding claim, wherein: Transforming the first sub-image to obtain the second sub-image includes: The first subgraph is transformed based at least in part on at least one operation represented by at least one node downstream of the input node in the dataflow graph.

14. The method of claim 12 or any other preceding claim, wherein: Transforming the first sub-image to obtain the second sub-image includes: At least one optimization is applied to the first subgraph to obtain the second subgraph.

15. The method of claim 14 or any other preceding claim, wherein: The at least one optimization includes at least one of the following: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; Changing the order of multiple nodes of the first subgraph; combining multiple nodes of the first subgraph; processing in parallel at least one operation represented by at least one node of the first subgraph; or Data in at least one node in the first subgraph is deleted so that the data is not used in a subsequent operation represented by a downstream node of the at least one node in the first subgraph.

16. The method of claim 11 or any other preceding claim, wherein: The transformation includes: identifying at least a portion of the dataflow graph to be transformed, the at least a portion comprising the first cataloged dataflow graph associated with the input node; and The at least a portion of the data flow graph is transformed to obtain the transformed data flow graph.

17. The method of claim 1 or any other preceding claim, wherein: The first cataloged dataflow graph is generated from a subgraph embedded in another dataflow graph having nodes representing data processing operations and links representing data flow between the nodes, wherein the other dataflow graph is separate from the dataflow graph of the software application.

18. The method of claim 17 or any other preceding claim, further comprising: displaying a graphical representation of the other data flow graph in the UI; as well as receiving, through the UI, user input indicating that a subgraph within the data flow graph is to be cataloged; as well as In response to receiving the user input indicating that a subgraph within the dataflow graph is to be cataloged, the subgraph is stored as the first cataloged dataflow graph.

19. The method of claim 1 or any other preceding claim, wherein: The first cataloged dataflow graph has only a single output link representing data output by the first cataloged dataflow graph by applying the one or more data processing operations to data obtained from the one or more data sources.

20. The method of claim 1 or any other preceding claim, wherein: The first cataloged dataflow graph is stored in a data storage device of the data processing system, and the first entry stores a reference to a location of the first cataloged dataflow graph in the data storage device.

21. The method of claim 1 or any other preceding claim, wherein: The first entry stores a reference to a file that stores information indicating nodes of the first cataloged data flow graph and / or configuration parameters of the first cataloged data flow graph.

22. The method of claim 1 or any other preceding claim, wherein: Configuring the dataflow graph of the software application to receive as input the output data generated when executing the first cataloged dataflow graph includes: receiving, via the user interface, an association of the first entry with the input node in the dataflow graph; and In response to receiving the user input associating the first entry with the input node in the dataflow graph: The dataflow graph is configured to receive at the input node data output through an output link of the first cataloged dataflow graph as a result of executing the first cataloged dataflow graph.

23. The method of claim 1 or any other preceding claim, wherein: Receiving an association of the first entry with the input node in the data flow graph includes: User input is received via the user interface, the user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node in the dataflow graph.

24. The method of claim 23 or any other preceding claim, wherein: The user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node includes dragging the first graphical element to the second graphical element in the user interface.

25. A data processing system for enabling efficient development of software applications in a dynamic environment having multiple data sets by providing software applications developed as data flow graphs with access to output data dynamically generated by one or more other data flow graphs using entries in a data set catalog associated with respective software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows, the system comprising: at least one computer hardware processor; as well as At least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to: providing a user interface by which a user can identify, in a data set catalog, one or more entries associated with one or more corresponding cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged data flow graph has one or more nodes representing one or more corresponding data sources and one or more nodes representing one or more corresponding data processing operations, wherein, when executing the first cataloged dataflow graph, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, identification of the first entry associated with the first cataloged dataflow graph; and The dataflow graph of the software application is configured to receive as input the output data generated when executing the first cataloged dataflow graph, the configuring comprising associating one of the input nodes in the dataflow graph with the first cataloged dataflow graph.

26. The system of claim 25, wherein: Receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving, via the user interface, a selection of the first entry.

27. The system of any one of claims 25 to 26, wherein: Providing the user interface includes generating a graphical user interface having a searchable menu of the one or more entries in the data set catalog; and Receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving, via the user interface, user input indicating a selection of the first entry in the searchable menu.

28. A system as claimed in any one of claims 25 to 27, wherein: The instructions further cause the at least one computer hardware processor to execute a data flow graph for executing the software application.

29. A system as claimed in claim 28 or any one of claims 25 to 27, wherein: The data flow diagram for executing the software application includes: executing the first cataloged dataflow graph to generate the output data; and The generated output data is provided as input to a data flow graph of the software application to perform at least one of the one or more data processing operations using the output data.

30. A system as claimed in claim 28 or any one of claims 25 to 27 and 29, wherein Execution of the configured dataflow graph results in execution of the first cataloged dataflow graph.

31. A system as claimed in claim 30 or any one of claims 25 to 29, wherein: The output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

32. The system of any one of claims 25 to 31, wherein: The data set directory includes a plurality of entries associated with corresponding cataloged data flow graphs and a plurality of entries associated with corresponding physical data sets previously stored in memory.

33. A system as claimed in any one of claims 25 to 32, wherein: The user interface allows the user to identify, in the data set catalog, at least one entry associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory.

34. A system as claimed in claim 33 or any one of claims 25 to 32, wherein: The one or more input nodes include a plurality of input nodes, and the instructions further cause the at least one computer hardware processor to: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and The dataflow graph of the software application is configured to receive data from the physical data set as input, the configuring comprising associating another one of the plurality of input nodes in the dataflow graph with the data from the physical data set.

35. The system of any one of claims 25 to 34, wherein: The data set directory includes a plurality of entries associated with corresponding cataloged data flow graphs and a plurality of entries associated with corresponding physical data sets previously stored in memory.

36. The system of any one of claims 25 to 35, wherein: The user interface allows the user to identify, in the data set catalog, at least one entry associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory.

37. A system as claimed in claim 36 or any one of claims 25 to 35, wherein: The one or more input nodes include a plurality of input nodes, and the instructions further cause the at least one computer hardware processor to: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and The dataflow graph of the software application is configured to receive data from the physical data set as input, the configuring comprising associating another one of the plurality of input nodes in the dataflow graph with the data from the physical data set.

38. The system of any one of claims 25 to 37, wherein: The instructions further cause the at least one computer hardware processor to: transforming the dataflow graph including the input node associated with the first cataloged dataflow graph to obtain a transformed dataflow graph; compiling the transformed data flow graph into a software application; and Execute the software application.

39. The system of claim 38, wherein: Transforming the data flow graph including the input node associated with the first cataloged data flow graph to obtain the transformed data flow graph includes: Incorporating the first cataloged dataflow graph into the dataflow graph as a first subgraph at the input node associated with the first cataloged dataflow graph; and The first sub-image is transformed to obtain a second sub-image different from the first sub-image.

40. The system of claim 39, wherein: Transforming the first sub-image to obtain the second sub-image includes: The first subgraph is transformed based at least in part on at least one operation represented by at least one node downstream of the input node in the dataflow graph.

41. The system of claim 39, wherein: Transforming the first sub-image to obtain the second sub-image includes: At least one optimization is applied to the first subgraph.

42. The method of claim 41, wherein: The at least one optimization includes at least one of the following: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; Changing the order of multiple nodes of the first subgraph; combining multiple nodes of the first subgraph; processing in parallel at least one operation represented by at least one node of the first subgraph; or Data in at least one node in the first subgraph is deleted so that the data is not used in a subsequent operation represented by a downstream node of the at least one node in the first subgraph.

43. The system of claim 38 or any one of claims 25 to 37, wherein: The transformation includes: identifying at least a portion of the dataflow graph to be transformed, the at least a portion comprising the first cataloged dataflow graph associated with the input node; and The at least a portion of the data flow graph is transformed to obtain the transformed data flow graph.

44. A system as claimed in any one of claims 25 to 43, wherein: The first cataloged dataflow graph is generated from a subgraph embedded in another dataflow graph having nodes representing data processing operations and links representing data flow between the nodes, wherein the other dataflow graph is separate from the dataflow graph of the software application.

45. The system of claim 44 or any one of claims 25 to 43, wherein: The instructions further cause the at least one computer hardware processor to: displaying a graphical representation of the other data flow graph in the UI; and User input indicating a subgraph within the dataflow graph is received through the UI.

46. ​​The system of any one of claims 25 to 45, wherein: The first cataloged dataflow graph has a single output link representing data output by the first cataloged dataflow graph by applying the one or more data processing operations to data obtained from the one or more data sources.

47. The system of any one of claims 25 to 46, wherein: The first entry stores a reference to a file that stores information indicating nodes of the first cataloged data flow graph and / or configuration parameters of the first cataloged data flow graph.

48. The system of any one of claims 25 to 47, wherein: Configuring the dataflow graph of the software application to receive as input the output data generated when executing the first cataloged dataflow graph includes: receiving, via the user interface, an association of the first entry with the input node in the dataflow graph; and In response to receiving the user input associating the first entry with the input node in the dataflow graph: The dataflow graph is configured to receive at the input node data output through an output link of the first cataloged dataflow graph as a result of executing the first cataloged dataflow graph.

49. The system of any one of claims 25 to 48, wherein: Receiving an association of the first entry with the input node in the data flow graph includes: User input is received via the user interface, the user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node in the dataflow graph.

50. The system of claim 45 or any one of claims 25 to 49, wherein: The user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the input node includes dragging the first graphical element to the second graphical element in the user interface.

51. At least one non-transitory computer readable storage medium storing instructions that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing a software application developed as a data flow graph with access to output data dynamically generated by one or more other data flow graphs by using entries in a data set directory associated with respective software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows, thereby enabling efficient development of software applications in a dynamic environment having multiple data sets, the method comprising: providing a user interface by which a user can identify, in a data set catalog, one or more entries associated with one or more corresponding cataloged dataflow graphs, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged data flow graph has one or more nodes representing one or more corresponding data sources and one or more nodes representing one or more corresponding data processing operations, wherein, when executing the first cataloged dataflow graph, the first cataloged dataflow graph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, identification of the first entry associated with the first cataloged dataflow graph; and The dataflow graph of the software application is configured to receive as input the output data generated when executing the first cataloged dataflow graph, the configuring comprising associating one of the input nodes in the dataflow graph with the first cataloged dataflow graph.

52. A method performed by a data processing system for providing software applications with access to output data dynamically generated by a data flow graph, thereby enabling efficient development of software applications in a dynamic environment having multiple data sets by generating entries in a data set catalog, the entries being associated with respective software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows, the method comprising: Use at least one computer hardware processor to do the following: Identify subgraphs, wherein the subgraph has one or more nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and an output link representing data output by the subgraph, wherein, when the subgraph is executed, the subgraph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry in the data set directory associated with the identified subgraph, the new entry being associated with information indicating nodes, links, and configuration parameters of the identified subgraph; and The dataset directory is configured to allow access to a new entry in the dataset directory associated with the identified subgraph to incorporate the subgraph into the software application.

53. The method of claim 52, wherein: Identifying the subgraph includes: displaying a graphical representation of the data flow graph in a user interface; and A first user input is received via the user interface indicating a subgraph within the dataflow graph.

54. The method of any one of claims 52 to 53, further comprising: receiving, via the user interface, a second user input commanding creation of the new entry associated with the indicated subgraph; Wherein creating the new entry associated with the identified subgraph is performed in response to receiving the second user input.

55. The method of any one of claims 52 to 54, wherein: The data processing system includes a data storage device storing a previously created data flow graph, and identifying the subgraph includes: User input is received via a user interface identifying the previously created dataflow graph as the subgraph.

56. The method of any one of claims 52 to 55, wherein: The new entry includes information indicating the nodes, links, and configuration parameters of the identified subgraph.

57. The method of any one of claims 52 to 56, wherein: Configuring the dataset catalog to allow access to the new entry associated with the identified subgraph includes providing a user interface through which a user can identify the new entry associated with the identified subgraph in the dataset catalog.

58. The method of any one of claims 52 to 56, wherein: The one or more corresponding data sources include a physical data set previously stored in memory, and the data set directory includes an entry associated with the physical data set.

59. The method of any one of claims 52 to 58, wherein: The data processing system includes a data storage device, and the method further comprises: storing the subgraph in a data storage device of the data processing system; and Therein, creating the new entry includes storing in the new entry a reference to a location of the stored subgraph in a data storage device of the data processing system.

60. A data processing system for enabling efficient development of software applications in a dynamic environment having multiple data sets by providing software applications with access to output data dynamically generated by data flow graphs by generating entries in a data set catalog, the entries being associated with respective software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows, the system comprising: at least one computer hardware processor; as well as At least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to: Identify subgraphs of a data flow graph, wherein the subgraph has one or more nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and an output link representing data output by the subgraph, wherein, when the subgraph is executed, the subgraph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry in the data set directory associated with the identified subgraph, the new entry being associated with information indicating nodes, links, and configuration parameters of the identified subgraph; and The dataset directory is configured to allow access to a new entry in the dataset directory associated with the identified subgraph.

61. The system of claim 60, wherein: Identifying the subgraph includes: displaying a graphical representation of the data flow graph in a user interface; and A first user input is received via the user interface indicating a subgraph within the dataflow graph.

62. The system of any one of claims 60 to 61, wherein: The instructions further cause the at least one computer hardware processor to: receiving, via the user interface, a second user input commanding creation of the new entry associated with the indicated subgraph; Wherein creating the new entry associated with the identified subgraph is performed in response to receiving the second user input.

63. The system of any one of claims 60 to 62, wherein: The data processing system includes a data storage device storing a previously created data flow graph, and identifying the subgraph includes: User input is received via a user interface identifying the previously created dataflow graph as the subgraph.

64. The system of any one of claims 60 to 63, wherein: The new entry includes information indicating the nodes, links, and configuration parameters of the identified subgraph.

65. The system of any one of claims 60 to 64, wherein: Configuring the dataset catalog to allow access to the new entry associated with the identified subgraph includes providing a user interface through which a user can identify the new entry associated with the identified subgraph in the dataset catalog.

66. The system of any one of claims 60 to 65, wherein: The one or more corresponding data sources include a physical data set previously stored in memory, and the data set directory includes an entry associated with the physical data set.

67. The system of any one of claims 60 to 66, wherein: The data processing system includes a data storage device, and the instructions further cause the at least one computer hardware processor to perform the following operations: storing the subgraph in a data storage device of the data processing system; and Therein, creating the new entry includes storing in the new entry a reference to a location of the stored subgraph in a data storage device of the data processing system.

68. At least one non-transitory computer readable storage medium storing instructions that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing software applications with access to output data dynamically generated by a data flow graph by generating entries in a data set directory, thereby enabling efficient development of software applications in a dynamic environment having multiple data sets, the entries being associated with respective software applications developed as data flow graphs having nodes representing data processing operations and links representing data flows, the method comprising: Identify subgraphs of a data flow graph, wherein the subgraph has one or more nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and an output link representing data output by the subgraph, wherein, when the subgraph is executed, the subgraph generates output data by applying the one or more data processing operations to data obtained from the one or more data sources; creating a new entry in the data set directory associated with the identified subgraph, the new entry being associated with information indicating nodes, links, and configuration parameters of the identified subgraph; and The dataset directory is configured to allow access to a new entry in the dataset directory associated with the identified subgraph.

69. The at least one non-transitory computer-readable storage medium of claim 68, wherein: Identifying the subgraph includes: displaying a graphical representation of the data flow graph in a user interface; and A first user input is received via the user interface indicating a subgraph within the dataflow graph.

70. The at least one non-transitory computer-readable storage medium of claim 68 or 69, wherein: The method further comprises: receiving, via the user interface, a second user input commanding creation of the new entry associated with the indicated subgraph; Wherein creating the new entry associated with the identified subgraph is performed in response to receiving the second user input.

71. A method performed by a data processing system for providing a software application developed as a data flow graph having nodes representing data processing operations and links representing data flow between the nodes with access to output data dynamically generated by one or more other data flow graphs, the method comprising: Use at least one computer hardware processor to do the following: providing a user interface configured to receive from a user an identification of one or more entries in a data set catalog associated with one or more corresponding cataloged dataflow graphs having nodes representing data processing operations and links representing data flows between the nodes, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged data flow graph has one or more input nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and output links representing data output from the first cataloged data flow graph, wherein, when executing the first cataloged dataflow graph, the first cataloged dataflow graph generates output data on the output link by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, identification of the first entry associated with the first cataloged dataflow graph; and A dataflow graph of the software application is configured using the first entry to receive as input output data generated when executing the first cataloged dataflow graph, the configuration comprising associating a first input node in the dataflow graph of the software application with the first cataloged dataflow graph.

72. The method of claim 71, wherein: Receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving, via the user interface, a selection of the first entry.

73. The method of claim 71 or 72, wherein: Providing the user interface includes generating a graphical user interface having a searchable menu of the one or more entries in the data set catalog; and Receiving identification of the first entry associated with the first cataloged dataflow graph includes receiving, via the user interface, user input indicating a selection of the first entry in the searchable menu.

74. The method of claim 71 or any one of claims 71 to 73, further comprising executing a configured data flow graph of the software application.

75. The method of claim 74 or any one of claims 71 to 73, wherein: The configured data flow diagram for executing the software application includes: executing the first cataloged dataflow graph to generate the output data; and The generated output data is provided as input to a data flow graph of the software application to perform at least one of the one or more data processing operations using the output data.

76. The method of claim 75 or any one of claims 71 to 74, wherein: Execution of the configured dataflow graph results in execution of the first cataloged dataflow graph.

77. The method of claim 76 or any one of claims 71 to 75, wherein: The output data is generated by the first cataloged dataflow graph during execution of the configured dataflow graph.

78. The method of any one of claims 71 to 77, wherein: The configured data flow diagram for executing the software application includes: compiling the configured data flow graph of the software application to obtain a compiled software application; and The compiled software application is executed.

79. The method of any one of claims 71 to 78, further comprising maintaining the data set catalog, wherein: The data set directory includes a plurality of entries associated with corresponding cataloged data flow graphs and preferably a plurality of entries associated with corresponding physical data sets previously stored in memory.

80. The method of any one of claims 71 to 79, wherein: The user interface allows the user to identify, in the data set catalog, at least one entry associated with at least one corresponding cataloged physical data set previously stored in the memory, the at least one entry comprising a second entry associated with the physical data set stored in the memory.

81. The method of claim 80 or any one of claims 71 to 79, wherein: The data flow graph includes a plurality of input nodes, including the first input node, and the method further includes: receiving, via the user interface, an identification of the second entry associated with the physical data set stored in the memory; and A dataflow graph of the software application is configured to receive data from the physical data set as input, the configuring comprising associating a second input node of the plurality of input nodes in the dataflow graph with the data from the physical data set.

82. The method of any one of claims 71 to 81, further comprising: A dataflow graph of the software application including a first input node associated with the first cataloged dataflow graph is transformed to obtain a transformed dataflow graph.

83. The method of any one of claims 82 or 71 to 81, wherein The transformation includes: merging the first cataloged dataflow graph into the dataflow graph at the first input node associated with the first cataloged dataflow graph; identifying at least a portion of the dataflow graph to be transformed, the at least a portion comprising the first cataloged dataflow graph incorporated at the first input node; and The first cataloged dataflow graph incorporated at the first input node is transformed to obtain the transformed dataflow graph.

84. The method of claim 82 or any one of claims 71 to 81, wherein: The transformation includes: Incorporating the first cataloged dataflow graph into the dataflow graph as a first subgraph at the first input node associated with the first cataloged dataflow graph; and The first sub-image is transformed to obtain a second sub-image different from the first sub-image.

85. The method of claim 84 or any one of claims 71 to 83, wherein: Transforming the first sub-image to obtain the second sub-image includes: The first subgraph is transformed based at least in part on at least one operation represented by at least one node downstream of the first input node in the dataflow graph.

86. The method of claim 84 or any one of claims 71 to 83, wherein Transforming the first sub-image to obtain the second sub-image includes: At least one optimization is applied to the first subgraph to obtain the second subgraph.

87. The method of claim 86 or any one of claims 71 to 85, wherein The at least one optimization includes at least one of the following: removing at least one node of the first subgraph; replacing at least one node of the first subgraph; Changing the order of multiple nodes of the first subgraph; combining multiple nodes of the first subgraph; processing in parallel at least one operation represented by at least one node of the first subgraph; or Data in at least one node in the first subgraph is deleted so that the data is not used in a subsequent operation represented by a downstream node of the at least one node in the first subgraph.

88. The method of claim 82 or any one of claims 71 to 81, further comprising: compiling the transformed data flow graph into a software application; as well as Execute the software application.

89. The method of any one of claims 71 to 88, wherein: The first cataloged dataflow graph is generated from a subgraph embedded in another dataflow graph having nodes representing data processing operations and links representing data flow between the nodes, wherein the other dataflow graph is separate from the dataflow graph of the software application.

90. The method of claim 89 or any one of claims 71 to 88, further comprising: displaying a graphical representation of the other data flow graph in the UI; as well as User input indicating a subgraph within the dataflow graph is received through the UI.

91. The method of any one of claims 71 to 90, wherein: The output link is the only output link of the first cataloged data flow graph.

92. The method of any one of claims 71 to 91, wherein: The first cataloged dataflow graph is stored in a data storage device of the data processing system, and the first entry stores a reference to a location of the first cataloged dataflow graph in the data storage device.

93. The method of any one of claims 71 to 92, wherein: The first entry stores a reference to a file that stores information identifying nodes of the first cataloged dataflow graph and / or operations at the nodes of the first cataloged dataflow graph.

94. The method of any one of claims 71 to 93, wherein: Configuring the dataflow graph of the software application to receive as input the output data generated when executing the first cataloged dataflow graph includes: receiving, via the user interface, an association of the first entry with the first input node in a data flow graph of the software application; and In response to receiving the user input associating the first entry with the first input node in the dataflow graph of the software application: The dataflow graph of the software application is configured to receive at the first input node data output through an output link of the first cataloged dataflow graph as a result of executing the first cataloged dataflow graph.

95. The method of any one of claims 71 to 94, wherein: Receiving an association of the first entry with the first input node in the data flow graph of the software application comprises: User input is received via the user interface, the user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the first input node in a dataflow graph of the software application.

96. The method of claim 95 or any one of claims 71 to 94, wherein: The user input indicating an association of a first graphical element representing the first entry with a second graphical element representing the first input node includes dragging the first graphical element to the second graphical element in the user interface.

97. A data processing system for providing a software application developed as a data flow graph having nodes representing data processing operations and links representing data flow between the nodes with access to output data generated by one or more other data flow graphs, the data processing system comprising: at least one computer hardware processor; as well as At least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to: providing a user interface configured to receive from a user an identification of one or more entries in a data set catalog associated with one or more corresponding cataloged dataflow graphs having nodes representing data processing operations and links representing data flows between the nodes, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged data flow graph has one or more input nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and output links representing data output from the first cataloged data flow graph, wherein, when executing the first cataloged dataflow graph, the first cataloged dataflow graph generates output data on the output link by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, identification of the first entry associated with the first cataloged dataflow graph; and In response to receiving identification of the first entry associated with the first cataloged dataflow graph: The configuring includes associating a first input node in the dataflow graph of the software application with the first cataloged dataflow graph by using the first entry to receive as input the output data dynamically generated when executing the first cataloged dataflow graph.

98. At least one non-transitory computer readable storage medium storing instructions that, when executed by at least one computer hardware processor of a data processing system, cause the at least one computer hardware processor to perform a method for providing a software application developed as a data flow graph having nodes representing data processing operations and links representing data flow between the nodes with access to output data dynamically generated by one or more other data flow graphs, the method comprising: providing a user interface configured to receive from a user an identification of one or more entries in a data set catalog associated with one or more corresponding cataloged dataflow graphs having nodes representing data processing operations and links representing data flows between the nodes, the one or more entries including a first entry associated with a first cataloged dataflow graph, wherein the first cataloged data flow graph has one or more input nodes representing one or more corresponding data sources, one or more nodes representing one or more corresponding data processing operations, and output links representing data output from the first cataloged data flow graph, wherein, when executing the first cataloged dataflow graph, the first cataloged dataflow graph generates output data on the output link by applying the one or more data processing operations to data obtained from the one or more data sources; receiving, via the user interface, identification of the first entry associated with the first cataloged dataflow graph; and A dataflow graph of the software application is configured using the first entry to receive as input output data generated when executing the first cataloged dataflow graph, the configuration comprising associating a first input node in the dataflow graph of the software application with the first cataloged dataflow graph.

Images