An editor for generating computational graphs

Abstract

A technique for generating a dataflow graph includes generating a first dataflow graph including a plurality of first nodes representing first computer operations in data processing, at least one of the first computer operations being a declarative operation that specifies one or more properties of one or more results of the data processing; and transforming the first dataflow graph into a second dataflow graph for processing data in accordance with the first computer operations, the second dataflow graph including a plurality of second nodes representing the second computer operations, at least one of the second nodes representing one or more instruction operations that implement logic specified by the declarative operations, the one or more instruction operations not represented by the first nodes in the first dataflow graph.

Description

[Technical field]

[0001] (Priority Claim) This application claims priority to U.S. Provisional Patent Application No. 62 / 966,768, filed January 28, 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates to generating computation graphs.

[0003] Complex computations can often be represented as data flows using directed graphs, with components of the computation associated with data flows between components that correspond to the vertices of the graph and the links (arcs, edges) of the graph. A system for implementing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," the entire contents of which are incorporated herein by reference. In some cases, the computations associated with the vertices are written in a human-readable form and are referred to as "business rules."

[0004] One technique for generating data flow graphs uses a business rule editor, an example of which is disclosed in U.S. Patent No. 8,069,129, entitled "Editing and Compiling Business Rules," the entire contents of which are incorporated herein by reference. Summary of the Invention [Means for solving the problem]

[0005] In general aspect 1, a method for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations, the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, at least some of the second computer operations not represented by a first node in the first dataflow graph, the method comprising: generating a first dataflow graph including a plurality of first nodes representing the first computer operations in a data processing, at least some of the first computer operations not represented by a first node in the first dataflow graph; the other is a declarative operation that specifies one or more properties of one or more results of the data processing; transforming the first dataflow graph into a second dataflow graph for processing data in accordance with the first computer operation, the second dataflow graph including a plurality of second nodes representing the second computer operations, at least one of the second nodes representing one or more instruction operations that implement logic specified by the declarative operations, the one or more instruction operations not represented by a first node in the first dataflow graph; and storing the second dataflow graph in a data store.

[0006] In aspect 2 described in aspect 1, the first data flow graph to the second data flow graph includes creating an instruction operation and creating a given second node representing the instruction operation, where the given second node is not represented in the first data flow graph.

[0007] In aspect 3 of either aspect 1 or 2, one of the second operations represented in the second data flow graph and not represented in the first data flow graph is selected from the group consisting of a reordering operation, a data type operation, a join with a specified key operation, and a split operation.

[0008] In aspect 4, described in any of aspects 1 to 3, one or more of the second operations are at least (i) necessary to process data in accordance with one or more of the first operations specified in the first data flow graph, or (ii) improve data processing in accordance with one or more of the first operations specified in the first data flow graph compared to data processing without the one or more additional operations.

[0009] In aspect 5 of any of aspects 1 to 4, the method further includes converting the second data flow graph into an optimized data flow graph by applying one or more data flow graph optimization rules to the second data flow graph to improve the computational efficiency of the second data flow graph compared to the computational efficiency of the second data flow graph before the application of the rules.

[0010] In aspect 6 of any of aspects 1 to 5, the one or more data flow graph optimization rules include at least one of removing redundant nodes from the second data flow graph, removing dead nodes from the second data flow graph, changing the order of nodes in the second data flow graph, reducing the strength of a node in the second data flow graph, merging two or more nodes in the second data flow graph, converting a node in the second data flow graph from a serial operation to a parallel operation, or inserting a split operation in the second data flow graph.

[0011] In aspect 7 according to any one of aspects 1 to 6, at least one of the second operations includes an automatic parallel operation or an automatic division operation.

[0012] In aspect 8 according to any of aspects 1 to 7, at least one of the second operations includes a rearrangement operation.

[0013] In an embodiment 9 according to any of the embodiments 1 to 8, at least one of the second operations includes an operation of specifying metadata between one or more of the second nodes.

[0014] In aspect 10, described in any of aspects 1 to 9, the method further includes providing data to generate a graphical editor interface including a canvas portion and a catalog portion, wherein the catalog portion includes one or more selectable icons for visually depicting logic of the calculation in the canvas portion; receiving icon selection data representing the logic of the calculation depicted in the canvas portion, wherein the icon selection data specifies at least one of the one or more selectable icons selected from the catalog portion and included in the canvas portion; and generating a first data flow graph from the received icon selection data, the first data flow graph including a plurality of first nodes representing the logic specified in the canvas portion, wherein at least one of the first nodes represents at least one of the one or more selectable icons selected from the catalog portion.

[0015] In an eleventh aspect according to any one of the first to tenth aspects, each selected icon indicates an instruction to access data from a data catalog that preformats the data or specifies a format of the data to be accessed via the data catalog.

[0016] In aspect 12 according to any one of aspects 1 to 11, the first data flow graph is a user-defined data flow graph.

[0017] In aspect 13, described in any of aspects 1 to 12, the method further includes providing data to generate a graphical editor interface including a canvas portion and a catalog portion, the catalog portion including a plurality of dataset selection icons and a plurality of transformation selection icons; generating an initial node in a first data flow graph according to elements stored in the storage unit represented by the selected dataset selection icons and the selected transformation selection icons; labeling the initial node to provide a labeled node; and rendering a visual representation of the labeled node on the canvas portion.

[0018] In aspect 14 according to any one of aspects 1 to 13, the initial node has an action placeholder field for holding an action, and a data placeholder field for holding a source or sink of data.

[0019] In aspect 15, described in any of aspects 1 to 14, the modifying further includes retrieving from the storage system an element of the action held in the action placeholder field, and retrieving from the storage system an element of a data source or data sink held in the data placeholder field, and adding a link to the data placeholder field pointing to the source or sink of the data.

[0020] In aspect 16 of any of aspects 1 to 15, the method further includes providing data to render the first data flow graph on a canvas portion of the graphical editor interface.

[0021] In aspect 17 of any of aspects 1 to 16, once all of the generated initial nodes have been labeled, the method further includes compiling all the labeled nodes of the first data flow graph into a second data flow graph, which is a computational data flow graph.

[0022] In aspect 18 of any of aspects 1 to 17, after labeling all of the modified initial nodes, the method further includes optimizing all labeled nodes of the first data flow graph, and optimizing the labeled nodes of the first data flow graph further includes optimizing elements stored in at least one of the labeled nodes.

[0023] In aspect 19 of any of aspects 1 to 18, the method further includes accessing a prototype node and applying an algorithm to copy parameters from the accessed prototype node to modify at least one of the initial nodes.

[0024] In aspect 20 according to any of aspects 1 to 19, at least one parameter of the initial node is a configuration parameter that is not overwritten by a prototype node.

[0025] In aspect 21 according to any of aspects 1 to 20, the prototype node declares at least one of an initial node, a port on the initial node, or a parameter of a component presented on a canvas of an editor interface.

[0026] In aspect 22 according to any of aspects 1 to 21, applying the prototype replaces the existing parameter's descriptor with the descriptor from the prototype, but does not replace the parameter's existing value.

[0027] In aspect 23 according to any of aspects 1 to 22, the method further comprises applying a transformation to compute values ​​described in the metadata and the first dataflow graph.

[0028] In aspect 24 according to any one of aspects 1 to 23, the label refers to one or more of the key, the value, the name, and the source.

[0029] In aspect 25, described in any of aspects 1 to 24, at least some of the multiple initial nodes that store one or more elements stored in the storage unit represented by the selected dataset selection icon and the storage unit represented by the selected transformation selection icon specify, at least in part, corresponding storage unit functions for at least some of the multiple initial nodes.

[0030] In general aspect 26, a system for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations, the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, at least some of the second computer operations not represented by a first node in the first dataflow graph, the system comprising one or more processors and one or more storage devices that store instructions, the instructions, when executed by the one or more processors, generating a first dataflow graph including a plurality of first nodes representing the first computer operations in data processing, the first processors generating a first dataflow graph including a plurality of first nodes representing the first computer operations in data processing, the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, at least some of the second computer operations not represented by a first node in the first dataflow graph, the system comprising one or more processors and one or more storage devices that store instructions, the instructions, when executed by the one or more processors, one or more storage devices that cause the one or more processors to perform operations including: at least one of the computer operations is a declarative operation that specifies one or more properties of one or more results of the data processing; transforming the first dataflow graph into a second dataflow graph for processing data in accordance with the first computer operation, the second dataflow graph including a plurality of second nodes representing the second computer operations, at least one of the second nodes representing one or more instruction operations that implement logic specified by the declarative operations, the one or more instruction operations not represented by a first node in the first dataflow graph; and storing the second dataflow graph in a data store.

[0031] In general aspect 27, a non-transitory computer-readable medium storing instructions for causing a computing system to: generate a first dataflow graph including a plurality of first nodes representing first computer operations in data processing, at least one of the first computer operations being a declarative operation that specifies one or more properties of one or more results of the data processing; transform the first dataflow graph into a second dataflow graph for processing data in accordance with the first computer operations, the second dataflow graph including a plurality of second nodes representing the second computer operations, at least one of the second nodes representing one or more instruction operations that implement logic specified by the declarative operations, the one or more instruction operations not represented by a first node in the first dataflow graph; and store the second dataflow graph in a data store.

[0032] In general aspect 28, a method for converting a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations, the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, at least some of the second computer operations not represented by first nodes in the first dataflow graph, the method including: generating a first dataflow graph including a plurality of first nodes representing first computer operations in data processing; converting the first dataflow graph into a second dataflow graph for processing data according to the first computer operations, the second dataflow graph including a plurality of second nodes representing second computer operations, at least a given one of the second computer operations being selected from the group consisting of a sort operation, a data type operation, a join with a specified key operation, and a split operation, and at least a given one of the second computer operations not represented by first nodes in the first dataflow graph; and storing the second dataflow graph in a data store.

[0033] In general aspect 29, a system for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations, the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, at least some of the second computer operations not represented by a first node in the first dataflow graph, the system including one or more processors and one or more storage devices that store instructions that, when executed by the one or more processors, transform the first dataflow graph including the plurality of first nodes representing the first computer operations in data processing into a second dataflow graph. one or more storage devices that cause the one or more processors to perform operations including generating a first dataflow graph; transforming the first dataflow graph into a second dataflow graph for processing data in accordance with first computer operations, the second dataflow graph including a plurality of second nodes representing second computer operations, at least a given one of the second computer operations being selected from the group consisting of a sort operation, a data type operation, a join with a specified key operation, and a split operation, and at least a given one of the second computer operations is not represented by a first node in the first dataflow graph; and storing the second dataflow graph in a data store.

[0034] In general aspect 30, a non-transitory computer readable medium storing instructions for causing a computing system to generate a first dataflow graph including a plurality of first nodes representing first computer operations in processing data; transforming the first dataflow graph into a second dataflow graph for processing data according to the first computer operations, the second dataflow graph including a plurality of second nodes representing second computer operations, at least a given one of the second computer operations being selected from the group consisting of a sort operation, a data type operation, a join with a specified key operation, and a split operation, and at least a given one of the second computer operations not represented by a first node in the first dataflow graph; and storing the second dataflow graph in a data store.

[0035] All or a portion of the foregoing (e.g., aspects 1-30 and any combination thereof) may be implemented as a computer program product including instructions stored on one or more non-transitory machine-readable storage media and executable on one or more processing devices. All or a portion of the foregoing may be implemented as an apparatus, method, or electronic system that includes one or more processing devices and memory that store executable instructions and that may perform the described functions.

[0036] The term "not represented" as used herein may mean that at least some of the second computer operations do not occur directly or indirectly in the first data flow graph, or that none of the multiple first nodes represent at least some of the second computer operations.

[0037] One or more of the above embodiments may provide one or more of the following advantages: The techniques described herein allow a user with minimal technical background to specify data processing functions using an easy-to-use graphical user interface. The dataflow graph system provides a graphical user interface including a canvas and a dataset catalog having a plurality of transformation icons and dataset icons. By selecting the desired icons and arranging them on the canvas, a user can easily create and modify a dataflow graph, and thus its associated computations, without having to specify the details of dataset access or other low-level implementation details such as sorting and splitting operations. In this manner, the dataflow graph system provides a visual representation of the dataflow graph as it is developed by the user and enables schema-driven development. Additionally, the system may include an auto-layout feature that expedites development by automatically connecting icons arranged by the user.

[0038] Once the dataflow graph is complete, the system converts the dataflow graph created by the user into a transformed dataflow graph that can be compiled and executed. As part of this process, the system automatically optimizes the dataflow graph by adding imperative operations required to execute the declarative operations specified in the dataflow graph, for example, by automatically adding parallel and split operations, adding sort operations for joins and rollups, and specifying intermediate metadata. The system also improves the data processing of the dataflow graph itself, such as by removing redundant components or reducing records.

[0039] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described herein will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0040] [Figure 1] FIG. 1 is a block diagram of a system for specifying a computation graph. [Figure 2A] FIG. 2 is a block diagram of a data flow graph. [Figure 2B] FIG. 2 is a block diagram of a data flow graph. [Figure 3A] FIG. 2 is a visual representation of a dataflow graph at different stages of construction. [Figure 3B] FIG. 2 is a visual representation of a dataflow graph at different stages of construction. [Figure 3C] FIG. 2 is a visual representation of a dataflow graph at different stages of construction. [Figure 4A] 2 is a block diagram of the system of FIG. 1 at different stages of operation. [Figure 4B] 2 is a block diagram of the system of FIG. 1 at different stages of operation. [Figure 4C] 2 is a block diagram of the system of FIG. 1 at different stages of operation. [Figure 4D] 2 is a block diagram of the system of FIG. 1 at different stages of operation. [Diagram 5] 1 is a flowchart of an exemplary process for generating a computational data flow graph. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] The techniques described herein relate to techniques for generating easily modifiable computation graphs that can be optimized and transformed for compilation. In some embodiments, a dataflow graph system provides a graphical user interface that includes a canvas and a dataset catalog having a number of transformation selection icons and dataset selection icons. By selecting the desired icons and placing them on the canvas, a user can easily create, visualize, and modify a dataflow graph without having to specify (or re-specify) the details of dataset access or other low-level implementation details such as sorting and splitting operations. In general, as used herein, an operation refers to a computer operation, e.g., one or more operations or instructions executed or performed by a machine, computer system, etc.

[0042] The system receives data indicative of an icon selected by a user and generates a first dataflow graph (sometimes referred to herein as an "intermediate dataflow graph") by modifying the nodes of the dataflow graph to include declarative operations and data sets referenced by the selected icon. Once the first dataflow graph is complete, the system converts it into a second dataflow graph (sometimes referred to herein as a "transformed dataflow graph") that can be compiled and executed. As part of this process, the system automatically optimizes the first dataflow graph to generate a second dataflow graph by adding instruction operations to the second dataflow graph required to execute the declarative operations specified in the first dataflow graph, e.g., automatically adding parallel and split operations, adding sort operations for joins and rollups, and specifying intermediate metadata. The system also improves data processing by the second dataflow graph itself, such as by removing redundant components or reducing records, as described below in connection with FIG. 4.

[0043] 1 shows a schematic diagram of a system 10 for specifying a computation graph. The system 10 (also referred to herein as an "execution environment") includes a dataflow graph generation system 12 having an editor user interface generator 14, a dataflow graph (DFG) engine 16, and a transformation engine 18. The graph generation system 12 is operatively coupled to a client device 20, which provides access to the graph generation system 12, thereby also providing access to the execution environment, and also providing access to a compiler system 22 to compile the dataflow graph generated by the graph generation system 12. The compiled dataflow graph is provided to a data processing system 24 and a storage system 26 for execution and storage. The storage system 26 includes connections to one or more data sources, such as storage devices or online data sources, each of which may store or provide data in any of a variety of formats (e.g., database tables, spreadsheet files, flat text files, or native formats used by mainframe computers, among others). In particular, storage system 26 may store input data, such as metadata and record formats, that can be retrieved by data processing system 24 to execute compiled dataflow graphs, as well as output data generated through execution of the compiled dataflow graphs.

[0044] In general, the execution environment may be hosted on one or more general-purpose computers under the control of a suitable system, such as a UNIX-based operating system, a Windows-based operating system, among others. For example, the execution environment may include a multi-node parallel computing environment, which includes configurations of computer systems using multiple processing units (such as CPUs) or processor cores, either local (e.g., a multiprocessor system such as a symmetric multi-processing (SMP) computer), or locally distributed (e.g., multiple processors coupled as a cluster or massively parallel processing (MPP) system), or remote, or remotely distributed (e.g., multiple processors coupled over a local area network (LAN) or wide-area network (WAN)), or any combination thereof.

[0045] Unlike other systems that access a generated representation of a dataflow graph and generate computer instructions that define the graph, system 10 accesses an intermediate graph (e.g., dataflow graph 17) and transforms the intermediate graph through optimizations and other operations to generate a transformed graph (e.g., transformed dataflow graph 19) for compilation. For example, dataflow graph engine 16 receives selection data from client device 20 that indicates data sources, data sinks, and data processing functions of a desired computation graph. A user of client device 20 need not specify data access details or other low-level implementation details because these details can be obtained by dataflow graph generation system 12. Based on the selection data, dataflow graph engine 16 generates dataflow graph 17 or modifies a previously created dataflow graph 17. A transformation engine 18 receives the completed dataflow graph 17 and transforms the dataflow graph into a transformed dataflow graph 19, e.g., by removing redundancies in the dataflow graph 17, adding reordering or partitioning to the dataflow graph 17, and specifying intermediate metadata (e.g., metadata for transforming or otherwise transforming the dataflow graph 17 into a transformed dataflow graph 19), among other optimizations and transformations, as described below in connection with FIG. 4D. The transformed dataflow graph 19 is then provided to a compiler system 22, which compiles the transformed dataflow graph 19 into a compiled computation graph (e.g., an executable program 23).

[0046] In general, dataflow graph 17 (sometimes referred to as a "modified dataflow graph") represents the core structure of a compiled graph, such as transformed dataflow graph 19, having nodes (or components). Dataflow graph 17 optionally includes parameters (e.g., name, value, location, interpretation). In some embodiments, dataflow graph 17 includes input and output ports in the graph itself, such as in graphs intended to be used as subgraphs.

[0047] In some embodiments, a node (or component) has or is a node "kind" that indicates the behavior or functionality of the node. The node's kind is used to select a prototype for the node, to facilitate pattern matching (e.g., to find reorder nodes that follow a reorder node), and to determine the components to be instantiated in the transformed dataflow graph 19. For example, a garbage node in dataflow graph 17 may be instantiated as a garbage node in the transformed dataflow graph 19. A node (or component) may include input ports, output ports, and parameters, as described below.

[0048] Nodes optionally have a label that identifies the node. In some embodiments, if a node does not have a label, the system assigns a label to the node. Node labels may include any collection of alphanumeric characters, whitespace, and punctuation, and do not need to be unique (although they may become unique during conversion to a graph). The system may use node labels to reference nodes (or their input ports, output ports, or parameters) to, for example, define inputs and outputs of nodes, or data flows between nodes.

[0049] 2A, an embodiment of a dataflow graph 17 is shown as including nodes 34a-34n. Each of the nodes 34 includes at least one action placeholder field and at least one data placeholder field. For example, an "initial" node 34a has an action placeholder field 35a that holds one or more action elements 35a' and a data placeholder field 35b that holds one or more data source or data sink elements 35b'. An action element 35a' may specify code or a location of code that performs a function on data input or output from the initial node 34a. A data source or data sink element 35b' may specify a data source or data sink (of a function of the initial node 34a) of the initial node 34a, or a location of a data source or data sink. In some embodiments, element 35a' or element 35b', or both, include a link or address to the storage system 26, such as a link to a database or a pointer to code contained in the storage system 26. In some embodiments, element 35a' or element 35b', or both, include a script.

[0050] During construction of the dataflow graph 17, each of the nodes 34 may be modified by retrieving and adding to the respective fields the operation elements to be placed in the operation placeholder fields and the data source or data sink elements to be placed in the data placeholder fields. For example, the initial node 34a is modified during construction by retrieving (e.g., from the storage system 26) an operation element 35a' and adding a link pointing to the specified function or function to the operation placeholder field 35a, and by retrieving a data source or data sink element 35b' and adding a link pointing to the source or sink of data to the data placeholder field 35b. Once modification of a particular node 34 is complete, the node may be labeled to provide a labeled node. After each of the nodes 34 has been modified (and labeled), the completed dataflow graph 17 is stored (e.g., in the storage system 26) and used to generate other dataflow graphs, as described below.

[0051] In some embodiments, each of the nodes 34 of the dataflow graph 17 is initially unmodified. For example, each of the nodes 34 may have empty operation placeholder fields 35a and data placeholder fields 35b that are later modified to include designated operation elements 35a' and data source or data sink elements 35b', as described above. In some embodiments, the dataflow graph 17 is a previously completed dataflow graph, where some or all of the nodes 34 have corresponding operation placeholder fields 35a that hold operation elements 35a' and data placeholder fields 35b that hold data source or data sink elements 35b'. Such a completed dataflow graph 17 may be further modified (e.g., by obtaining additional or alternative elements 35a', 35b' to be placed in the respective fields 35a, 35b) and stored as a new or modified dataflow graph.

[0052] In some embodiments, a particular node, such as the initial node 34a, is "reused" to generate a new, optionally labeled, node associated with the previous node 34a. This iterative process of generating new nodes from the initial node 34a continues until the user has the specified functionality for the desired computation graph. Once the iterative process is complete, a completed dataflow graph 17 is provided. The completed dataflow graph 17 includes, for example, multiple nodes 34a-34n instantiated from the initial node 34a. The completed dataflow graph 17 may be stored (e.g., in storage system 26) and used to generate other dataflow graphs, as described below.

[0053] FIG. 2B illustrates one embodiment of a completed (e.g., modified) dataflow graph 17. The modified dataflow graph 17 is shown as including seven nodes labeled OP-0 through OP-6, each having a corresponding operation placeholder field 35a that holds an operation element, and a data placeholder field 35b that holds a data source or data sink element. For example, node 34a labeled OP-0 includes a read operation element 37a', indicating that a data source element 37b' of "Data Set I" is to be read. The dataflow graph 17 corresponds to the dataflow graph illustrated in FIG. 1 and described in more detail below with reference to FIGS. 2-4. The modified dataflow graph 17 is stored in storage system 26, for example, as a data structure.

[0054] 3A-3C, visualizations of dataflow graph 17 and transformed dataflow graph 19 are shown at various stages of construction. For example, FIG. 3A depicts visualization 70 of completed (e.g., modified) dataflow graph 17. FIG. 3B shows visualization 72 of completed dataflow graph 17 after optimization by transformation engine 18. As shown in visualization 72, optimization of completed dataflow graph 17 removed reformatting and filtering operations that transformation engine 18 determined to be redundant or unnecessary to the function of the graph, as described below in connection with FIG. 4D. FIG. 3C shows visualization 74 of transformed graph 19 generated by transformation engine 18 to perform the same function as optimized, completed dataflow graph 17 (FIG. 3B). As shown in visualization 74, additional reordering and splitting components that were not necessary for completed dataflow graph 17 have been added to transformed dataflow graph 19 to facilitate compilation and execution.

[0055] In general, the transformation engine 18 performs optimizations and / or other transformations that may be required to process data according to one or more of the operations specified in the dataflow graph 17, or to improve data processing according to one or more of the operations specified in the dataflow graph 17, as compared to data processing without the optimizations or transformations. For example, the transformation engine 18 adds, among other things, one or more reordering operations, data type operations, join operations such as join operations based on keys specified in the dataflow graph 17, split operations, automatic parallel operations, or operations that specify metadata to generate a transformed dataflow graph 19 that has the desired functionality of the dataflow graph 17. In some embodiments, the transformed dataflow graph 19 is an optimized dataflow graph (or is transformed into an optimized dataflow graph) by applying one or more dataflow graph optimization rules to the transformed dataflow graph to improve the computational efficiency of the transformed dataflow graph as compared to the computational efficiency of the transformed dataflow graph before the optimizations were applied. The dataflow graph optimization rules may include, among other things, removal of dead or redundant components, early filtering, or record reduction, as described below in connection with FIG. 4D.

[0056] 4A, an exemplary graphical editor interface 50 for editing a computation graph (e.g., dataflow graph 17) is shown. Editor UI generator 14 provides UI data 30 to client device 20 to cause client device to generate graphical editor interface 50. In general, the graphical editor interface includes a canvas portion 52 and a catalog portion 54. Catalog portion 54 includes a dataset selection portion 55 including dataset selection icons 55a-55n and a transform selection portion 57 including transform selection icons 57a-57n. Dataset selection icons 55a-55n reference corresponding elements (e.g., data source or data sink elements 35b') stored in storage system 26 and represented by the dataset selection icons. In some embodiments, these elements may include datasets (or pointers to datasets) on which operations may be performed or data may be stored. Transform selection icons 57a-57n reference corresponding elements (e.g., operation elements 35a') stored in storage system 26 and represented by the transform selection icons. In some embodiments, these elements include data (or a pointer to data) that specifies the type of operation to be performed (eg, a write operation, a join operation, etc.).

[0057] In operation, a user of the client device 20 selects an icon from the transformation selection portion 57 (e.g., one of the transformation selection icons 57a-57n) and, for example, drags the icon to the canvas 52. The user also selects an icon from the catalog selection portion 55 (e.g., one of the dataset selection icons 55a-55n) and, for example, drags the icon to the canvas 52 (which may include dragging the dataset selection icon onto the desired transformation selection icon on the canvas 52). In this example, the stored data structure or other stored data associates the icon with elements (e.g., the operation element 35a' and the data source or data source element 35b') that are used to modify the dataflow graph 17, as described below in connection with FIG. 4C. Upon user selection, the dataflow graph engine 16 receives icon selection data 32 from the generated editor interface, such as data indicative of the selected dataset selection icon and the selected transformation selection icon. The icons in the canvas 52 may be automatically connected, for example, by the graph generation system 12. In some embodiments, icons are automatically connected such that placing one icon below another in the canvas 52 causes the graph generation system 12 to automatically make a connection between the icons. In some embodiments, the user has the option to connect icons.

[0058] 4B illustrates editor interface 50 with a visual representation 60 of a completed dataflow graph in canvas portion 52. The visual representation 60 of the completed dataflow graph (also referred to herein for convenience as completed dataflow graph 60) was generated by a user selecting icons from catalog portion 54 (e.g., selecting from dataset selection portion 55 and transformation selection portion 57) and arranging the icons in a desired arrangement on canvas 52. To proceed from FIG. 4A to FIG. 4B, a user can drag icons (e.g., icons 55a-n, 57a-n) onto canvas 52 to generate the desired visual dataflow graph 60. The completed data flow graph 60 includes a first component icon 60a representing a read operation applied to dataset I, a second component icon 60b representing a read operation applied to dataset II, a third component icon 60c representing a join operation applied to dataset I and dataset II based on a key value, a fourth component icon 60d representing a reformat operation applied to the output of the join operation 60c, a fifth component icon 60e representing a filter operation applied to a result of the reformat operation 60d, a sixth component icon 60f representing a rollup operation applied to the result of the filter operation 60e, and a seventh component icon 60g representing a write operation applied to data sink I.

[0059] Icon selection data 32 indicating the selected dataset icons and selected transformation icons (and in some embodiments, their arrangement) that make up the completed dataflow graph 60 is received by the dataflow graph engine 16. As shown in FIG. 4C, the dataflow graph engine 16 modifies the dataflow graph 17 based on the selection data (e.g., the icon selection data 32) to include the selected elements to generate a completed dataflow graph 17 that has the functionality of a specified dataflow graph (e.g., the specified dataflow graph visualized in the visualization 60). To do so, the dataflow graph engine 16 processes the icon selection data 32 to identify operation elements 35a' or data elements 35b', or both, that correspond to the icon selection data. The dataflow graph engine 16 then generates the dataflow graph 17 by adding (or modifying) fields 35a, 35b of each node 34a-34n of the dataflow graph 17 using the identified elements 35a', 35b'. For example, the icon selection data 32 received by the dataflow graph engine 16 may specify that the dataflow graph 60 includes a transform selection icon 57a ​​representing a read operation to be applied to a data selection icon 55a representing data set I. The dataflow graph engine 16 may process this icon selection data 32 with one or more storage data structures or other stored data that associate the selected icon 57a ​​with an action element 35a' representing the read operation. The dataflow graph engine 16 may then add (or modify) an action element 35a' (or a link pointing to the action element 35a') corresponding to the read operation to the action field 35a of a node (e.g., node 34a) of the dataflow graph 17. Similarly, in this example, the dataflow graph engine 16 may process the received icon selection data 32 with one or more storage data structures or other stored data that associate the data set selection icon 55a with a data source element 35b' representing data set I.The dataflow graph engine 16 may then add (or modify) to the data field 35b of the node 34a of the dataflow graph 16 a data source element 35b' (or a link pointing to the data source element 35b') corresponding to the data set I.

[0060] 4D, the completed (eg, modified) dataflow graph 17 is provided to a transformation engine 18. In general, the completed data flow graph 17 may (1) contain nodes that represent redundant data processing operations; (2) may require the execution of data processing operations whose results are not subsequently used; (3) may require the execution of sequential processing unnecessarily when parallel processing is possible; (4) may apply data processing operations to more data than is required to obtain the desired result; (5) may result in computations across multiple nodes, where the computational cost of performing the computations may increase significantly if the data processing for each data flow graph node is performed by a dedicated thread of a computer program, a dedicated computer program (e.g., a process within an operating system), or a dedicated computing device; (6) may require the execution of stronger types of data processing operations that require a greater amount of computation (e.g., sort operations, rollup operations, etc.) when weaker types of data processing operations that require a smaller amount of computation would suffice (e.g., intra-group sort operations, intra-group rollup operations, etc.); (7) may require duplication of processing work; or (8) may not include operations or other transformations, or combinations thereof, that are useful or necessary for data processing, among other things.

[0061] Thus, the transformation engine 18 applies one or more of the following optimizations or other transformations, or both, to the dataflow graph 17 that may be useful or required to process the data in accordance with the operations specified in the dataflow graph 17, or to improve the processing of the data in accordance with the operations specified in the dataflow graph 17, as compared to the processing of the data without the optimization or transformation. For example, as described above, a user may create the dataflow graph 17 by selecting desired icons and placing those icons on a canvas without having to specify low-level implementation details such as sorting and splitting operations. However, these operations may be useful or required in the transformed dataflow graph 19 to compile and execute the dataflow graph to process the data in accordance with the operations specified in the dataflow graph 17, or may improve the processing of the data in accordance with the operations specified in the dataflow graph 17 (e.g., by increasing processing speed, reducing consumption of computational resources, etc.), or both. Thus, transformation engine 18 may add one or more operations to the transformed dataflow graph 19, such as a reordering operation, a data type operation, a join operation with a specified key, a split operation, an automatic parallel operation, or an operation that specifies metadata, among others, to optimize or implement the operations specified in dataflow graph 17. At least some of the operations added to the transformed dataflow graph 19 may not be present or otherwise represented in dataflow graph 17.

[0062] In some examples, to add these operations, the transformation engine 18 may insert one or more nodes 34 representing the operations to be added into the dataflow graph 17 used to generate the transformed dataflow graph 19. In some examples, the transformation engine 18 may directly insert the operations to be added in the transformed dataflow graph 19 without modifying the nodes of the dataflow graph 17. The transformation engine 18 may add these operations to all dataflow graphs 17 upon generation of their corresponding transformed dataflow graphs 19, may add these operations based on operations included in the dataflow graphs 17 (which may be identified using pattern matching techniques, as described below), or may add these operations based on some other optimization rules.

[0063] The transformation engine 18 may also optimize the dataflow graph 17 (or the transformed dataflow graph 19 itself) by applying one or more dataflow graph optimization rules to the dataflow graph to improve the computational efficiency of the transformed dataflow graph, among other things, by removing dead or redundant components (e.g., by removing one or more nodes 34 corresponding to dead or redundant components), by moving previous filtering steps in the dataflow (e.g., by moving one or more nodes 34 corresponding to filtering components), or by shrinking records. In this way, the transformation engine 18 optimizes the easily modifiable dataflow graph 17 and transforms it into an optimized, transformed dataflow graph 19 that is suitable for compilation.

[0064] Optimization of dataflow graphs as described herein may include one or more of the following optimizations: Additional optimizations of dataflow graphs may include one or more of the optimizations described in U.S. patent application Ser. No. 15 / 993,284, entitled "Systems and Methods for Dataflow Graph Optimization," the entire contents of which are incorporated herein by reference.

[0065] In some examples, the transformation engine 18 may identify two adjacent nodes (e.g., nodes 34) in the dataflow graph 17 that represent respective operations, where the second operation duplicates or negates the effect of the first operation such that one of the operations is redundant. Thus, the transformation engine 18 may optimize the dataflow graph 17 by removing the nodes 34 that represent redundant operations (e.g., the nodes that represent the duplicated or negated operations) when generating the transformed dataflow graph 19. For example, the transformation engine 18 may identify two adjacent nodes 34 that have the same operation. Since two adjacent nodes that perform the same operation are typically redundant, it is not necessary to perform both operations, and one of the two adjacent nodes may be removed to optimize the dataflow graph. As another example, the transformation engine 18 may identify two adjacent nodes 34 with a first node that represents a repartition operation (splitting data for parallel processing on different computing devices), followed by a node that represents a serialization operation (operating to consolidate all data for sequential processing by a single computing device). Because the effects of the subdivision are nullified by a subsequent serialization operation, there is no need to perform the subdivision operation (e.g., the subdivision operation is redundant) and the subdivision operation may be removed by the transformation engine 18 during the optimization process.

[0066] In some examples, the transformation engine 18 may identify a first node representing a first operation that is interchangeable with one or more other nodes that represent other operations. When the first node is interchangeable with one or more other nodes, the transformation engine 18 may optimize the dataflow graph by reordering the first node with at least one of the one or more other nodes (e.g., by shuffling the order of nodes 34). In this manner, the transformation engine 18 may optimize the dataflow graph by ordering the nodes and corresponding operations in a manner that improves efficiency, speed, or otherwise optimizes processing through the dataflow graph without changing the results. Additionally, by swapping nodes in this manner, the transformation engine 18 may be able to apply other optimizations. Referring again to the previous example, the transformation engine 18 may reorder the first node with at least one of the one or more other nodes such that the first node representing the first reordering operation is positioned adjacent to the second node representing the second reordering operation. As a result, the first and second reordering operations become redundant and the transformation engine 18 can optimize the dataflow graph by removing one of the first or second reordering operations (e.g., by removing the corresponding node 34 from the dataflow graph 17 when generating the transformed dataflow graph 19).

[0067] In some examples, transformation engine 18 may identify and remove "dead" nodes that represent operations that are unused or otherwise unnecessary. For example, transformation engine 18 may identify one or more nodes that represent operations whose results are not referenced or used (e.g., reordering operations that are not referenced because the ordering resulting from the reordering operations is not necessary or is not relied upon in subsequent processing). Thus, transformation engine 18 may optimize the dataflow graph by removing the dead or unused operations (e.g., by removing the corresponding nodes 34 when generating transformed dataflow graph 19).

[0068] In some examples, transformation engine 18 may perform a reduced strength optimization on one or more nodes. For example, transformation engine 18 may identify a first node that represents a first operation of a first type (e.g., a first sort operation on a primary key, a first rollup operation on a primary key, etc.), and subsequently identify a second node that represents a second operation of a second, weaker type (e.g., a second operation on a non-primary key, a sort within group operation, a second rollup operation on a non-primary key, a rollup by group operation, etc.). Because processing data with the first operation may require more computational resources than processing data with the second, weaker operation, transformation engine 18 may perform a reduced strength optimization to replace the first operation with the second operation (e.g., by replacing operation element 35a' or data element 35b', or both, of the first node with those of the second node).

[0069] As yet another example, transformation engine 18 may optimize the dataflow graph by aggregating two or more nodes. For example, transformation engine 18 may identify separate nodes that represent operations that may be performed by different processes executing on one or more computing devices, and may optimize the dataflow graph by aggregating the separate nodes and their respective operations into a single node, such that all operations are performed by a single process executing on a single computing device, reducing inter-process (and potentially inter-device) communication overhead. Transformation engine 18 may also identify other nodes that may be combined, such as filtering operations that may be combined with two or more separate join or rollup operations that may be combined, among many other combinations.

[0070] In some examples, the transformation engine 18 may identify nodes configured to perform some operations that may be more efficient when executed separately, and may perform a serial-parallel optimization of the dataflow graph that splits one or more of the operations (e.g., an auto-parallel operation) into separate nodes for parallel processing. The operations may then be executed in parallel using different processes executing on one or more computing devices. The transformation engine 18 may then add a merge operation to merge the results of the parallel operations. Similarly, in some examples, the transformation engine 18 may identify points in the dataflow graph that contain large chunks of data (e.g., data corresponding to large tables and indexes) and may perform a partition optimization of the dataflow graph (e.g., an auto-partition operation) to split the data into smaller partitions. The partitions may then be processed in series or in parallel (e.g., by integrating the auto-partition operation with the auto-parallel operation). By reducing the size of the data being processed, or by separating the operations for parallel processing, or both, the transformation engine 18 may significantly improve the efficiency of the resulting dataflow graph 19.

[0071] In some examples, transformation engine 18 may perform width reduction optimizations upon generation of transformed dataflow graph 19. For example, transformation engine 18 may identify data (e.g., one or more data columns) to be removed at a particular point in the dataflow graph prior to execution of a subsequent operation because the data (e.g., the data to be removed) is not used in subsequent operations and does not need to be propagated as part of the processing. As another example, a node in a dataflow graph may be configured to perform several operations, and the results of some of these operations may be unused. Thus, transformation engine 18 may perform width reduction optimizations to remove unused or unnecessary data (e.g., by inserting a node to remove data at the identified point in time, by replacing a node configured to perform multiple operations with another node configured to perform only operations whose results are used). In this manner, transformation engine 18 optimizes the dataflow graph by reducing the amount of computational resources required by the dataflow graph to carry the data through subsequent operations (e.g., by reducing network, memory, and processing resources utilized).

[0072] To identify portions of a dataflow graph (e.g., dataflow graph 17) to which one or more optimizations are applied, the transformation engine 18 (or another component of the graph generation system 12) can use a dataflow graph pattern matching language. The dataflow subgraph pattern matching language can include one or more expressions to identify specific nodes or operations in the dataflow graph for optimization, as described in more detail below. For example, the pattern matching language can include expressions to identify a series of nodes of at least a threshold length (e.g., at least 2, 3, 4, 5, etc.) that represent respective series of computations that can be consolidated and represented by a single node in the dataflow graph using the consolidated operation optimization rules. Identifying such patterns can facilitate application of the consolidated operation optimization rules described above. A preferred, but non-limiting example of one such expression is "A→B→C→D," which can be useful for identifying a series of four consecutive data processing operations that can be consolidated.

[0073] As another example, a pattern matching language may include expressions for identifying portions of a dataflow graph where certain types of nodes are interchangeable with other nodes to optimize the dataflow graph. This may facilitate applying multiple different types of optimization rules to the dataflow graph. When a data processing system determines that the order of one or more nodes in the dataflow graph can be changed without changing the processing result, the data processing system may consider modifications to the structure of the dataflow graph (as permitted by the degrees of freedom available through the interchange operations) to identify portions to which the optimization rules are applicable. As a result of considering the interchange-based modifications, one or more optimization rules may become applicable to portions of the graph to which the rules would not otherwise be applicable.

[0074] For example, as described above, an optimization rule may include identifying two adjacent nodes in an initial dataflow graph that represent respective reordering operations, where a second reordering operation nullifies the effect of the first operation such that the first operation is redundant. By definition, such optimization rules are not applied to dataflow graphs that do not have adjacent nodes that represent reordering operations. However, if a first node that represents a first reordering operation is interchangeable with one or more other nodes, it may be possible to reorder the first node with at least one of the one or more other nodes such that the first node that represents the first reordering operation is positioned adjacent to a second node that represents the second reordering operation. As a result of such node swapping, an optimization rule that removes the redundant first reordering operation may be applied to the dataflow graph.

[0075] Thus, in some examples, the pattern matching language may include one or more expressions to identify subgraphs of a dataflow graph in situations where the order of nodes in the dataflow graph may be changed. As an example, the expression "A→(...)→B" (where each of A and B may be any suitable data processing operation, such as reordering, merging, etc.) may be used to find a portion of a dataflow graph having node "A" (e.g., a node representing operation "A") and node B (representing operation B), and one or more nodes between node A and node B that are interchangeable with node A (e.g., when the order of the nodes is changed, the results of the processing performed by these nodes do not change). Once such a portion is identified, the dataflow graph may be modified or optimized by moving node A adjacent to node B to obtain portion "AB". As a specific example, if the dataflow graph has nodes ACDB and operation A is interchangeable with operations C and D, the dataflow graph may be modified to become "CDAB". The transformation engine 18 may then consider whether an optimization rule applies to portion "AB". For example, if operation A is a permutation and operation B is a permutation, transformation engine 18 may attempt to determine whether these two permutations could be replaced with a single permutation to optimize the dataflow graph.

[0076] As another example, the expression "A → (...) → B * " may be used to find a portion of a dataflow graph that has a node A, a second node B, and one or more nodes between these nodes that are interchangeable with node B. As a specific example, if a dataflow graph has nodes ACDB, and operation B is interchangeable with operations C and D, then the dataflow graph may be modified or optimized to become "ABCD". The transformation engine 18 may then consider whether an optimization rule applies to portion "AB".

[0077] As another example, the expression "A → (...) → B **" may be used to find portions of a dataflow graph that have node A, node B, and one or more nodes between node A and node B (e.g., C and D) that are inexchangeable with node B. The system may then attempt a "push" exchange (nodes C and D are pushed to the left of node A, if possible). As a specific example, if a dataflow graph has node ACEDB, and operation B is exchangeable with operation E, but not with operations C and D, then the dataflow graph may be modified to become "CDABE"; that is, B is exchanged with E, but C and D are pushed to the left of A.

[0078] As yet another example, the expression "A ** "→(...)→B" may be used to find portions of a dataflow graph that have node A, node B, and one or more nodes between node A and node B (e.g., C and D) that are inexchangeable with node A. The system may then attempt a "push" exchange (nodes C and D are pushed to the right of node B, if possible). As a specific example, if a dataflow graph has node ACEDB, where operation A is exchangeable with operation E, but not with operations C and D, then the dataflow graph may be modified to become "EABCD", i.e., node A is exchanged with E, but C and D are pushed to the right of B.

[0079] Generally, the optimization and transformation process is iterative, with each iteration of optimization or transformation (18a) transforming the dataflow graph 17 until testing (18b) indicates that further optimization or transformation is not possible, unnecessary, or desirable. For example, the transformation engine 18 may transform the dataflow graph 17 by (1) selecting a first optimization rule, (2) identifying a first portion of the dataflow graph 17 to which to apply the first optimization rule, and (3) applying the first optimization rule to the first portion of the dataflow graph 17. The data processing system may then determine whether one or more additional optimizations described herein are applicable to the dataflow graph 17 or are necessary to generate a transformed dataflow graph 19 that can be compiled and executed. If additional optimizations are applicable, the transformation engine 18 may continue updating the data flow graph 17 by (1) selecting a second optimization rule that is different from the first optimization rule, (2) identifying a second portion of the data flow graph 17 to which to apply the second optimization rule, and (3) applying the second optimization rule to the second portion of the data flow graph 17.

[0080] In the absence of further optimizations or transformations, the transformation engine 18 outputs the transformed data flow graph 19 to the compiler system 22, which compiles the transformed data flow graph 19 into an executable computation graph (e.g., executable program 23), which is provided to the data processing system 24 and storage system 26 for execution and storage.

[0081] 5 illustrates a flowchart of an example process 500 for generating a computational data flow graph. Process 500 may be performed, for example, by one or more components of system 10.

[0082] Operations of process 500 include providing data to generate a graphical editor interface including a canvas portion and a catalog portion, where the catalog portion includes one or more selectable icons in the canvas portion to visually depict the logic of the calculation (502). For example, the catalog portion may include a dataset selection icon and a transform selection icon, where each selected icon may represent an operation or an instruction to access data from a data catalog specifying a data source or sink.

[0083] Icon selection data representative of the logic of the computation depicted in the canvas portion is received (504). The icon selection data may specify at least one of one or more selectable icons selected from the catalog portion and included in the canvas portion.

[0084] A first dataflow graph is generated based on the received icon selection data (506). The first dataflow graph includes a plurality of first nodes representing first computer actions in processing the data, at least one of the first computer actions being a declarative action. In general, a declarative action specifies one or more characteristics of one or more results of processing the data without necessarily specifying how the results are achieved. The first nodes may represent logic specified in the canvas portion, and at least one of the first nodes represents a selectable icon selected from the catalog portion. In some examples, some or all of the first nodes include an action placeholder field for holding an action and a data placeholder field for holding a source or sink of data. Generating the first dataflow graph may include retrieving from a storage system elements of operations held in operation placeholder fields and adding (or modifying) operations (or links to operations) in the operation placeholder fields, and retrieving from the storage system elements of data sources or data sinks held in data placeholder fields and adding (or modifying) links pointing to sources or sinks of data in the data placeholder fields.

[0085] In some examples, a visualization of the first dataflow graph is rendered, for example, on a canvas portion of a graphical editor interface. In some examples, one or more of the first nodes are labeled to provide labeled nodes. The labels may refer to one or more of a key, a value, a name, and a source. In some examples, a visualization of the labeled nodes is rendered on the canvas portion.

[0086] The first dataflow graph is transformed (508) into a second dataflow graph to process data according to the first computer operation. The second dataflow graph includes a plurality of second nodes representing the second computer operations, where one or more of the second nodes represent one or more instruction operations. In general, the instruction operations specify (e.g., in a programming language or other machine-readable code) how to implement the logic specified by the declarative operations. The one or more instruction operations are not represented by a first node in the first dataflow graph. In some embodiments, at least one of the second operations represented in the second dataflow graph and not represented in the first dataflow graph includes a reordering operation, a data type operation, a join with a specified key operation, or a split operation. In some embodiments, at least one of the second operations includes an auto-parallel operation or an auto-split operation, which may include inserting an auto-parallel node or an auto-split node into the second dataflow graph. In some embodiments, at least one of the second operations includes an operation of specifying metadata between one or more of the second nodes. In some embodiments, a first node of a first dataflow graph, which may be a labeled node, may be compiled into a second dataflow graph, which is a computational dataflow graph. The node or elements stored in the node may be optimized.

[0087] In some embodiments, one or more of the second operations (i) are necessary to process data in accordance with one or more of the first operations specified in the first dataflow graph, or (ii) improve data processing in accordance with one or more of the first operations specified in the first dataflow graph compared to data processing without the one or more additional operations.

[0088] In some embodiments, the second dataflow graph is transformed into an optimized dataflow graph by applying one or more dataflow graph optimization rules to the second dataflow graph to improve the computational efficiency of the second dataflow graph compared to the computational efficiency of the second dataflow graph prior to the application of the one or more dataflow graph optimization rules, which may include, for example, dead component elimination, early filtering, or record reduction, among others, as described below in connection with Figures 4A-4D.

[0089] The second data flow graph is stored in a data store (510). In some embodiments, the second data flow graph is compiled by a compiler system to generate a compiled data flow graph (e.g., an executable program). The compiled data flow graph may be provided to data processing systems and storage systems for execution and storage.

[0090] In some embodiments, a prototype node is accessed and an algorithm is applied that copies parameters from the accessed prototype node to modify at least one of the first nodes. At least one parameter of the at least one node may be a configuration parameter that is not overwritten by the prototype node. The prototype node may declare ports on the at least one node or the node itself. In some embodiments, the prototype node declares parameters of a component that is shown on the canvas of the editor interface. Applying the prototype may replace existing parameter descriptors with descriptors from the prototype, but may not replace existing values ​​of the parameters.

[0091] Lexical structure: A dataflow graph, such as dataflow graph 17, has a lexical structure. In some embodiments, the lexical structure includes symbols, keywords, numbers, strings, code-like strings, and other lexical elements.

[0092] The symbols may include letters, numbers, and punctuation marks (e.g., an underscore or a period), where at least one character is not a punctuation mark character. In some implementations, other rules may be employed, such as a rule that the first character of a symbol cannot be a number.

[0093] The keywords may include short symbols that contain only lowercase letters. A structure (eg, a data structure) may be developed for all the listed keywords.

[0094] Numbers can be signed integers of any length. For numbers that are parameter values, numbers that are too large to represent (e.g., as an int8) can be converted to parameters using a string value instead of a numeric value.

[0095] Strings may be quoted using single or double quotes. Within a string, the closing quote character may be escaped with a backslash.

[0096] Code-like strings may be quoted with parentheses. Within the parentheses, balanced and nested parentheses are permitted, as are single- and double-quoted strings. This representation may be used for key specifiers, transformations, metadata, and multi-line text parameter values. Also, within the parentheses, constant indentation is removed during parsing and reinserted during serialization. Thus, it is an indented data manipulation language (DML) that matches the graph structure.

[0097] Ports, flows, and branches: Ports can be input and output ports. Each port has a name (e.g., "in", "out", "reject0") unless the port is the default input or output port of those components (in which case the name can be omitted). Ports have flows. A flow can connect one output port to one input port. Ports have parameters. Ports can be bound to graph ports. For example, an input port can be bound to a graph input port and an output port can be bound to a graph output port.

[0098] Flows can be specified as elements of the graph, separate from nodes: they can be specified at a top level of the node, indicating a flow to a default input port, or they can be specified within an input port.

[0099] A flow may be specified implicitly by putting a sequence of nodes into a branch. Within a branch, the node keyword is not needed to introduce a new node. A branch has an ID, and a branch can be specified if a flow can specify a node by ID. A flow from a branch may be a flow from the last node of the branch. A flow to a branch may be a flow to the first node of the branch.

[0100] If there is only one flow from a node or branch, then that node or branch may be designated as part of a flow that is nested inside the node that consumes its output.

[0101] Graph elements may reference other elements of the graph. For example, flows may reference nodes, parameters may reference other parameters, key parameters may reference values, among others. To allow these references, when an element is introduced it may optionally be given an ID. Different element types may have separate ID spaces. In some implementations, IDs are optional, but without an ID an element cannot be referenced.

[0102] Parameters: A parameter may have a name, a value which may be a locator, and an interpretation. A parameter may be a symbol. Parameter names preceded by punctuation characters (e.g., a period) are considered private to the dataflow graph and are not used to generate parameters in other graph models (such as transformed dataflow graphs). Parameter names starting with other character patterns (e.g., two leading periods) are considered transition states and, by default, are not saved with the graph. This may be overridden by command.

[0103] Parameters are defined by giving them values. Parameter values ​​may be translated into graph parameters as strings, but other forms may be allowed. These other forms are preserved in the internal model and on serialization. For graph algorithms such as deduplication, two parameters may be considered identical if their structural type is identical and their values ​​are identical. For pattern matching, the type of the pattern parameter controls (e.g., the actual parameter is compared to a pattern that is converted to that type). This allows a pattern to assert a true-valued parameter, for example, using "param sorted=true", which will match nodes where the parameter "sorted" has the integer value 1 or the string value "true" (or any of the other parameter values ​​that would typically be interpreted as "true").

[0104] The parameter can be located at a specified location. When creating another graph model, the locator can be saved. An algorithm that uses the value of the parameter can resolve the locator path and read the contents of the file.

[0105] Graph-level parameters can be used declarations when translating dataflow graph 17 into other graph models such as transformed dataflow graph 19. A declaration has many parts (e.g., input or output parameters, parameter types, required parameters, exported parameters, parameter command line interface, parameters read from the environment), each of which may have a default state. The declaration defaults can be overridden by providing values ​​for particular parameters.

[0106] Within a node prototype or replacement node, a parameter may have a descriptor that can be used to determine the user interface used to edit the parameter and the validation applied to the parameter value. In one example, the descriptor is a set of properties immediately following the parameter keyword.

[0107] Parameter descriptors can be open-ended in the language. For example, new fields can be used in a prototype file and realized in the user interface at runtime without any code changes. Parameter values ​​are validated against the descriptors in the built-in "apply-prototypes" pass described below every time a substitution occurs.

[0108] Read-write mutable graph: The dataflow graph may be read from a file or from command line arguments, such as by using a command line utility.

[0109] Below is an example of a complete dataflow graph using the lexical structures described herein.

[0110] [Table 1]

[0111] Dataflow graph prototype: Parameter values ​​may appear in scripts as arguments rather than as parameters, and do not generate parameters in other graph models. Instead, the parameters are instructions to the dataflow graph transformation code, and set the component paths of the generated, translated components. Prototypes allow each node of a type to have these same parameter values.

[0112] In some embodiments, prototypes are specified as part of the dataflow graph. Prototypes may not be immediately applied, so parsing and serializing the graph results in no changes. Built-in algorithms may be provided to copy parameters from prototypes. Pre-set parameters are not overwritten by prototypes. Common component types used by the translator are enumerated and have names. A mutable utility generates a default set of component prototypes. This makes graph generation easier, as fewer parameters need to be explicitly specified.

[0113] Prototypes are applied in some circumstances, such as by applying a prototype algorithm, by explicitly calling a prototype routine in the code within the algorithm that introduces a new node, or always upon creation of a node by application of mapping rules.

[0114] Prototypes can also be used to declare parameters of nodes, ports, and components for use in presenting an editing interface. This usage is described below in "PROTO Files."

[0115] Applying a prototype does not replace any existing values ​​of parameters. However, it replaces the existing parameter's descriptors with the descriptors from the prototype. This allows information about the parameter's type to be placed in the descriptors, which are then applied to the parameter value.

[0116] Dataflow graph labels: A node may have zero or more labels. When a component is created from a node, the node's labels are capitalized and combined to form the component label. When nodes are merged, their label sets may also be merged. In some embodiments, priorities may be assigned to labels to reduce label proliferation and generate meaningful final labels. Labels may be named labels and may refer to sources, keys, or values, among others.

[0117] Dataflow graph algorithms: In some embodiments, one or more algorithms may be applied to transform or optimize the data flow graph, including, among others, a pattern matching pass, a built-in pass that applies prototypes (described above), a built-in pass that performs deduplication, a built-in pass that performs explicit duplication, and a built-in pass that uncrosses flows.

[0118] In some embodiments, the data flow graph algorithm includes a pattern matching path. The pattern matching path includes rules, each rule having a pattern and a replacement. The pattern may specify parameter values ​​required for the pattern to match. The pattern may include more than one node. Node types may be wildcarded, but doing so may be computationally expensive. A replacement may consist of zero nodes (to completely remove the matching section of the graph), a single node, or multiple nodes connected by flows. If a wildcard node type is used in the pattern, the wildcard may be bound to the matched node type and used in the replacement.

[0119] In some embodiments, the data flow graph algorithm includes a de-duplication pass, which finds sub-graphs that are functionally identical and merges them. Two nodes are considered identical if they have no input flows or if their input flows are from the same node and their parameters match. In some embodiments, output flows are not considered when checking for node equality. Labels may also not be considered for testing node equality.

[0120] Two nodes that are combined in deduplication can collect all labels from the input nodes. Deduplication can result in a node having multiple output flows from a single port. Deduplication can also result in bubbles in the graph where flows diverge from a single port and then reconverge. Other algorithms include explicit duplication-built-in paths and uncross-flow built-in paths, as described below.

[0121] In some embodiments, the dataflow graph algorithm includes an explicit duplicate path. In a dataflow graph, very few component types support multiple flows from an output port. However, the dataflow graph model may allow multiple flows into and out of any port. An explicit duplicate path creates a node of type "duplicate" and with no other properties downstream of any port with multiple output flows, and moves the flow to that node. Subsequent pattern matching passes may reformat some of these to multiple outputs.

[0122] In some embodiments, the data flow graph algorithm includes uncross flow paths, which attempts to flatten the graph by reordering flows into gather ports and renumbering input ports when joining. This is a heuristic algorithm since many graphs are not planar.

[0123] Dataflow graph integration with the Dataset Catalog: The data source catalog stores metadata about the data sources, among other things the URL used to access the data, the record format of the data, sort and partition keys, if any, etc. The dataflow graph 17 can use the catalog of query instances or a catalog exported to a Catalog Interchange Format (CIF) file, or can embed data source information directly in the graph.

[0124] For example, if the system assigns a variable utility to a catalog file, the graph can reference the data source by name in that catalog. Having introduced the data source, its properties can be referenced elsewhere in the dataflow graph. Source parameters can be resolved at a later stage in processing the dataflow graph 17, so that the source of a particular parameter value remains unambiguous. This is particularly useful when the parameters are large.

[0125] Dataflow graph extensions: Dataflow graphs can be extended for value-oriented processing and graph semantics. In some embodiments, dataflow graphs can be extended by using aspects, expression values, built-in paths, vias, gathers, merges, flow integration, and aspect propagation, among others.

[0126] An aspect is some information or meta-information that propagates along the flow of a graph. A prime example is a value: a value is computed at one node in the graph and can be used by any node further downstream. Other aspects include, among others, the set of rows that exist along the flow (rowset), the order of those rows (order), and partitioning keys for parallel data (partitions). The layout of nodes is also propagated by the same mechanism, so the layout can be treated as a separate aspect.

[0127] New aspects are introduced with the (optional) "new" keyword. Values ​​are usually given an ID so that they can be referenced later. Other aspects can be referenced without an ID, as they are singletons and only the most recent aspect can be referenced. Aspects are available for reference in nodes downstream of the node or port where they are introduced until they are explicitly removed or (in the non-value case) replaced.

[0128] Aspects are used to model the effects of nodes, so that the system can indicate which nodes have effects and which may be reorderable. Values ​​are used in this way and also determine the metadata on ports at the time of generation of transformation functions.

[0129] New expression values ​​can be constructed from DML expressions. Expressions can be constant or can reference other values ​​by ID. Expression values ​​are also used to construct values ​​just as they are in the appropriate type for output.

[0130] In some embodiments, built-in paths can be used to combine common aspects and collapse combinations: for example, if a value has been calculated once and is still valid, a new value using the same formula is redundant and the existing value can be used instead.

[0131] In some implementations, built-in paths may be used to remove empty nodes. For example, a node has a "side effect" if it writes to a data source, introduces a new value or other aspect, or ends the life of a value or other aspect with "del". Nodes that have no side effects are considered empty, and the path "remove-empty-nodes" removes empty nodes. Some nodes may be excluded, for example any node with the "keep-alive" flag set, or any node with a port bound to a graph-level port, or any node with multiple input flows to a single unified port.

[0132] In a graph that contains paths that diverge and then reconverge (as in a self-join), it is possible for the same value to reach a node along multiple paths. In a join, the values ​​along different paths end up being different values. This is done by introducing "via" values ​​that indicate the port at which a value arrived. Via values ​​end the life of the value they reference, and any downstream references use the via value.

[0133] In some embodiments, via values ​​are only needed when there is ambiguity, and should not be needed for graphs that have a tree shape. The de-duplication pass introduces via values ​​because it introduces re-convergence flows.

[0134] When multiple flows arrive at a single input port, multiple row sets may be merged into a single row set. The values ​​of the input flows are matched together to produce a unified form for output. If values ​​arrive at an input port along all flows, no explicit unification is required and the values ​​are visible as a single value downstream of the input port. If values ​​arrive at an input port along only part of the flows, they are not visible downstream of the input port.

[0135] A new aspect (a specific value) can be introduced at a node or a port. This aspect is then visible downstream of that port according to the propagation rules. In general, aspects propagate along flows, across bonds to graph-level ports. An aspect introduced at an output port propagates out of the flow from that output port. An aspect introduced at a node propagates to and from all of the node's output ports. An aspect introduced at an input port propagates to the node as a whole, and from there to its output ports and down their flows.

[0136] As mentioned above for "unity", input ports with multiple input flows allow propagation of a value if the value appears in all input flows. Additionally, error and log ports may be configurable to not propagate data from upstream - they are tagged with "passthrough none". Other ports only propagate from a single input port - they are tagged with the name of that port.

[0137] Name, metadata, and transformations: Once a dataflow graph has been optimized to provide an optimized dataflow graph, the graph may be transformed into a transformed dataflow graph (e.g., transformed dataflow graph 19). To do so, transformations may be identified that compute values ​​described in the metadata and the dataflow graph.

[0138] Values ​​may be given a name when introduced. Such a name is a hint used to assign a final value name, but is not required. If two values ​​with the same given name are valid across flows, the system may choose a different name for one of the values. Value names may be locked, indicating that the value will have that name when it appears in record form.

[0139] A value may have a structured comment. If a value with a structured comment is included as a field in record format, the value's structured comment is applied to the field. If the value does not have a direct structured comment, the value is computed by extracting the fields from the record where the structured comment exists, and the structured comment is propagated forward.

[0140] The built-in path "assign-names" ensures that certain rules are followed: all values ​​that are valid across flows have names, and these names determine the fields needed in the record format for that flow; the set of values ​​that are valid across flows determines the fields needed in the record format for that flow; the names of the values ​​are used to determine the field names in the record format; two values ​​that work across the same flow have different names; the record format for multiple flows entering a single port (gather) is identical; for certain values ​​that are unified to a single value at a port, those values ​​have the same name; values ​​with locked names retain their names;

[0141] In some implementations, the built-in "assign-names" path may generate an error if a locked name prevents any resolution. Otherwise, a name is assigned by adding a new node that potentially introduces a new value with the desired name. Because this path is relatively late in the ordering, there are fewer flows over which it is valid, and it is narrow.

[0142] The built-in path "set-metadata" assigns a "metadata" parameter to all ports in the graph, assuming that all values ​​have names. An attempt is made to enforce metadata similarity. If the metadata on a port has the same structure and field names as the input metadata, the input metadata is reused as is. If the only difference is the dropping of a trailing newline field, that field is preserved in the output. If the metadata is not identical, but some field names from the input appear in the output, those fields appear first and in the original order. If the metadata is an exact structural match for the record format of any data source present in the dataflow graph, the record format of the metadata is set to the metadata string or locator that exactly matches that source. The built-in path "set-metadata" may introduce other nodes that unify and make the record formats identical upstream of the gatherer.

[0143] Another built-in path assumes that all ports have metadata and builds transformations, packages, and expression-valued parameters from the values ​​of each node.

[0144] PROTO file: A dataflow graph engine's language has two parts: a presentation layer, which describes the node types allowed and how to edit the nodes, and a compilation layer, which describes the implementation of each node in terms of lower level nodes. Prototype files describe the presentation layer of the language. A single prototype file can contain multiple prototype sections.

[0145] A prototype can be as simple as naming a type of node, such as a node that performs a statistical operation by calling a statistical function.

[0146] The node descriptor tells the editor how to display the node and where in the organizer it should appear, for example by specifying a display name, a description of the node (e.g., "Read from Dataset"), the category the node belongs to (e.g., "Dataset"), and the shape of the node.

[0147] Nodes may have input and output ports. Ports can be made visible in the editor by mentioning them in the prototype. Each of the input and output ports may be named. Like nodes, ports may have descriptors that describe various functions of the port, such as bindings.

[0148] Implementations of the subject matter and operations described herein may be implemented in digital electronic circuitry, computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. An embodiment of the subject matter described herein may be implemented as one or more computer programs (also called data processing programs), i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus. A computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random access or serial access memory array or device, or one or more combinations thereof. A computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The subject matter may be implemented in computer program instructions stored on a non-transitory computer storage medium.

[0149] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or data received from other sources.

[0150] The term "data processing apparatus" encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the above. An apparatus may include special purpose logic circuitry (such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)). In addition to hardware, an apparatus may also include code that provides an execution environment for the computer program in question (e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof). The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0151] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, such as compiled or interpreted, declarative, procedural, etc., and may be deployed in any form, such as a stand-alone program, or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may or may not correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program may be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0152] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform operations by operating on input data to generate output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)).

[0153] Processors suitable for executing a computer program include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing operations according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, transfer data to, or both, one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, although a computer need not have such devices. Furthermore, a computer may be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive)). Devices suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0154] An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described herein), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0155] A computing system may include users and servers. The users and servers are generally remote from each other and typically interact through a communications network. The relationship of users and servers arises by virtue of computer programs running on the respective computers and having a user-server relationship to each other. In some examples, the server sends data (e.g., HTML pages) to a user device (e.g., for the purposes of displaying the data and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., results of user interaction) can be received from the user device at the server.

[0156] Although the present specification contains many specific embodiment details, these should not be construed as limitations on the scope of any embodiment or what may be claimed, but rather as descriptions of features specific to the particular embodiment. Certain features described in the present specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described as acting in a particular combination, and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0157] Similarly, although operations are shown in the figures in a particular order, it should not be understood that such operations need to be performed in the particular order or sequence shown, or that all of the illustrated operations need to be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above examples should not be understood as requiring such separation in all examples, and it should be understood that the program components and systems described may generally be integrated in a single software product or packaged in multiple software products.

[0158] Other embodiments are within the scope of the following claims.

Claims

1. 1. A computer-implemented method for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations and the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, the method comprising: generating the first dataflow graph including the first nodes representing the first computer operations in a data processing, at least one of the first computer operations being a declarative operation that specifies one or more properties of one or more results of the data processing; identifying patterns among at least some of the declarative actions represented in the first dataflow graph; transforming the first dataflow graph into the second dataflow graph for processing data according to the plurality of first computer operations based on the pattern identified among the at least some of the declarative operations, the second dataflow graph including the plurality of second nodes representing the plurality of second computer operations, at least one of the second nodes representing one or more instruction operations implementing logic specified by the declarative operations; storing the second dataflow graph in a data store.

2. Transforming the first data flow graph into the second data flow graph includes: creating said command actions; 2. The computer-implemented method of claim 1, further comprising: creating a given second node that represents the instruction operation, the given second node not being represented in the first dataflow graph.

3. 2. The computer-implemented method of claim 1, wherein one of the plurality of second computer operations represented in the second dataflow graph and not represented in the first dataflow graph is selected from the group consisting of a reorder operation, a data type operation, a join with a specified key operation, and a split operation.

4. One or more of the plurality of second computer operations comprises at least (i) is necessary to process data in accordance with one or more of the first plurality of computer operations specified in the first dataflow graph; or 2. The computer-implemented method of claim 1, further comprising: (ii) improving data processing in accordance with one or more of the first plurality of computer operations specified in the first dataflow graph as compared to data processing without the one or more additional operations.

5. 2. The computer-implemented method of claim 1, further comprising transforming the second dataflow graph into an optimized dataflow graph by applying one or more dataflow graph optimization rules to the second dataflow graph to improve computational efficiency of the second dataflow graph compared to computational efficiency of the second dataflow graph before said application of the one or more dataflow graph optimization rules.

6. 6. The computer-implemented method of claim 5, wherein the one or more dataflow graph optimization rules include at least one of: removing redundant nodes from the second dataflow graph, removing dead nodes from the second dataflow graph, reordering nodes in the second dataflow graph, reducing strength of nodes in the second dataflow graph, consolidating two or more nodes in the second dataflow graph, converting a node in the second dataflow graph from a serial operation to a parallel operation, or inserting a split operation in the second dataflow graph.

7. The computer-implemented method of claim 1, wherein at least one of the plurality of second computer operations includes an automatic parallel operation or an automatic partitioning operation.

8. The computer-implemented method of claim 1, wherein at least one of the plurality of second computer operations includes a sorting operation.

9. The computer-implemented method of claim 1, wherein at least one of the plurality of second computer operations includes an operation of specifying metadata between one or more of the second nodes.

10. providing data to generate a graphical editor interface including a canvas portion and a catalog portion, the catalog portion including one or more selectable icons in the canvas portion for visually depicting logic of a computation; receiving icon selection data indicative of logic of a computation depicted in the canvas portion, the icon selection data specifying at least one of the one or more selectable icons selected from the catalog portion and included in the canvas portion; 2. The computer-implemented method of claim 1, further comprising: generating the first dataflow graph from the received icon selection data, the first dataflow graph including the plurality of first nodes representing the logic specified in the canvas portion, at least one of the first nodes representing the at least one of the one or more selectable icons selected from the catalog portion.

11. 11. The computer-implemented method of claim 10, wherein each selected icon represents an instruction to access data from a data catalog that preformats the data or specifies a format of the data to be accessed via the data catalog.

12. The computer-implemented method of claim 1 , wherein the first dataflow graph is a user-defined dataflow graph.

13. providing data to generate a graphical editor interface including a canvas portion and a catalog portion, the catalog portion including a plurality of data set selection icons and a plurality of transform selection icons; generating an initial node in the first data flow graph according to elements stored in a storage unit represented by a selected dataset selection icon and a selected transformation selection icon; labelling the initial nodes to provide labelled nodes; The computer-implemented method of claim 1 , further comprising: rendering a visual representation of the labeled nodes on the canvas portion.

14. 14. The computer-implemented method of claim 13, wherein the initial node has an action placeholder field for holding an action and a data placeholder field for holding a source or sink of data.

15. The fix is, retrieving the action elements held in the action placeholder fields from a storage system; 15. The computer-implemented method of claim 14, further comprising: retrieving from the storage system an element of a data source or data sink held in the data placeholder field and adding a link to the data placeholder field that points to the source or sink of the data.

16. The computer-implemented method of claim 13 , further comprising providing data to render the first dataflow graph on the canvas portion of the graphical editor interface.

17. Once all of the generated initial nodes have been labeled, the method comprises:

14. The computer-implemented method of claim 13, further comprising compiling all labeled nodes of the first dataflow graph into the second dataflow graph, the second dataflow graph being a computational dataflow graph.

18. Once all of the modified initial nodes have been labeled, the method comprises:

14. The computer-implemented method of claim 13, further comprising optimizing all labeled nodes of the first dataflow graph, wherein optimizing the labeled nodes of the first dataflow graph further comprises optimizing the element stored in at least one of the labeled nodes.

19. Accessing a prototype node; 14. The computer-implemented method of claim 13, further comprising: copying parameters from the accessed prototype nodes and applying an algorithm to modify at least one of the initial nodes.

20. 20. The computer-implemented method of claim 19, wherein at least one parameter of the initial node is a configuration parameter that is not overwritten by the prototype node.

21. 20. The computer-implemented method of claim 19, wherein the prototype node declares at least one of the initial node, a port on the initial node, or a parameter of a component presented in the canvas portion of the graphical editor interface.

22. 20. The computer-implemented method of claim 19, wherein applying a prototype replaces existing parameter descriptors with descriptors from the prototype but does not replace existing values ​​of the parameters.

23. The computer-implemented method of claim 1 , further comprising applying a transformation to compute values ​​described in metadata and the first dataflow graph.

24. The computer-implemented method of claim 13 , wherein the label references one or more of a key, a value, a name, and a source.

25. 14. The computer-implemented method of claim 13, wherein at least some of the plurality of initial nodes that store one or more elements stored in a storage unit represented by a selected dataset selection icon and a storage unit represented by a selected transformation selection icon specify, at least in part, a corresponding storage unit function for the at least some of the plurality of initial nodes.

26. 1. A system for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations and the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, the system comprising: one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, generating the first dataflow graph including the first nodes representing the first computer operations in a data processing, at least one of the first computer operations being a declarative operation that specifies one or more properties of one or more results of the data processing; identifying patterns among at least some of the declarative actions represented in the first dataflow graph; transforming the first dataflow graph into the second dataflow graph for processing data according to the plurality of first computer operations based on the pattern identified among the at least some of the declarative operations, the second dataflow graph including the plurality of second nodes representing the plurality of second computer operations, at least one of the second nodes representing one or more instruction operations implementing logic specified by the declarative operations; one or more storage devices that cause the one or more processors to perform operations including: storing the second dataflow graph in a data store.

27. A non-transitory computer readable medium storing instructions, the instructions comprising: generating a first dataflow graph including a plurality of first nodes representing first computer operations in a data processing, at least one of the first computer operations being a declarative operation that specifies one or more properties of one or more results of the data processing; identifying patterns among at least some of the declarative actions represented in the first dataflow graph; transforming the first dataflow graph into a second dataflow graph for processing data according to the first computer operations based on the pattern identified among the at least some of the declarative operations, the second dataflow graph including a plurality of second nodes representing second computer operations, at least one of the second nodes representing one or more instruction operations implementing logic specified by the declarative operations; storing the second data flow graph in a data store; and

28. 1. A computer-implemented method for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations and the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, the method comprising: generating the first dataflow graph including the first nodes representing the first computer operations in processing data; identifying patterns among at least some of the declarative actions represented in the first dataflow graph; transforming the first dataflow graph into the second dataflow graph for processing data according to the plurality of first computer operations based on the pattern identified among the at least some of the declarative operations, the second dataflow graph including the plurality of second nodes representing the plurality of second computer operations, at least a given one of the plurality of second computer operations being selected from the group consisting of a reorder operation, a data type operation, a join with a specified key operation, and a split operation; storing the second dataflow graph in a data store.

29. 1. A system for transforming a first dataflow graph into a second dataflow graph, the first dataflow graph including a plurality of first nodes representing a plurality of first computer operations and the second dataflow graph including a plurality of second nodes representing a plurality of second computer operations, the system comprising: one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, generating the first dataflow graph including the first nodes representing the first computer operations in processing data; identifying patterns among at least some of the declarative actions represented in the first dataflow graph; transforming the first dataflow graph into the second dataflow graph for processing data according to the plurality of first computer operations based on the pattern identified among the at least some of the declarative operations, the second dataflow graph including the plurality of second nodes representing the plurality of second computer operations, at least a given one of the plurality of second computer operations being selected from the group consisting of a reorder operation, a data type operation, a join with a specified key operation, and a split operation; one or more storage devices that cause the one or more processors to perform operations including: storing the second dataflow graph in a data store.

30. A non-transitory computer readable medium storing instructions, the instructions comprising: generating a first dataflow graph including a plurality of first nodes representing first computer operations in processing data; identifying patterns among at least some of the declarative actions represented in the first dataflow graph; transforming the first dataflow graph into a second dataflow graph for processing data according to the first computer operations based on the pattern identified among the at least some of the declarative operations, the second dataflow graph including a plurality of second nodes representing second computer operations, at least a given one of the second computer operations being selected from the group consisting of a reorder operation, a data type operation, a join with a specified key operation, and a split operation; storing the second data flow graph in a data store; and 31. The computer-implemented method of claim 1, wherein at least some of the plurality of second computer operations are not represented by the first node in the first data flow graph.

32. The computer-implemented method of claim 1, wherein the one or more instruction actions are not represented by the first node in the first data flow graph.

33. The system of claim 26, wherein at least some of the plurality of second computer operations are not represented by the first node in the first data flow graph.

34. The system of claim 26, wherein the one or more instruction actions are not represented by the first node in the first data flow graph.

35. The non-transitory computer-readable medium of claim 27, wherein the one or more instruction operations are not represented by the first node in the first data flow graph.

36. The computer-implemented method of claim 28, wherein at least some of the plurality of second computer operations are not represented by the first node in the first data flow graph.

37. The computer-implemented method of claim 28, wherein the at least a given one of the plurality of second computer operations is not represented by a first node in the first data flow graph.

38. The system of claim 29, wherein at least some of the plurality of second computer operations are not represented by the first node in the first data flow graph.

39. The system of claim 29, wherein the at least one of the plurality of second computer operations is not represented by a first node in the first data flow graph.

40. The non-transitory computer-readable medium of claim 30, wherein the at least given one of the second computer operations is not represented by the first node in the first data flow graph.

Images