SYSTEM AND METHOD FOR PERFORMING DATA PROCESSING OPERATIONS WITH VARIABLE LEVELS OF PARALLELISM - Patent application

Abstract

A technique for determining a processing layout for nodes of a dataflow graph, the technique including: obtaining information specifying a dataflow graph, the dataflow graph including a plurality of nodes and a plurality of edges connecting the plurality of nodes, the plurality of edges representing a flow of data between the nodes in the plurality of nodes, the plurality of nodes including a first set of one or more nodes and a second set of one or more nodes independent of the first set of nodes, obtaining a first set of one or more processing layouts for the first set of nodes, determining a processing layout for each node in the second set of nodes based on the first set of processing layouts, one or more layout decision rules including at least one rule for selecting among processing layouts having different degrees of parallelism, and information indicating that data generated by at least one node in the first set of nodes and / or the third set of nodes is not used by any node in the dataflow graph downstream from the at least one node.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. patent application Ser. No. 63 / 250,763, filed Sep. 30, 2021, and entitled “Systems and Methods for Performing Data Processing Operations Using Variable Level Parallelism” (Attorney Docket No. A1041.70071US00), which is incorporated herein by reference in its entirety. [Background technology]

[0002] A data processing system may process data using one or more computer programs. One or more of the computer programs utilized by the data processing system may be deployed as a data flow graph. A data flow graph may contain components called "nodes" or "junctions," which represent data processing operations performed on input data, and links between components that represent the flow of data. The nodes of a data flow graph may include one or more input nodes representing each input data set, one or more output nodes representing each output data set, and one or more nodes representing data processing operations performed on the data. Techniques for performing computations encoded by data flow graphs are described in U.S. Pat. No. 5,966,072, entitled "Executing Computations Expressed as Graphs," and U.S. Pat. No. 7,716,630, entitled "Managing Parameters for Graph-Based Computations," each of which is incorporated herein by reference in its entirety. Summary of the Invention [Means for solving the problem]

[0003] Some embodiments include a method for implementing a data flow graph, using at least one computer hardware processor, comprising: obtaining information specifying a data flow graph, the data flow graph including a plurality of nodes and a plurality of edges connecting the plurality of nodes, the plurality of edges representing a flow of data between the plurality of nodes, the plurality of nodes including: a first set of one or more nodes, each node in the first set of nodes representing a respective input dataset in a set of one or more input datasets; a second set of one or more nodes, each node in the second set of nodes representing a respective output dataset in a set of one or more output datasets; and a third set of one or more nodes, each node in the third set of nodes representing at least one respective data processing operation; obtaining a first set of one or more processing layouts for a set of output data sets, where the first set of processing layouts includes processing layouts having different degrees of parallelism and / or the second set of processing layouts has different degrees of parallelism; and determining a processing layout for nodes in the third set of nodes using (a) the first set of processing layouts, (b) the second set of processing layouts, (c) one or more layout decision rules including at least one rule for selecting among the processing layouts having different degrees of parallelism, and (d) information indicating that data generated by at least one node in the first set of nodes and / or the third set of nodes (e.g., only the first set, only the third set, or both the first and third sets) is not used by any node in the dataflow graph downstream from the at least one node.

[0004] In some embodiments, the method further includes identifying at least one node in the first set of nodes and / or the third set of nodes.

[0005] In some embodiments, determining the processing layout is performed using two layout propagation passes: in a forward pass starting from a node in the first set of nodes, determining one or more initial processing layouts for one or more nodes in the third set of nodes using a first set of processing layouts, one or more layout decision rules, and information indicating that data generated by the at least one node is not used by any node in the dataflow graph downstream from the at least one node, in accordance with the structure of the dataflow graph; and in a backward pass starting from a node in the second set of nodes, determining a processing layout for one or more nodes in the third set of nodes by using a second set of processing layouts, the one or more initial processing layouts, and the one or more layout decision rules in accordance with the structure of the dataflow graph.

[0006] In some embodiments, the method further includes identifying at least one node in the forward pass.

[0007] In some embodiments, the processing layout associated with at least one node is not propagated from the at least one node in the dataflow graph to one or more downstream nodes during a forward pass.

[0008] In some embodiments, the processing layout associated with at least one node is not propagated from the at least one node to one or more upstream nodes in the dataflow graph during the backward pass.

[0009] In some embodiments, the method further includes determining whether the dataflow graph is processed as a micrograph based on a processing layout of the nodes in the first set of nodes, the second set of nodes, and the third set of nodes.

[0010] In some embodiments, determining whether the dataflow graph is to be executed as a micrograph includes determining to execute the dataflow graph as a micrograph if the processing layouts of the first set of nodes, the second set of nodes, and the third set of nodes have the same degree of parallelism.

[0011] In some embodiments, determining whether the dataflow graph is to be executed as a micrograph includes determining to execute the dataflow graph as a micrograph if processing layouts of the first set of nodes, the second set of nodes, and the third set of nodes, excluding at least one node, have the same degree of parallelism.

[0012] In some embodiments, the third set of nodes includes a first node, and the plurality of edges includes a first edge between the first node and a second node that precedes the first node in the dataflow graph, and determining one or more initial processing layouts for the one or more nodes in the third set of nodes includes determining a first initial processing layout for the first node based on a second initial processing layout determined for the second node.

[0013] In some embodiments, the plurality of edges includes a second edge between the first node and a third node that precedes the first node in the data flow graph, and a third initial processing layout is associated with the third node, and determining a first initial processing layout for the first node includes selecting one of the second initial processing layout determined for the second node or the third initial processing layout determined for the third node as the first initial processing layout.

[0014] In some embodiments, the second initial processing layout specifies a first degree of parallelism and the third initial processing layout specifies a second degree of parallelism different from the first degree of parallelism, and the selection includes selecting the second initial processing layout if the first degree of parallelism is greater than the second degree of parallelism and selecting the third initial processing layout if the first degree of parallelism is less than the second degree of parallelism.

[0015] In some embodiments, the second initial processing layout and the third initial processing layout each specify parallel processing layouts having the same or different degrees of parallelism, the first edge represents a data flow of a first number of data records and the second edge represents a data flow of a second number of data records, and the selection includes selecting the second initial processing layout if the first number of data records is greater than the second number of data records, and selecting the third initial processing layout if the first number of data records is less than the second number of data records.

[0016] In some embodiments, during the determination, a first processing layout is determined for a first node in the third set of nodes, the first processing layout specifying a first degree of parallelism and a second processing layout for a second node immediately preceding the first node in the graph specifying a second degree of parallelism different from the first degree of parallelism, and the method further includes configuring at least one node of the dataflow graph to perform at least one repartitioning operation.

[0017] In some embodiments, during the determination, a first processing layout is determined for a first node in the third set of nodes, the first processing layout specifying a first degree of parallelism and a second processing layout for a second node immediately preceding the first node in the graph specifying a second degree of parallelism different from the first degree of parallelism, and the method further includes adding a new node to the dataflow graph between the first node and the second node, the new node representing the at least one repartitioning operation.

[0018] In some embodiments, the determining includes determining a first processing layout for a first node in the third set of nodes, the first node representing a first data processing operation, and determining the first processing layout includes determining a degree of parallelism for performing the first data processing operation, and identifying a set of one or more computing devices for performing the first data processing operation in accordance with the determined degree of parallelism.

[0019] In some embodiments, determining the first processing layout includes determining that a single processor will be used to perform the first data processing operation and identifying a computing device for performing the first data processing operation.

[0020] In some embodiments, the processing layout determination is made using at least one rule for selecting among processing layouts having different degrees of parallelism.

[0021] In some embodiments, the method further includes, after determining a processing layout for each node in the dataflow graph, executing the dataflow graph in accordance with the processing layout determined for each node in the dataflow graph.

[0022] In some embodiments, the method further includes receiving a Structured Query Language (SQL) query, generating a query plan from the SQL query, and generating a dataflow graph from the generated query plan.

[0023] In some embodiments, the processing layout of a node representing an operation specifies the degree of parallelism used to perform the operation.

[0024] In some embodiments, determining the processing layout of the nodes in the third set of nodes is done automatically.

[0025] In some embodiments, determining the processing layout of the nodes in the third set of nodes includes, in a forward pass made starting from a node in the first set of nodes, determining, in accordance with the structure of the dataflow graph and using the first set of processing layouts and the one or more layout decision rules, an initial processing layout for each particular node of at least some of the nodes of the third set of nodes, such that for each particular node, a processing layout of a node preceding the particular node in the dataflow graph is selected as an initial processing layout for the particular node during the forward pass, wherein if there are multiple nodes preceding the particular node in the dataflow graph, as indicated by the at least one rule, a parallel processing layout of one node of the multiple preceding nodes is selected during the forward pass as the initial processing layout for the particular node if the processing layout of other nodes of the multiple preceding nodes is sequential, or a processing layout of one node of the multiple preceding nodes used to process the largest number of records is selected as the initial processing layout for the particular node during the forward pass.

[0026] In some embodiments, if multiple nodes preceding a particular node in the dataflow graph have only sequential processing layouts or only parallel processing layouts with the same or different degrees of parallelism, the processing layout of one of the multiple preceding nodes used to process the largest number of records is selected during the forward pass as the initial processing layout.

[0027] In some embodiments, when selecting an initial processing layout for a particular node during a forward pass, the processing layout associated with at least one node is ignored.

[0028] In some embodiments, determining the processing layout of the nodes in the third set of nodes further includes, in a backward pass made starting from a node in the second set of nodes, determining a final processing layout for each particular node of at least some of the nodes of the third set of nodes according to the structure of the dataflow graph, the initial processing layout, and the one or more layout decision rules, such that for each particular node, a parallel processing layout according to one of the initial processing layout of the particular node or the processing layout of the subsequent node of the particular node, as indicated by the at least one rule, is selected as the final processing layout of the particular node during the backward pass if the other of the processing layout of the initial processing layout of the particular node and the processing layout of the subsequent node of the particular node is sequential, or a processing layout used for processing a maximum number of records out of the initial processing layout of the particular node and the processing layout of the subsequent node of the particular node is selected as the final processing layout of the particular node during the backward pass.

[0029] In some embodiments, when the initial processing layout of a particular node and the processing layout of a node subsequent to the particular node both have sequential processing layouts or both have multiple parallel processing layouts but with the same or different degrees of parallelism, the processing layout that is used to process the largest number of records among the initial processing layout of the particular node and the processing layout of the node subsequent to the particular node is selected as the final processing layout of the particular node during the backward pass.

[0030] In some embodiments, when selecting a final processing layout for a particular node during the backward pass, the processing layout associated with at least one node is ignored.

[0031] In some embodiments, the method further includes configuring the dataflow graph to perform a repartitioning operation on data processed by adjacent nodes in the dataflow graph having processing layouts with different degrees of parallelism after performing the forward pass and / or the backward pass.

[0032] In some embodiments, the processing layout of a node representing an operation further specifies one or more computing devices to be used to perform the operation according to a specified degree of parallelism, where the processing layout also specifies the number of computing devices to be used to perform the operation and identifies the specific computing device or devices to be used to perform the operation.

[0033] In some embodiments, the third set of nodes includes a first node, the first node representing a first data processing operation, and after performing the forward pass and / or the reverse pass, the execution includes executing the dataflow graph in accordance with a processing layout determined for each node in the dataflow graph, and performing a repartitioning operation on data processed by adjacent nodes in the dataflow graph having processing layouts with different degrees of parallelism; identifying a set of one or more computing devices based on the initial or final processing layout determined for the first node of the third set of nodes; and performing the first data processing operation using the identified set of computing devices in accordance with the degree of parallelism specified by the initial or final processing layout determined for the first node of the third set of nodes.

[0034] In some embodiments, the method further includes using the at least one computer hardware processor to add a new node to the dataflow graph between adjacent nodes, the new node representing a repartitioning operation such that when a data record is processed according to the dataflow graph having the added new node, partitioning of the data record occurs according to the repartitioning operation during processing of the data record by the adjacent nodes.

[0035] In some embodiments, the method further includes configuring, using at least one computer hardware processor, one of adjacent nodes in the dataflow graph having a processing layout with different degrees of parallelism to perform a repartitioning operation, such that as a data record is processed according to the dataflow graph having the configured one of the adjacent nodes, partitioning of the data record is performed according to the repartitioning operation during processing of the data record by the adjacent node.

[0036] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method according to any one of the preceding embodiments.

[0037] Some embodiments provide a data processing system including at least one computer hardware processor and at least one non-transitory computer readable storage medium storing processor executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method according to any one of the preceding embodiments.

[0038] Some embodiments include a method for generating a data flow graph using at least one computer hardware processor, the data flow graph including a plurality of nodes and a plurality of edges connecting the plurality of nodes, the plurality of edges representing a flow of data between the plurality of nodes, the plurality of nodes comprising: a first set of one or more nodes, each node in the first set of nodes representing a respective input data set in a set of one or more input data sets; and a second set of one or more nodes, each node in the second set of nodes representing at least one respective data processing operation; obtaining a first set of one or more processing layouts for a set of input data sets, the first set of processing layouts including processing layouts having different degrees of parallelism; and determining a processing layout for nodes in a second set of nodes using the first set of processing layouts, one or more layout decision rules including at least one rule for selecting among the processing layouts having different degrees of parallelism, and information indicating that data generated by at least one node in the second set of nodes is not used by any node in the dataflow graph downstream from the at least one node.

[0039] Some embodiments, when executed by at least one computer hardware processor, include obtaining information specifying a dataflow graph, the dataflow graph including a plurality of nodes and a plurality of edges connecting the plurality of nodes, the plurality of edges representing a flow of data between the plurality of nodes, the plurality of nodes including a first set of one or more nodes, each node in the first set of nodes representing a respective input dataset in a set of one or more input datasets, and a second set of one or more nodes, each node in the second set of nodes representing at least one respective data processing operation; and determining a processing layout for nodes in the second set of nodes using the first set of processing layouts, one or more layout decision rules including at least one rule for selecting among the processing layouts having different degrees of parallelism, and information indicating that data generated by at least one node in the second set of nodes is not used by any node in the dataflow graph downstream from the at least one node.

[0040] Some embodiments include at least one computer hardware processor and a method, when executed by the at least one computer hardware processor, for obtaining information specifying a data flow graph, the data flow graph including a plurality of nodes and a plurality of edges connecting the plurality of nodes, the plurality of edges representing a flow of data between the plurality of nodes, the plurality of nodes including a first set of one or more nodes, each node in the first set of nodes representing a respective input data set in a set of one or more input data sets, and a second set of one or more nodes, each node in the second set of nodes representing at least one respective data processing operation; and at least one non-transitory computer-readable storage medium storing processor-executable instructions to cause at least one computer hardware processor to perform a method comprising: obtaining a first set of one or more processing layouts to be used in a data flow graph, the first set of processing layouts including processing layouts having different degrees of parallelism; and determining a processing layout for nodes in the second set of nodes using the first set of processing layouts, one or more layout decision rules including at least one rule for selecting among the processing layouts having different degrees of parallelism, and information indicating that data generated by at least one node in the second set of nodes is not used by any node in a data flow graph downstream from the at least one node.

[0041] Some embodiments provide a method that includes using at least one computer hardware processor to obtain information identifying at least one node whose results are not used by a data processing operation represented by a node in the dataflow graph downstream from at least one node in a dataflow graph representing at least one respective data processing operation, and using the obtained information to determine a processing layout for at least some of the nodes in the dataflow graph.

[0042] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause at least one computer hardware processor to perform a method including obtaining information identifying at least one node whose results are not used by a data processing operation represented by a node in the dataflow graph downstream from at least one node in the dataflow graph representing at least one respective data processing operation, and using the obtained information to determine a processing layout of at least some of the nodes in the dataflow graph.

[0043] Some embodiments provide a data processing system including at least one computer hardware processor; and at least one non-transitory computer readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method including: obtaining information identifying at least one node whose results are not used by a data processing operation represented by a node in the dataflow graph downstream from at least one node in a dataflow graph representing at least one respective data processing operation; and using the obtained information to determine a processing layout of at least some of the nodes in the dataflow graph.

[0044] Various aspects and embodiments are described with reference to the following drawings, which it should be understood are not necessarily drawn to scale, and items that appear in more than one drawing are designated with the same or similar reference numerals in all the drawings in which they appear. [Brief description of the drawings]

[0045] [Figure 1A] FIG. 1 is an illustrative dataflow graph in which each node is associated with the same processing layout. [Figure 1B] FIG. 1 is an illustrative data flow graph having different processing layouts for nodes in a first set of nodes and no processing layout determined for nodes in a second set of nodes, in accordance with some embodiments of the techniques described herein. [Figure 1C] FIG. 1C is an illustrative data flow graph diagram of FIG. 1B showing a processing layout determined for nodes in the second set of nodes and insertion nodes associated with respective repartitioning operations in accordance with some embodiments of the techniques described herein. [Diagram 2] 1 is a flowchart of an illustrative process for determining a processing configuration of a dataflow graph, at least in part by determining a processing layout of the nodes of the dataflow graph, in accordance with some embodiments of the techniques described herein. [Figure 3A-3D] 1 illustrates the determination of a processing layout for nodes of an illustrative dataflow graph using one or more processing layout decision rules, according to some embodiments of the techniques described herein. [Figure 4A-4C] 13 illustrates the determination of a processing layout for nodes of another illustrative dataflow graph using one or more layout decision rules, according to some embodiments of the techniques described herein. [Figure 5A-5D]13 illustrates the determination of a processing layout for nodes of yet another illustrative dataflow graph using one or more layout decision rules, according to some embodiments of the techniques described herein. [Figure 6] FIG. 1 is a block diagram of an illustrative computing environment in which some embodiments of the techniques described herein can operate. [Figure 7] 7 illustrates not propagating the processing layout of "unused" nodes in illustrative dataflow graph 700, in accordance with some embodiments of the techniques described herein. [Figure 8A] 1 shows an illustrative SQL query that is processed as a dataflow graph in accordance with some embodiments of the techniques described herein. [Figure 8B] 8B shows an illustrative dataflow graph 800 generated from the SQL query shown in FIG. 8A, in accordance with some embodiments of the techniques described herein. [Figure 8C] 8C illustrates a processing layout determined for nodes in the illustrative dataflow graph of FIG. 8B during a forward layout propagation pass, according to some embodiments of the techniques described herein. [Figure 8D] 8C illustrates a process layout determined for nodes in the dataflow graph of FIG. 8B during a backward layout propagation pass, according to some embodiments of the techniques described herein. [Figure 8E] 8 shows an illustrative dataflow graph 830 generated in part using an SQL query 831 in accordance with some embodiments of the techniques described herein. [Figure 8F] 8 illustrates a processing layout determined for nodes in a dataflow graph 830, according to some embodiments of the techniques described herein. [Figure 8G] 8F shows a dataflow graph 840 generated from the dataflow graph 835 shown in FIG. 8F after a repartitioning step to insert nodes for performing a repartitioning operation according to some embodiments of the techniques described herein. [Figure 8H]8 shows an illustrative dataflow graph 845 generated in part using an SQL query 851 in accordance with some embodiments of the techniques described herein. [Figure 8I] 8 illustrates a processing layout determined for nodes in a dataflow graph 845 according to some embodiments of the techniques described herein. [Figure 8J] 8H shows a dataflow graph 855 generated from the dataflow graph 845 shown in FIG. 8H after a repartitioning step to insert nodes for performing a repartitioning operation according to some embodiments of the techniques described herein. [Figure 9A] 1 shows an illustrative SQL query that is processed as a dataflow graph in accordance with some embodiments of the techniques described herein. [Figure 9B] 9B illustrates a dataflow graph 910 generated from the SQL query shown in FIG. 9A, according to some embodiments of the techniques described herein. [Figure 9C] 9C illustrates a processing layout determined for nodes in the illustrative dataflow graph of FIG. 9B in accordance with some embodiments of the techniques described herein. [Figure 9D] 9C according to some embodiments of the techniques described herein. [Figure 10] A comparison is shown between dataflow graph 855 and dataflow graph 830, illustrating that dataflow graph 930 can be executed using a micrograph server in accordance with some embodiments of the techniques described herein. [Figure 11] FIG. 1 is a block diagram of an illustrative computing system environment that can be used to implement some embodiments of the technology described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] Aspects of the technology described herein relate to increasing the speed and throughput of data processing systems by improving upon conventional techniques for performing data processing operations using dataflow graphs.

[0047] As discussed above, a node of a dataflow graph may represent each data processing operation that may be applied to data accessed from one or more input data sets. Before a data processing operation can be applied to data, a processing layout for performing the data processing operation needs to be determined. The processing layout may specify how many computing devices are used to perform the data processing operation, and may identify the specific computing devices that are used. Thus, before a data processing system can process data using a dataflow graph, the processing layout of the nodes in the dataflow graph needs to be determined.

[0048] Some conventional techniques for automatically determining the processing layout of nodes in a dataflow graph involve assigning a processing layout to each node of the graph such that all processing layouts have the same degree of parallelism. For example, each node of the graph may be assigned an N-ary processing layout that specifies that each of the data processing operations represented by the nodes of the dataflow graph is performed using N computing devices, where N is an integer equal to or greater than 1. Different data processing operations may be performed by different computing device groups, but each such group must have the same number of computing devices (i.e., N devices). As a result, conventional techniques do not allow one node of a dataflow graph to have an N-ary (N>1) processing layout and another node to have an M-ary (N≠M>1) processing layout.

[0049] The inventors have recognized that if processing layouts with different degrees of parallelism can be assigned to different nodes of a dataflow graph, the data processing system may process data faster and more efficiently. Providing different degrees of parallelism to different data processing operations represented by a dataflow graph may significantly increase the speed and throughput of a data processing system using the dataflow graph. As an example, consider a situation in which different data sets accessed by a dataflow graph are stored with different degrees of parallelism. For example, one input data set ("A") may be a file stored in one location, another input data set ("B") may be stored in four different locations using a distributed file system (e.g., the Hadoop distributed file system), and an output data set ("C") may be stored in three different locations using a distributed database system. Reading data from input dataset A using a sequential processing layout, reading data from input dataset B using a 4-way parallel processing layout, and writing data to output dataset C using a 3-way parallel processing layout may be more efficient than performing all of these data processing operations using processing layouts with the same degree of parallelism, because using a processing layout with a degree of parallelism that matches the degree of parallelism of the input and output datasets increases the speed of accessing and subsequently processing the data contained therein. In addition, some datasets may be accessed (e.g., read and / or written) using only a specified degree of parallelism. Different datasets may require different degrees of parallelism. Such datasets may not be accessed using the same data flow graph without the techniques described herein.

[0050] For example, consider an illustrative dataflow graph 100, shown in FIG. 1A, which includes input nodes 102a, 102b, 102c, nodes 104, 110, and 111 representing respective filtering operations, node 106 representing a sorting operation, nodes 108 and 112 representing respective join operations, and output nodes 114a and 114b. As shown in FIG. 1A, data from an input dataset represented by node 102a is filtered, sorted, and combined with a filtered version of the data from the input dataset represented by node 102b before being written to an output dataset represented by output node 114a. Data from the input dataset represented by node 102b is also filtered and combined with data from an input dataset represented by node 102c before being written to an output dataset represented by node 114b. As shown in FIG. 1A, application of conventional techniques for automatically determining a processing layout results in the same parallel processing layout PL1 being assigned to each node of dataflow graph 100. On the other hand, as shown in FIG. 1B, if the input and output data sets are stored with different degrees of parallelism, different processing layouts may need to be assigned to different nodes of graph 100.

[0051] Some of the embodiments described herein address all of the above-mentioned problems identified by the inventors with conventional techniques for performing data processing operations using dataflow graphs. However, not all of the embodiments described below address each and every one of these problems, and some embodiments may not address any of them. Thus, it is to be understood that embodiments of the technology described herein are not limited to addressing all or any of the above-mentioned problems of conventional techniques for performing data processing operations using dataflow graphs.

[0052] Some embodiments of the techniques described herein are directed to techniques for automatically determining a processing layout for performing data processing operations represented by one or more nodes of a dataflow graph. Unlike conventional techniques for performing computations using dataflow graphs, the processing layouts determined for different nodes need not be the same, and data processing operations represented by different nodes of the graph may be performed using different processing layouts, particularly processing layouts having different degrees of parallelism.

[0053] As used herein, the processing layout of a node of a dataflow graph refers to the processing layout used to perform the data processing operation represented by that node. For example, the processing layout of an input node of a dataflow graph refers to the processing layout used to read data from an input data set represented by that input node. As another example, the processing layout of an output node of a dataflow graph refers to the processing layout used to write data to an output data set represented by that output node. As yet another example, the processing layout of a node representing a data processing operation (e.g., a filtering operation, a join operation, a rollup operation, etc.) refers to the processing layout for performing that data processing operation.

[0054] In some embodiments, the processing layout of a node representing a data processing operation may indicate the degree of parallelism used to perform the operation. Additionally, the processing layout of a node may also specify one or more computing devices to be used to perform the operation according to the specified degree of parallelism. For example, the processing layout of a node may be a sequential processing layout with a single degree of parallelism (i.e., sequential processing rather than parallel) and may optionally specify the computing devices (e.g., processors, servers, laptops, etc.) to use to perform the data processing operation represented by that node. As another example, the processing layout of a node may be an N-ary (N≧1) parallel processing layout with N degrees of parallelism and may optionally specify the N computing devices to use to perform the data processing operation represented by that node. As can be appreciated from the above, in some embodiments, in addition to specifying the degree of parallelism of a node, the processing layout of a node may specify one or more computing devices and / or one or more processes running on one or more computing devices to use to perform the data processing operation represented by that node.

[0055] In some embodiments, determining a processing layout of nodes in a dataflow graph may include: (A) obtaining information specifying the dataflow graph; (B) obtaining a processing layout of input nodes in the dataflow graph; (C) obtaining a processing layout of output nodes in the dataflow graph; and (D) determining a processing layout of one or more other nodes (i.e., nodes that are not input or output nodes) based on the processing layout of the input nodes, the processing layout of the output nodes, and one or more layout decision rules. Dataflow graph nodes other than input and output nodes may be referred to herein as "intermediate" nodes. Examples of layout decision rules are described herein, such as with reference to FIG. 2.

[0056] In some embodiments, at least two of the processing layouts obtained for the input and output nodes of a dataflow graph may have different degrees of parallelism. For example, the processing layouts obtained for two different input nodes may have different degrees of parallelism. As another example, the processing layouts obtained for two different output nodes may have different degrees of parallelism. As yet another example, the processing layout obtained for the input node may have a different degree of parallelism than the processing layout obtained for the output node. Nevertheless, the techniques described herein can be used to automatically determine the processing layout of nodes of a graph in which at least two of the processing layouts obtained for the input and output nodes have different degrees of parallelism. As an illustrative example, the techniques described herein can be applied to determining the processing layout of the dataflow graph 100 shown in FIG. 1B. The input and output nodes of dataflow graph 100 are associated with three different processing layouts (i.e., a sequential layout SL1, a parallel layout PL1, and a parallel layout PL2), which can be used in conjunction with the layout decision rules described herein to automatically determine the processing layout of nodes 104, 106, 108, 110, 111, and 112, as shown in FIG. 1C.

[0057] In some embodiments, the processing layout of one or more intermediate nodes of a dataflow graph may be determined by (1) determining an initial processing layout of at least some (e.g., all) of the intermediate nodes by performing a forward pass (from the input nodes toward the output nodes) and later (2) determining a final processing layout of the intermediate nodes by performing a backward pass (from the output nodes toward the input nodes). During the forward pass, the initial processing layout may be determined based on the processing layout assigned to the input nodes and one or more layout decision rules described herein. For example, the initial processing layout of nodes 104, 106, 108, 110, 111, and 112 may be determined based on the processing layout assigned to nodes 102a, 102b, and 102c. During the backward pass, the final processing layout of the intermediate nodes may be determined based on the processing layout assigned to the output nodes, the initial processing layout assigned to at least some of the intermediate nodes, and one or more layout decision rules. For example, the final processing layouts of nodes 104, 106, 108, 110, 111, and 112 may be determined based on the initial processing layouts determined for these nodes during the forward pass, the processing layouts assigned to the output nodes 114a and 114b, and one or more layout decision rules. The use of forward and backward passes in determining the final processing layouts of intermediate nodes (data processing nodes between input and output nodes) allows the layout decision to be automated in a particularly fast, accurate, and reliable manner. This is particularly evident when considering a large number of data flow paths (edges) in the data flow graph. For example, the forward and backward passes ensure that no data flow paths are forgotten when determining the final processing layout of the nodes of the data flow graph. Such forgotten paths may include nodes having processing layouts that are used for processing a large number of data records, in particular, so that the final processing layouts of other nodes of the graph may differ in a computationally efficient manner when incorporating these nodes of these paths in the process of determining the layout of the nodes of the graph.

[0058] In some embodiments, after the processing layout of the nodes in the dataflow graph has been determined (e.g., after performing a forward and reverse pass), the dataflow graph may be configured to perform a repartitioning operation on data that is processed using a processing layout with a particular degree of parallelism and then processed using a processing layout with a different degree of parallelism. In some embodiments, the dataflow graph may be configured to perform a repartitioning operation on data that flows between adjacent nodes in the graph having processing layouts with different degrees of parallelism. In this manner, data that has been processed using one processing layout (using N computing devices, with N≧1) may be adapted for subsequent processing using another processing layout (using M≠N computing devices, with M≧1).

[0059] For example, as illustrated in FIG. 1C, adjacent nodes 102a and 104 have processing layouts with different degrees of parallelism, such that one of the layouts is sequential (processing is performed using one computing device) and the other is parallel (processing is performed using multiple computing devices). In this example, the dataflow graph may be configured to partition the data such that data flowing from node 102a to node 104 may be processed using a single computing device according to a processing layout determined for node 102a (SL1) and then processed using multiple computing devices according to a processing layout determined for node 104 (PL1). To this end, the dataflow graph may be configured to perform a repartitioning operation (e.g., a partition-by-key operation) that increases the degree of parallelism. Also, in the same example, adjacent nodes 102b and 111 may have different processing layouts, and adjacent nodes 112 and 114 may have different processing layouts, and dataflow graph 100 may be configured to perform a repartitioning operation on data flowing from node 102 to node 111, and a repartitioning operation (e.g., an aggregation operation) on data flowing from node 112 to node 114.

[0060] In some embodiments, a dataflow graph may be configured to perform a repartitioning operation (at execution time of the graph) by extending the graph with nodes that represent the repartitioning operation. When the graph is executed, software configured to perform the repartitioning operation may be executed. For example, as shown in FIG. 1C, the dataflow graph may be extended with nodes 130, 132, and 134, each associated with a respective repartitioning operation. In other embodiments, one or more existing nodes of the dataflow graph may be configured to perform the repartitioning operation, and no new nodes are added.

[0061] In some embodiments, if a node (node ​​"A") of a dataflow graph is associated with a processing layout that has a degree of parallelism that is higher than the degree of parallelism of a subsequent adjacent node (node ​​"B") in the graph, the dataflow graph may be configured to perform a repartitioning operation on the data after the data has been processed in accordance with the data processing operation represented by node "A" and before the data is processed in accordance with the data processing operation represented by node "B". In this case, the repartitioning operation may reduce the degree of parallelism and may be used to, for example, reduce the degree of parallelism. 1 or merge operation 2 For example, as shown in Figure 1C, the parallel processing layout (PL2) of node 112 has a higher degree of parallelism than the sequential processing layout (SL1) of its subsequent adjacent node 114b. In this example, a node 134 associated with an aggregation operation is added between nodes 112 and 114b.

[0062] In some embodiments, if a node (node ​​"A") of a dataflow graph is associated with a processing layout that has a degree of parallelism that is lower than the degree of parallelism of a subsequent adjacent node (node ​​"B") in the graph, the dataflow graph may be configured to perform a repartitioning operation on the data after the data has been processed in accordance with the data processing operation represented by node "A" and before the data is processed in accordance with the data processing operation represented by node "B". In this case, the repartitioning operation may increase the degree of parallelism and may be used to, for example, perform a partition-by-key operation. 3 , round-robin partition operation, partition-by-range operation 41C, the sequential processing layout (SL1) of node 102a has a lower degree of parallelism than the parallel processing layout (PL1) of the subsequent adjacent node 114b. Similarly, in this example, the parallel processing layout (PL1) of node 102b has a lower degree of parallelism than the parallel processing layout (PL2) of node 111. In this example, a node 130 representing a partition-by-key operation is added between node 102a and node 104, and a node 132 representing a round-robin partition operation is followed by an aggregation operation to achieve the desired reduction in the degree of parallelism. 1 An aggregation operation performed on multiple sets of data records may combine the multiple sets of data records into a single set of data records, but does not necessarily preserve the sortability of the data records in the single set. 2 A merge operation performed on multiple sets of data records may combine the multiple sets of data records into a single set of data records while maintaining the sortability of the data records in the single set. 3 In a partition-by-key operation, data records that have one or more identical values ​​for one or more of the same fields (e.g., in the same one or more columns) are assigned to the same partition. 4 In a partition-by-range operation, different partitions are associated with different non-overlapping ranges of values, and data records having a field value within a range are assigned to the partition associated with that range.

[0063] In some embodiments, if a node in a dataflow graph (node ​​"A") is associated with a processing layout that has the same degree of parallelism as the degree of parallelism of its subsequent adjacent node in the graph (node ​​"B"), then a repartitioning operation is not required.

[0064] In some embodiments, the processing layout of intermediate nodes of a dataflow graph may be determined based on processing layouts assigned to input and output nodes of the graph, although the techniques described herein are not limited to determining the layout of intermediate nodes from the layout of input and output nodes. In some embodiments, for example, processing layouts may be obtained for any subset of one or more nodes of a dataflow graph, and the processing layout of any other nodes of the dataflow graph may be determined based on these obtained processing layouts, the structure of the dataflow graph, and one or more layout decision rules.

[0065] Some embodiments of the techniques described herein may be applied to managing database queries, such as Structured Query Language (SQL) queries, by a data processing system. In some embodiments, the data processing system may execute the received database query by (1) receiving a database query (e.g., an SQL query), (2) generating a query plan for executing the SQL query (e.g., a plan indicating database operations that may occur when the database query is executed), (3) generating a dataflow graph from the query plan, and (4) at least in part executing the dataflow graph. Such embodiments are described in further detail in U.S. Patent No. 9,116,955, issued Aug. 25, 2015, entitled “MANAGING DATA QUERIES,” which is incorporated herein by reference in its entirety. The specification of U.S. Patent No. 9,116,955 is a continuation of the specification of U.S. patent application Ser. No. 13 / 098,823, entitled "MANAGING DATA QUERIES," filed May 2, 2011, the contents of which are incorporated herein by reference in their entirety.

[0066] In some embodiments, the techniques described herein may be used to automatically determine a processing layout for one or more nodes of a dataflow graph that was automatically generated from a database query (e.g., an SQL query).

[0067] In some embodiments, a data processing system may (1) receive a database query (e.g., SQL) query, (2) convert the received database query into computer code including computer code portions that, when executed, execute the database query, and (3) automatically determine a process layout for executing each computer code portion. In some embodiments, the process layout for executing the computer code portions may be determined using information indicative of an execution order of the computer code portions. For example, in some embodiments, each computer code portion may be associated with a respective node in a dataflow graph, and the structure of the graph (e.g., as embodied in the connections between the nodes) may be used to assign a process layout to the node and, relative to the computer code portions associated with the node. However, it should be appreciated that in some embodiments, the process layout for executing the computer code portions may be determined without using a dataflow graph, since information indicative of an execution order of the computer code portions is not limited to being encoded in a dataflow graph.

[0068] It should be understood that the embodiments described herein may be implemented in any of numerous ways. For illustrative purposes only, examples of specific implementations are provided below. It should be understood that these embodiments and provided features / functions may be used individually, all together, or in any combination of two or more (although aspects of the technology described herein are not limited in this respect).

[0069] 2 is a flowchart of an illustrative process 200 for determining a processing configuration of a dataflow graph by, at least in part, assigning a processing layout to nodes of the dataflow graph in accordance with some embodiments of the techniques described herein. Process 200 may be performed by any suitable system and / or computing device, and may be performed by, for example, a data processing system 602 as described herein, such as with reference to FIG. 6. After describing process 200, certain aspects of process 200 are illustrated with reference to the examples shown in FIGS. 3A-3D, 4A-4C, and 5A-5D. The examples of FIGS. 3A-3D are described in detail after describing process 200, but the examples are also referred to throughout the description of process 200 for clarity of explanation.

[0070] Process 200 begins with Act 202, where information specifying a dataflow graph may be accessed. As described herein, a dataflow graph may include a plurality of nodes, including (a) one or more input nodes representing one or more respective input data sets, (b) one or more output nodes representing one or more respective output data sets, and / or (c) one or more nodes representing data processing operations that may be performed on data. Directed links or edges between nodes of the dataflow graph represent data flows between the nodes. Thus, in Act 202, information specifying the nodes (including any of the above types of nodes) and links of the dataflow graph may be accessed. This information may be accessed from any suitable source, any suitable data structure, and may be in any suitable format (although aspects of the technology described herein are not limited in this respect). For example, with reference to the example shown in FIGS. 3A-D, in Act 202, information specifying a dataflow graph 300 may be accessed. In some embodiments, described in more detail below, the data flow graph from which information is accessed in act 202 may be automatically generated, and may be automatically generated from, for example, a Structured Query Language (SQL) query.

[0071] In some embodiments, a dataflow graph may have a large number of nodes. For example, a dataflow graph may include at least 50 nodes, at least 100 nodes, at least 1000 nodes, at least 5000 nodes, 50-500 nodes, 100-1000 nodes, 100-5000 nodes, or any other suitable range within these ranges. In such situations, manually assigning processing layouts to the various nodes may be impractical, if not simply impossible. However, processing layouts may be practically assigned in accordance with the techniques described herein, including with reference to FIG. 2.

[0072] Then, in Act 204, a processing layout may be obtained for each input node (i.e., each node representing an input dataset) of the dataflow graph accessed in Act 202. For example, with reference to the example of FIGS. 3A-3D, in Act 204, a processing layout may be obtained for input nodes 302 and 304. In some embodiments, the processing layout of the input node specifies a degree of parallelism (e.g., sequential, 2-way parallel, 3-way parallel, ..., N-way parallel, where N is any suitable integer number) for reading data from the input dataset represented by the input node. In some embodiments, the processing layout of the input node identifies a set of one or more computing devices (e.g., a set of one or more processors, servers, and / or other suitable devices) to use to read data from the input dataset.

[0073] The processing layout of the input nodes may be obtained in any suitable manner. In some embodiments, the processing layout of the input nodes may be determined prior to the start of execution of process 200, and during act 204, the previously determined processing layout may be accessed. In other embodiments, the processing layout of the input nodes may be dynamically determined during execution of process 200. In some embodiments, the processing layout of the inputs may be partially determined prior to the start of execution of process 200, and the unknown information is dynamically determined during execution of process 200. For example, the processing layout of the input nodes may be known to be sequential or parallel prior to execution of process 200, but the particular computing devices used to perform the input operations (e.g., reading data from one or more sources) may be determined during execution of process 200. As another example, it may be known prior to execution of process 200 that a parallel processing layout will be assigned to an input node, but the degree of parallelism may be determined during run-time.

[0074] Whether the processing layout of the input nodes is determined prior to or during execution of process 200, the determination may be made in any suitable manner. For example, in some embodiments, the processing layout of the input nodes may be manually specified by a user using a user interface (e.g., a graphical user interface, a configuration file, etc.). As another example, in some embodiments, the processing layout of the input nodes may be automatically determined by a data processing system. For example, the data processing system may automatically determine the processing layout of the input nodes based on how the input data set represented by the input nodes is stored. For example, if the input data set is stored on multiple devices (e.g., four servers, using a Hadoop cluster, etc.), the data processing system executing process 200 may determine that a parallel processing layout (e.g., a four-way parallel processing layout, the number of nodes in the Hadoop cluster) is used to read data records from the input data set.

[0075] Then, in Act 206, a processing layout may be obtained for each output node (i.e., each node representing an output dataset) of the dataflow graph accessed in Act 202. For example, with reference to the example of FIGS. 3A-3D, in Act 206, a processing layout of output node 314 may be obtained. In some embodiments, the processing layout of the output node specifies a degree of parallelism (e.g., sequential, 2-way parallel, 3-way parallel, ..., N-way parallel, with any suitable integer N) for writing data to the output dataset represented by the output node. In some embodiments, the processing layout of the output node identifies a set of one or more computing devices (e.g., a set of one or more processors, servers, and / or other suitable devices) to use to write data to the output dataset.

[0076] The processing layout of the output nodes may be obtained in any suitable manner. In some embodiments, the processing layout of the output nodes may be determined prior to the start of execution of process 200, and during act 206, the previously determined processing layout may be accessed. In other embodiments, the processing layout of the output nodes may be dynamically determined during execution of process 200. In some embodiments, the processing layout of the output nodes may be partially determined prior to the start of execution of process 200, and the unknown information is dynamically determined during execution of process 200. For example, the processing layout of the output nodes may be known to be sequential or parallel prior to execution of process 200, but the particular computing devices used to perform the output operations (e.g., writing data to one or more output data sets) may be determined during execution of process 200. As another example, it may be known prior to execution of process 200 that a parallel processing layout will be assigned to an output node, but the degree of parallelism may be determined during run-time.

[0077] Whether the processing layout of the output nodes is determined prior to or during execution of process 200, it may be determined in any suitable manner, including any of the manners described above for determining the processing layout of the input nodes. For example, the processing layout of the output nodes may be manually specified by a user using a user interface, or may be automatically determined by a data processing system (e.g., based on how the output data sets represented by the output nodes are stored).

[0078] Process 200 then proceeds to act 208 where a processing layout is determined for nodes in the dataflow graph other than the input and output nodes for which a processing layout was obtained in acts 204 and 206. In some embodiments, the processing layout of the intermediate node specifies a degree of parallelism (e.g., sequential, 2-way parallel, 3-way parallel, ..., N-way parallel, where N is any suitable integer) for performing the data processing operations represented by the intermediate node. In some embodiments, the processing layout of the intermediate node identifies a set of one or more computing devices (e.g., a set of one or more processors, servers, and / or other suitable devices) to use to perform the data processing operations.

[0079] In some embodiments, the processing layout of the intermediate nodes may be determined, at least in part, by using the processing layouts of the input and output nodes (obtained in Acts 204 and 206). For example, with reference to the example of Figures 3A-3D, in Act 208, the processing layout of the intermediate nodes 306, 308, 310, and 312 may be determined using the processing layouts of the input nodes 302 and 304, and the output node 314. In some embodiments, the processing layout of the intermediate nodes may be determined further based on the structure of the dataflow graph and one or more layout decision rules.

[0080] In some embodiments, the layout decision rules may specify how the processing layout of a node of the dataflow graph may be determined based on the processing layout of one or more other nodes of the dataflow graph. For example, in some embodiments, the layout decision rules may specify how the processing layout of a particular node that is not associated with any processing layout may be determined based on the processing layout of one or more other nodes adjacent to the particular node in the graph. As an illustrative example, referring to the example of FIG. 3A , the layout processing rules may specify how to determine the processing layout of the intermediate node 306 based on the processing layout of the input node 302. As another illustrative example, once the processing layouts of the nodes 308 and 310 are determined, the layout processing rules may specify how to determine the processing layout of the node 312 based on the processing layouts determined for the nodes 308 and 310.

[0081] As another example, in some embodiments, a layout decision rule may specify how to determine a processing layout for a particular node already associated with a particular processing layout based on the particular processing layout and the processing layouts of one or more other nodes adjacent to the particular node in the graph. As an illustrative example, with reference to the example of Figure 3C, the processing layout of node 312 may be determined based on an initial processing layout determined for node 312 (layout PL1) and a processing layout determined for output node 314 (layout SL2).

[0082] Non-limiting, illustrative examples of specific layout decision rules are described below. It should be appreciated that in some embodiments, one or more other layout decision rules may be used in addition to or in place of the example layout decision rules described herein. It should also be appreciated that any suitable combination of one or more of the example layout rules described herein may be used in some embodiments. The layout decision rules described herein may be implemented in any suitable manner (e.g., using software code, one or more configuration parameters, etc.) (aspects of the technology described herein are not limited in this respect).

[0083] In some embodiments, according to an example layout decision rule, when determining a processing layout for a particular node that is not yet associated with a processing layout, if the particular node has a neighboring node (e.g., a node immediately preceding the particular node in the dataflow graph, or a node immediately following the particular node in the dataflow graph) with an associated processing layout, the layout of the neighboring node may be determined as the processing layout of the particular node. In this manner, the processing layout of the neighboring node may be "copied" to the particular node. As an illustrative example, in the example of FIG. 3A, the processing layout of node 306 may be determined to be the processing layout of the neighboring node 302 that precedes it. Then, the processing layout of node 308 may be determined to be the processing layout of the node 306 that precedes it. As can be appreciated from this example, the layout decision rule may be applied iteratively to propagate the layout of an input node (e.g., node 302) to one or more other nodes (e.g., nodes 306, 308, and 312).

[0084] In some embodiments, according to another example layout decision rule, when determining the processing layout of a particular node that is not yet associated with a particular processing layout, if the particular node has multiple neighboring nodes (e.g., multiple preceding neighboring nodes or multiple succeeding neighboring nodes) with associated processing layouts, the processing layout of the particular node may be selected from among the layouts of its neighboring nodes. For example, for the dataflow graph of FIG. 3A, assuming that the layouts of nodes 308 and 310 have been determined but the layout of node 312 has not yet been determined, the layout of node 312 may be selected to be one of the layouts determined for nodes 308 and 310.

[0085] In some embodiments, according to another example layout decision rule, when determining a processing layout for a particular node already associated with a particular processing layout, if the particular node has one or more neighboring nodes associated with respective processing layouts, the layout of the particular node may be determined by selecting from among the particular processing layout already associated with the node and the processing layouts of its neighboring nodes. For example, as shown in FIG. 3C, node 312 is associated with an initial processing layout (PL1) and has neighboring node 314 associated with another processing layout (SL2). In this example, one of these two layouts (i.e., PL1 and SL2) may be selected as the updated (e.g., final) processing layout for node 312.

[0086] As can be appreciated from the above, in some embodiments, applying a certain layout determination rule involves selecting a processing layout from among two or more processing layouts. This may be done in a number of ways. For example, when selecting a processing layout for a node from a group consisting of two or more processing layouts, a processing layout having the greatest degree of parallelism may be selected. For example, when selecting whether the processing layout of a node is an N-way parallel processing layout (e.g., a 10-way parallel layout) or an M-way (M < N) parallel processing layout (e.g., a 5-way parallel layout), the N-way parallel processing layout may be selected. As another example, when selecting a processing layout for a node from a parallel processing layout and a sequential processing layout, the parallel processing layout may be selected. As an example for illustration purposes, referring to FIG. 3B, node 312 may be assigned an initial sequential processing layout (SL1) (e.g., as a result of propagating that layout from input node 302), and node 310 may be assigned a parallel layout PL1 (e.g., as a result of propagating that layout from input node 304). Then, since layout PL1 clearly has a greater degree of parallelism between PL1 and SL1, the processing layout of node 312 may be updated to be the parallel layout PL1.

[0087] As another example of how to select a processing layout from among two or more processing layouts, when selecting a parallel processing layout for a node from parallel processing layouts having the same or different degrees of parallelism, the processing layout used to process a larger number of records may be selected. For example, when selecting a processing layout for a node in a dataflow graph from a 4-way layout PL1 assigned to a first preceding neighboring node of the node and used to process 10 million data records, and a 4-way layout PL2 assigned to a second preceding neighboring node of the node and used to process 10,000 data records, layout PL1 may be selected for the node. In this way, a data processing operation (e.g., a join operation) associated with the node may be performed using the same processing layout (e.g., the same computing device) as that used to process the 10 million data records. As a result, when layouts PL1 and PL2 are implemented using non-overlapping sets of computing devices, at most 10,000 data records need to be moved to the computing device used to process the 10 million data records, thereby leading to a more efficient use of computing resources compared to the opposite case. On the other hand, specifically, if layout PL2 is selected, all of the possibly 10 million data records need to be moved to a computing device used to process only 10,000 data records, which is obviously inefficient. Therefore, selecting a layout used to process a larger number of records may help improve the computational performance of the data processing system. An example of this is further described below with reference to Figures 4A-4C.

[0088] When selecting a processing layout for a node in a data flow graph from a ternary layout PL1 assigned to the node's first preceding neighbor node and used to process 10 million data records, and a 6-ary layout PL2 assigned to the node's second preceding neighbor node and used to process 10,000 data records, layout PL1 may be selected for the node. Although the degree of parallelism is reduced, in this manner, the data processing operation (e.g., a join operation) associated with the node may be performed using the same processing layout (e.g., the same computing device) as that used to process the 10 million data records. As a result, in this example as before, when layouts PL1 and PL2 are implemented using a non-overlapping set of computing devices, at most 10,000 data records need to be moved to the computing device used to process the 10 million data records, thereby leading to a more efficient use of computing resources compared to the opposite case. On the other hand, specifically, when layout PL2 is selected, potentially all 10 million data records need to be moved to the computing device used to process only 10,000 data records, which is clearly inefficient. Therefore, selecting a layout that is used to process a larger number of records can help improve the computational performance of the data processing system.

[0089] In some embodiments, according to another example layout decision rule, after the process layouts are determined for the input and output nodes of the dataflow graph, these process layouts are not subsequently changed. In embodiments in which this rule is utilized, after the process layouts of the input and output nodes are obtained in acts 204 and 206, these process layouts are not subsequently changed.

[0090] In some embodiments, according to another example layout decision rule, a sequential processing layout may be assigned to a node that represents a limiting action that, when applied to a group of data records, outputs a certain number of data records (e.g., outputting the data records with the top ten scores after the data records are sorted based on their respective scores).

[0091] In some embodiments, one or more internal nodes of a dataflow graph may be associated with a predefined processing layout. In some embodiments, certain types of nodes may be associated with a predefined processing layout.

[0092] In some embodiments, according to another example layout decision rule, once a process layout is assigned to a particular node in the dataflow graph, instructions may be provided (e.g., by a user using a user interface, such as a graphical user interface or a configuration file) to not propagate the process layout assigned to the particular node to other nodes. For example, in some embodiments, instructions to not propagate the process layout assigned to one or more input nodes and / or one or more output nodes may be provided as part of obtaining the input and / or output process layouts in acts 204 and / or 206.

[0093] One example of this is further described below with reference to Figures 5A-5D.

[0094] Any of the above layout decision rules may be used to determine the processing layout of the intermediate nodes in act 208 of process 200. Although some of the above layout decision rules are "local" because they specify how to determine the processing layout of a particular node based on layouts already assigned to its neighboring nodes, in some embodiments, one or more of these layout decision rules may be applied iteratively to propagate the processing layouts obtained for the input and output processing nodes to the intermediate nodes. This propagation may be done in any suitable manner.

[0095] In some embodiments, the processing layout of the intermediate nodes may be determined in Act 208 by (1) determining an initial processing layout of at least some (e.g., all) of the intermediate nodes by performing a forward pass in Act 208a, and (2) determining a final processing layout of at least some (e.g., all) of the intermediate nodes by performing a backward pass in Act 208b.

[0096] During the forward pass, the processing layout obtained for one or more input nodes may be propagated to intermediate nodes in the dataflow graph using one or more of the layout decision rules described herein. The structure of the dataflow graph may guide the order in which processing layouts are determined for nodes during the forward pass. For example, first, the processing layout for the neighboring nodes of the input node may be determined, then the processing layout for the neighboring nodes of the input node may be determined, and so on until all flows from the input node reach their termination at the output node. As an illustrative example, referring to FIG. 3B, the processing layouts of input nodes 302 and 304 may be propagated during the forward pass using one or more layout decision rules to obtain initial processing layouts for nodes 306, 308, 310, and 312.

[0097] During the backward pass, the processing layout obtained for one or more output nodes may be propagated to intermediate nodes in the dataflow graph using one or more of the layout decision rules described herein. As with the forward pass, the structure of the dataflow graph may guide the order in which processing layouts are determined for nodes during the backward pass. For example, first the processing layout for the output node's neighbors may be determined, then the processing layout for the output node's neighbors' neighbors may be determined, and so on until all edges from the output node reach their termination at the output node. The path followed is reversed during the backward pass and may be the inverse of the path followed in the forward pass. As an illustrative example, with reference to FIG. 3C, the processing layout of output node 314 and the initial processing layouts of nodes 306, 308, 310, and 312 may be used along with one or more layout decision rules during the backward pass to obtain the final processing layouts of nodes 306, 308, 310, and 312. This is described in more detail below with reference to FIGS. 3A-3D.

[0098] After the processing layout for the intermediate node is determined in act 208, the process 200 proceeds to decision block 210 where it is determined whether adjacent nodes of the data flow graph have mismatched layouts. Adjacent nodes "A" and "B" have mismatched layouts if the processing layout determined for node A has a different degree of parallelism than the processing layout determined for node B. For example, if an N-ary (N>1) parallel processing layout is determined for node A and a sequential processing layout is determined for subsequent node B, then these nodes have mismatched layouts (there is an N to 1 transition). As another example, if a sequential processing layout is determined for node A and an M-ary (M>1) parallel processing layout is determined for subsequent node B, then these nodes have mismatched layouts (there is a 1 to M transition). As another example, if an N-ary parallel processing layout is determined for node A and an M-ary parallel processing layout is determined for adjacent node B (M ≠ N), then these nodes have mismatched layouts (there is an N to M transition).

[0099] If at decision block 210 it is determined that there are adjacent node pairs having processing layouts with different degrees of parallelism, the process 200 proceeds to act 212 where the data flow graph may be configured to perform one or more repartitioning operations. A repartitioning operation allows data records that are processed using one number of processors with one processing layout to be transitioned for processing using a different number of processors with another processing layout. Examples of repartitioning operations are described herein and include, for example, repartitioning operations to increase the degree of parallelism in processing data (e.g., partition-by-key operations, round-robin partition operations, partition-by-range operations, and / or other suitable types of partition operations) and repartitioning operations to decrease the degree of parallelism in processing data (e.g., merge operations and aggregation operations). For example, if there is an N to 1 transition between adjacent nodes A and B, the dataflow graph may be configured to perform a repartitioning operation to reduce the degree of parallelism of the data processed according to the operation represented by node A (from N to 1) before the data is processed according to the operation represented by node B. As another example, if there is a 1 to M transition between adjacent nodes A and B, the dataflow graph may be configured to perform a repartitioning operation to increase the degree of parallelism of the data processed according to the operation represented by node A (from 1 to M) before the data is processed according to the operation represented by node B. As yet another example, if there is an N to M transition between adjacent nodes A and B, the dataflow graph may be configured to perform multiple repartitioning operations to change the degree of parallelism (from N to M) for the data processed according to the operation represented by node A before the data is processed by the operation represented by node B.The multiple repartitioning operations may include a first repartitioning operation that reduces the degree of parallelism (e.g., from N to K) and a second repartitioning operation that increases the degree of parallelism (e.g., from K to M, where K is a common divisor of N and M).

[0100] In some embodiments, a dataflow graph may be configured to perform a repartitioning operation by adding a new node that represents the repartitioning operation. Examples of this are shown in Figures 3D, 4C, and 5D described below. In such embodiments, a processing layout may be determined for the node that represents the repartitioning operation. If the repartitioning operation increases the degree of parallelism (e.g., a partition-by-key operation), the processing layout assigned to the node that represents the repartitioning operation may be the processing layout assigned to the preceding node. If the repartitioning operation decreases the degree of parallelism (e.g., a merge or aggregate operation), the processing layout assigned to the node that represents the repartitioning operation may be the processing layout assigned to the subsequent node in the graph. In other embodiments, an existing node of the dataflow graph may be configured to perform a repartitioning operation.

[0101] In some embodiments, a data processing system performing process 200 may be programmed to configure a dataflow graph to perform a particular type of repartitioning operation in certain circumstances. For example, in some embodiments, when a dataflow graph is configured to perform a repartitioning operation to reduce the degree of parallelism and the data is sorted, if the sortability of the data is maintained through the repartitioning, the dataflow graph may be configured to perform a merge operation to reduce the degree of parallelism. Otherwise, an aggregation operation may be used to reduce the degree of parallelism. As another example, in some embodiments, when a dataflow graph is configured to perform a repartitioning operation to increase the degree of parallelism, if a particular partitioning of the data is desired, the dataflow graph may be configured to perform a partition-by-key operation with respect to a particular key or keys. Otherwise, a round-robin partition operation, or another type of partition operation, may be used. As another example, in some embodiments, applying a rollup operation to parallel data may require repartitioning if the data has not already been partitioned with respect to a subset of the rollup keys. In this case, if the rollup is estimated to reduce the amount of data significantly (e.g., by at least a factor of 10), a double bubble rollup may be performed (i.e., first a rollup in the source layout and partitioning scheme, then a second rollup in the destination layout and partitioning scheme).

[0102] On the other hand, if it is determined at decision block 210 that there are no adjacent nodes having processing layouts with different degrees of parallelism, or for adjacent nodes having layouts with different degrees of parallelism, appropriate repartitioning logic has been added to the dataflow graph, then process 200 is complete.

[0103] In some embodiments, after a processing layout has been assigned using process 200, the dataflow graph may be executed in accordance with the assigned layout. In this manner, each of the one or more data processing operations in the dataflow graph is executed in accordance with the processing layout assigned to that data processing operation.

[0104] In some embodiments, process 200 may be applied to an automatically generated dataflow graph. For example, in some embodiments, process 200 may be applied to a dataflow graph that is automatically generated from an SQL query, from information specifying a query provided by another database system, and / or from another dataflow graph.

[0105] In some embodiments, a dataflow graph may be generated from an SQL query by (1) receiving an SQL query, (2) generating a query plan from the received SQL query, and (3) generating a dataflow graph from the query plan. Process 200 may then be applied to the dataflow graph so generated. Each of these three acts (which automatically generate a dataflow graph to which process 200 may be applied) is described in more detail below.

[0106] In some embodiments, the SQL query may be received by a data processing system (e.g., a data processing system executing process 200, such as data processing system 602) as a result of a user providing the SQL query as input to the data processing system. The user may enter the SQL query using a graphical user interface or any other suitable type of interface. In other embodiments, the SQL query may be provided to the data processing system by another computer program. For example, the SQL query may be provided by a computer program configured to cause the data processing system to execute one or more SQL queries (each of which may be specified by a user or may be automatically generated). The SQL query may be of any suitable type and provided in any suitable format (aspects of the technology described herein are not limited in this respect).

[0107] In some embodiments, the received SQL query may be used to generate a query plan. The generated query plan may identify one or more data processing operations to be performed when the SQL query is executed. The generated query plan may further specify the order in which the identified data processing operations are to be performed. Thus, the generated query plan may represent a sequence of data processing operations to be performed to execute the received SQL query. The generated query plan may be generated using any suitable type of query plan generator. Some exemplary techniques for generating query plans are described in U.S. Pat. No. 9,116,955, entitled "Data Query Management," which is incorporated herein by reference in its entirety.

[0108] In some embodiments, a dataflow graph may then be generated from the query plan (which itself was generated using the received SQL query). In some embodiments, the dataflow graph may be generated, at least in part, from the query plan by generating the dataflow graph to include respective nodes for at least a subset (e.g., some or all) of the data processing operations identified in the query plan. In some embodiments, a single node in the query plan may result in the inclusion of multiple nodes in the dataflow graph. The order of the data processing operations specified in the query plan may then be used to generate links connecting the nodes in the dataflow graph. For example, if the generated query plan indicates that a first data processing operation is performed before a second data processing operation, then the generated dataflow graph may have a first node (representing the first data processing operation) and a second node (representing the second data processing operation) and one or more links specifying a path from the first node to the second node.

[0109] In some embodiments, generating a dataflow graph from the query plan includes adding one or more nodes to the graph that represent input and / or output data sources. For example, generating the dataflow graph may include adding an input node for each of the data sources from which data records are read during execution of the SQL query. Each input node may be configured with parameter values ​​associated with the respective data source. These values ​​may indicate how to access the data records in the data source. As another example, generating the dataflow graph may include adding an output node for each of the data sinks to which data records are written during execution of the SQL query. Each output node may be configured with parameter values ​​associated with the respective data sink. These values ​​may indicate how to write the data records to the data source.

[0110] It should be appreciated that a dataflow graph generated from a query plan is distinct from the query plan itself. While a dataflow graph can be executed by using a graph execution environment (e.g., collaborative system 610 or other suitable execution environment for executing dataflow graphs), a query plan cannot be executed by a graph execution engine (it is an intermediate representation used to generate a dataflow graph, which is executed by a graph execution engine to execute SQL queries). A query plan is not executable and must be further processed to generate an execution strategy, even in the context of a relational database management system. In contrast, a dataflow graph is executable by a graph execution engine to execute SQL queries. Additionally, even after further processing by a relational database system, the resulting execution strategy does not allow for reading and / or writing of data to other types of data sources and / or data sinks, although dataflow graphs are not limited in this respect.

[0111] In some embodiments, a dataflow graph generated from a query plan may contain nodes that represent data processing operations that are not present in the query plan. Conversely, in some embodiments, a dataflow graph generated from a query plan may not contain nodes that represent data processing operations that are present in the query plan. This situation may arise due to various optimizations that may be performed during the process of generating a dataflow graph from a query plan. In some embodiments, a dataflow graph may contain nodes that represent data processing operations other than database operations performed on a database computer system (e.g., a relational database management system).

[0112] In some embodiments, the query plan and the data flow graph may be embodied in different types of data structures. For example, in some embodiments, the query plan may be embodied in a directed graph (e.g., a tree, e.g., a binary tree) where each node has a single parent node, while the data flow graph may be embodied in a directed acyclic graph that may have at least one node with multiple parent nodes. The data flow graph may be embodied using one or more data structures having fields that store information specifying the nodes and references (e.g., pointers or any other suitable references) to other nodes that represent links between the nodes in the graph.

[0113] It should be appreciated that process 200 is illustrative and variations exist. For example, in the illustrated embodiment of FIG. 2, the processing layout of the intermediate nodes is determined using a forward pass followed by a backward pass, but in other embodiments, the processing layout may instead be determined using a backward pass followed by a forward pass. As another example, in the illustrated embodiment, the processing layout of the intermediate nodes was determined based on layouts assigned to input and output nodes, but the processing layout determination techniques described herein may be applied more generally. For example, a processing layout may be obtained for a set of one or more nodes in a dataflow graph, and a processing layout may be obtained for other nodes in the dataflow graph based on (1) the processing layout obtained for the set of nodes, (2) the link structure of the dataflow graph, and (3) one or more layout determination rules. The set of nodes may, but need not, include input and output nodes. Thus, the set of nodes may include any suitable number (e.g., zero, at least one, all) of input nodes, any suitable number (e.g., zero, at least one, all) of output nodes, and any suitable number (e.g., zero, at least one, all) of other nodes. The only requirement is that the set of nodes is non-empty.

[0114] In some embodiments, the data processing system may (1) receive a database query (e.g., SQL) query, (2) convert the received database query into computer code including computer code portions that, when executed by the data processing system, execute the received database query, and (3) automatically determine a process layout for executing each computer code portion. In some embodiments, the process layout for executing the computer code portions may be determined using information indicative of an execution order of the computer code portions. For example, in some embodiments, each computer code portion may be associated with a respective node in a dataflow graph, and the structure of the graph (e.g., as embodied in the connections between the nodes) may be used, along with the layout determination rules described herein, to assign a process layout to the node and, relative to the computer code portions associated with the node. However, it should be appreciated that in some embodiments, the process layout for executing the computer code portions may be determined without using a dataflow graph, as the information indicative of the execution order of the computer code portions is not limited to being encoded in a dataflow graph, but may be encoded in other suitable manners (e.g., another type of data structure or structures) (aspects of the technology described herein are not limited in this respect).

[0115] Thus, in some embodiments, a data processing system may obtain (e.g., received from a remote source and / or over a network connection, accessed from local storage, etc.) computer code that, when executed by the data processing system, causes a data process to perform a database query, the computer code including (A) a first set of one or more computer code portions, each of which represents a data processing operation to read a respective input data set, (B) a second set of one or more computer code portions, each of which represents a data processing operation to write a respective output data set, and (C) a third set of one or more computer code portions, each of which represents a respective data processing operation. The data processing system may then determine a process layout for executing each of the computer code portions that are part of the computer code. For example, in some embodiments, a data processing system may (A) obtain (e.g., receive, access, etc.) a first set of one or more processing layouts for one or more code portions in a first set of code portions, (B) obtain a second set of one or more processing layouts for one or more code portions in a second set of code portions, and (C) determine a processing layout for each code portion in a third set of code portions based on one or more layout decision rules described herein, including at least one rule for selecting among the first set of processing layouts, the second set of processing layouts, and processing layouts having different degrees of parallelism.

[0116] In some embodiments, computer code may be generated from a database query. For example, in some embodiments, a received database query (e.g., an SQL query) may be converted into a query plan, and the query plan may be processed to generate computer code. For example, the query plan may be converted into a dataflow graph including a number of nodes and edges (as described above), and the computer code may include computer code portions, each code portion including code for performing a data processing operation represented by a node in the dataflow graph. Thus, in some embodiments, a computer code portion may be associated with a respective node in the dataflow graph.

[0117] In some embodiments in which the computer code is associated with a data flow graph, the nodes of the data flow graph may include: (A) a first set of one or more nodes, where each node in the first set of nodes represents a respective input data set and where each computer code portion in the first set of computer code portions (described above) is associated with a respective node in the first set of nodes; (B) a second set of one or more nodes, where each node in the second set of nodes represents a respective output data set and where each computer code portion in the second set of computer code portions (described above) is associated with a respective node in the second set of nodes; and one or more third set of one or more nodes, where each node in the third set of nodes represents a respective data processing operation. The data processing system may use (1) a processing layout with the first and second sets of nodes, (2) one or more of the layout decision rules described herein, and (3) a structure of the graph (indicating an ordering between data processing operations) to assign one or more processing layouts to nodes in the third set of nodes. These processing layouts may then be used by the data processing system to execute computer code portions associated with the nodes in the third set of nodes.

[0118] Figures 3A-3D illustrate determining a processing layout for nodes in an illustrative dataflow graph 300 using one or more layout decision rules in accordance with some embodiments of the techniques described herein, including the embodiment described with reference to Figure 2. In particular, the examples in Figures 3A-3D illustrate that in some embodiments, in determining a processing layout for a node by selecting a layout from two different processing layouts having different degrees of parallelism, a processing layout having a greater degree of parallelism may be selected as the processing layout for the node.

[0119] 3A shows a dataflow graph 300 with nodes 302 and 304 representing respective input data sets, nodes 306, 308, 310, and 312 representing respective data processing operations, and node 314 representing an output data set. As can be seen from the structure of dataflow graph 300, the input data set represented by node 302 is filtered, sorted, and then combined with a filtered version of the input data set represented by node 304 before being written to the output data set represented by node 314. In this example, after the processing layouts of the input and output data sets are obtained, it can be determined that a sequential processing layout SL1 is used to read data from the data set represented by node 302, a parallel layout PL1 is used to read data from the data set represented by node 304, and a sequential processing layout SL2 is used to write to the data set represented by node 314, as shown in FIG. Note that although each of the sequential processing layouts SL1 and SL2 indicates that data is processed sequentially (i.e., with one degree of parallelism), these sequential layouts need not be the same, as the sequential processing may be performed by different processors (e.g., by a processor at a database storing the input data set represented by node 302, and by a processor at another database storing the output data set represented by node 314). At this stage, the processing layouts for the data processing operations represented by nodes 306, 308, 310, and 312 have not yet been determined.

[0120] 3B and 3C illustrate the determination of a processing layout for data processing operations represented by nodes 306, 308, 310, and 312 based on the processing layouts obtained for nodes 302, 304, and 314. First, as shown in FIG. 3B, an initial processing layout is determined for nodes 306, 308, 310, and 312 in a forward pass starting from nodes 302 and 304 according to the structure of dataflow graph 300 and the layout determination rules described herein. For example, the processing layout of node 306 is determined based on the layout of its predecessor node (node ​​302) in the dataflow graph. Then, the processing layout of node 308 is determined based on the layout of its predecessor node (node ​​306) in the dataflow graph. The processing layout of node 310 is determined based on the layout of its predecessor node (i.e., node 304), and after the processing layout of node 308 has been determined, the processing layout of node 312 is determined based on the layout of nodes 308 and 310 (which respectively precede and are connected to node 312 in dataflow graph 300).

[0121] In this example, during the forward pass, since there is no node other than node 302 immediately preceding node 306 and there is no layout already associated with node 306, it is determined that the sequential layout SL1 of node 302 is used to perform the data processing operation represented by node 306. Then, since there is no node other than node 306 preceding node 308 and there is no layout already associated with node 308, it is determined that the layout SL1 of node 306 is used to perform the data processing operation represented by node 308. Similarly, since there is no node other than node 304 preceding node 310 and there is no layout already associated with node 310, it is determined that the parallel layout PL1 of node 304 is used to perform the data processing operation represented by node 310. In this manner, the layouts SL1 and PL1 are propagated through the graph 300 from the input nodes 302 and 304 to nodes whose layout has not yet been determined and which are connected to a single preceding node (i.e., in this illustrative example, nodes 306, 308, and 310).

[0122] During the forward pass, a processing layout for node 312, which represents a join operation, is selected from the sequential layout SL1 of predecessor node 308 and the parallel layout PL1 of predecessor node 310. As shown in Figure 3B, during the forward pass, the parallel layout PL1 is selected for node 312 using a layout decision rule that indicates, for example, that a parallel layout is always selected over a sequential layout, or that when choosing between two different potential processing layouts with different degrees of parallelism, the processing layout with the greater degree of parallelism should be selected. During the forward pass, the parallel processing layout PL1 is selected for node 312 because PL1 is parallel while SL1 is sequential, or because the parallel processing layout PL1 has a greater degree of parallelism than the processing layout SL1.

[0123] Then, as shown in FIG. 3C, a final processing layout is determined for nodes 306, 308, 310, and 312 in a backward pass starting from node 314 according to the structure of dataflow graph 300, the initial processing layout shown in FIG. 3B, and the layout decision rules described herein. For example, the final processing layout of node 312 is determined based on the initial processing layout determined for node 312 and the layout of node 314. As shown in FIG. 3C, for example, a parallel layout PL1 is finally selected for node 312 during the backward pass using a layout decision rule indicating that a parallel layout is always selected over a sequential layout. The final processing layout of node 308 is determined based on the initial processing layout determined for node 308 and the final processing layout determined for node 312. The final processing layout of node 306 is determined based on the initial processing layout determined for node 306 and the final processing layout determined for node 308. The final processing layout of node 310 is determined based on the initial processing layout determined for node 310 and the final processing layout determined for node 312.

[0124] In this example, during the backward pass, a final processing layout for node 312 is selected from an initial processing layout PL1 determined for node 312 during the forward pass and a sequential processing layout SL2 associated with node 314. As shown in FIG. 3C, layout PL1 is determined to be the final processing layout for node 312 because layout PL1 has a greater degree of parallelism than layout SL2. A final processing layout for node 308 is selected from an initial layout SL1 determined for node 308 during the forward pass and a final processing layout PL1 determined for node 312 during the backward pass. As shown in FIG. 3C, layout PL1 is determined to be the final processing layout for node 308 because layout PL1 has a greater degree of parallelism than layout SL1. A final processing layout for node 306 is selected from an initial layout SL1 determined for node 306 during the forward pass and a final processing layout PL1 determined for node 308 during the backward pass. 3C, because layout PL1 has a greater degree of parallelism than layout SL1, layout PL1 is determined to be the final processing layout for node 304. The final processing layout for node 310 is determined to be PL1 because the initial layout determined for node 310 during the forward pass was PL1, and the final layout determined for node 312 during the backward pass is also PL1.

[0125] As shown in Figure 3C, after a processing layout has been determined for each node of the dataflow graph 300, the dataflow graph 300 may be configured to perform one or more repartitioning operations. As described herein, a dataflow graph may be configured to perform a repartitioning operation on data records when adjacent nodes in the dataflow graph are configured to perform data processing operations on data records using processing layouts having different degrees of parallelism. For example, as shown in Figure 3C, the processing layouts of adjacent nodes 302 (SL1) and 306 (PL1) have different degrees of parallelism. The processing layouts of adjacent nodes 312 (PL1) and 314 (SL2) also have different degrees of parallelism.

[0126] In some embodiments, a dataflow graph may be configured to perform a repartitioning operation by adding a new node to the graph, the new node representing the repartitioning operation. For example, as shown in FIG. 3D, a new node 342 representing a partitioning operation (e.g., a partition-by-key operation) may be added to the dataflow graph between nodes 302 and 306. When a data record is processed according to a dataflow graph with node 342, the data record is partitioned according to the partitioning operation represented by node 342 after it has been read using the processing layout SL1 of node 302, but before it has been filtered using the processing layout PL1 of node 306. The partitioning operation may be performed according to the layout SL1 of node 308. In addition, as shown in FIG. 3D, a new node 344 representing a merge operation may be added to the dataflow graph between nodes 312 and 314. When a data record is processed according to a dataflow graph with node 344, the data record is merged after it has been processed using the processing layout PL1 of node 312, but before it is output using the processing layout SL2 of node 314. The merging operation may be performed according to the layout SL2 of node 314.

[0127] In the illustrative example of FIG. 3D, two new nodes are added to dataflow graph 300 to obtain dataflow graph 340. However, it should be appreciated that in some embodiments, a dataflow graph may be configured to perform one or more repartitioning operations without adding new nodes to the graph. In some embodiments, each of one or more existing nodes may be configured to perform a respective repartitioning operation. For example, rather than adding new node 342 as shown in the illustrative embodiment of FIG. 3D, either node 302 or node 306 may be configured to perform a partitioning operation. As another example, rather than adding new node 344 as shown in the illustrative embodiment of FIG. 3D, either node 312 or node 314 may be configured to perform a merging operation.

[0128] 4A-4C illustrate the determination of a processing layout for a node in an illustrative dataflow graph 400 using one or more layout decision rules according to some embodiments of the techniques described herein, including the embodiment described with reference to FIG. 2. In particular, the examples of FIGS. 4A-4C illustrate that in some embodiments, when determining a processing layout for a node by selecting a layout from two parallel processing layouts having the same or different degrees of parallelism, a processing layout that applies to a larger number of records may be selected as the processing layout for the node. For example, instead of selecting a processing layout with the highest degree of parallelism among multiple parallel processing layouts having different degrees of parallelism, a processing layout that applies to a largest number of records among multiple parallel processing layouts may be selected as the processing layout for the node. This leads to a more efficient use of computing resources, since the number of data records moved between computing devices is kept as small as possible.

[0129] 4A shows a dataflow graph 400 with nodes 402 and 404 representing respective input data sets, node 406 representing a data processing operation, and node 408 representing an output data set. As can be recognized from the structure of the dataflow graph 400, an input data set having N data records and represented by node 402 is combined with an input data set having M records (M is smaller than N) and represented by node 404. After the data sets are combined, they are written to an output data set represented by node 408. In this example, after the processing layouts of the input data set and the output data set are obtained, it can be determined that a parallel processing layout PL1 is used to read data from the data set represented by node 402, a parallel processing layout PL2 is used to read data from the data set represented by node 404, and a sequential layout SL1 is used to write data records to the output data set represented by node 408. Each of the parallel processing layouts PL1 and PL2 has the same or different degree of parallelism. At this stage, the processing layout of the join operation represented by node 406 has not yet been determined.

[0130] During the forward path, based on the processing layouts of nodes 402 and 404 that precede node 406 in the data flow graph 400, the initial processing layout of node 406 is determined. In the illustrated example, the initial processing layout of node 406 is selected from the processing layout PL1 associated with node 402 and the processing layout PL2 associated with node 404. Regardless of the degree of parallelism of the parallel layouts PL1 and PL2, layout PL1 processes a larger number of record counts N (e.g., reads N data records from the input data set represented by node 402) than the layout PL2 applied to process M < N data records (e.g., reads M data records from the input data set represented by node 404), so PL1 is selected as the initial processing layout for node 406. This selection can be made for efficiency purposes because the number of data records that need to be moved (e.g., M < N records) is less than the number of records that need to be moved (e.g., N records) when processing the join operation represented by node 406 according to layout PL1 compared to when the join operation is processed according to layout PL2.

[0131] Then, during the backward pass, a final processing layout for node 406 is determined based on the initial processing layout (PL1) determined for node 406 and the processing layout (SL1) associated with node 408. According to the rules, the parallel layout PL1 is selected over the sequential layout SL1, so PL1 is determined to be the final processing layout for node 406. Thus, after the forward and backward passes are completed, PL1 is determined to be the final processing layout for node 406, as shown in FIG. 4B. During both the forward and backward passes, parallel processing layouts may generally be preferred for selection over sequential processing layouts, and if only sequential processing layouts are considered, or if multiple parallel processing layouts are considered, the processing layout considered that is used to process the largest number of data records is selected for that particular node.

[0132] As shown in Figure 4B, after a processing layout has been determined for each node of dataflow graph 400, dataflow graph 400 may be configured to perform one or more repartitioning operations. As described herein, a dataflow graph may be configured to perform a repartitioning operation on data records when adjacent nodes in the dataflow graph are configured to perform data processing operations on data records using processing layouts having different degrees of parallelism. For example, as shown in Figure 4B, the processing layouts of adjacent nodes 406 (PL1) and 408 (SL1) have different degrees of parallelism.

[0133] As described herein, in some embodiments, a dataflow graph may be configured to perform a repartitioning operation by adding a new node to the graph that represents the repartitioning operation. For example, as shown in FIG. 4C, a new node 407 that represents a merge operation may be added to dataflow graph 400 to obtain dataflow graph 430. When data records are processed according to a dataflow graph having node 407, the data records are merged after being processed using the processing layout PL1 of node 406, but before being output using the processing layout SL1 of node 408. The merging operation may be performed according to the layout SL1 of node 408. In other embodiments, instead of adding a new node to dataflow graph 400 to perform the merge operation, one of the existing nodes (e.g., 406 or 408) may be configured to perform the merge operation.

[0134] Figures 5A-5D illustrate the determination of a processing layout of nodes in an illustrative dataflow graph 500 using one or more layout decision rules in accordance with some embodiments of the techniques described herein, including the embodiment described with reference to Figure 2. In particular, the examples in Figures 5A-5D illustrate that in some embodiments, a processing layout may be chosen in determining the processing layout of other nodes as the layout of a particular node or nodes that is not propagated beyond the particular node.

[0135] 5A shows a dataflow graph 500 with nodes 502 and 504 representing respective input datasets, nodes 506, 508, 510, and 512 representing respective data processing operations, and node 514 representing an output dataset. As can be seen from the structure of the dataflow graph 500, the input dataset represented by node 502 is first filtered, then a rollup operation is performed on the filtered data, and the data records obtained as a result of the rollup operation are combined with the filtered version of the input dataset represented by node 504 before being written to the output dataset represented by node 514. In this example, after the processing layouts of the input dataset and the output dataset are obtained, it can be determined that a parallel processing layout PL1 is used to read data from the input dataset represented by node 502, a sequential layout SL1 is used to read data from the input dataset represented by node 504, and a sequential processing layout SL2 is used to write to the output dataset represented by node 514, as shown in FIG. 5A. In addition, in this example, an indication may be obtained that the processing layout PL1 is not to be propagated to other nodes. This indication may be obtained in any suitable manner, for example from a user via a graphical user interface. At this stage, the processing layout of the data processing operations represented by nodes 506, 508, 510, and 512 has not yet been determined.

[0136] 5B and 5C illustrate the determination of processing layouts for data processing operations represented by nodes 506, 508, 510, and 512 based on the processing layouts obtained for nodes 502, 504, and 514. First, as shown in FIG. 5B, an initial processing layout is determined for nodes 506, 508, 510, and 512 in a forward pass starting from nodes 502 and 504 according to the structure of dataflow graph 500 and the layout determination rules described herein. In this example, an indication has been obtained that the processing layout PL1 of node 502 is not propagated, so the layout PL1 is copied only to node 506 representing a filtering operation (as a filtering operation is a type of operation that can be performed using the same processing layout (indeed, the same computing device) as the layout used to read data records from the input data set represented by node 502) and not to other nodes, such as node 508 representing a rollup operation.

[0137] Thus, during the forward pass, the initial processing layout of node 506 is determined to be PL1, and the initial processing layout of node 508 is not determined because PL1 is not propagated beyond node 506. As described below, the processing layout of node 508 is determined in the backward pass.

[0138] In addition, during the forward pass, since there is no node other than node 504 immediately preceding node 510 and there is no layout already associated with node 510, the initial processing layout of node 510 is determined to be the sequential layout SL1 of node 504. Then, since node 510 is the only node preceding node 512 associated with a particular layout (as mentioned above, node 508 precedes node 512, but it is not associated with any initial processing layout), the initial processing layout SL1 of node 510 is also determined to be the initial processing layout of node 512. The initial processing layouts determined as a result of the forward pass are shown in FIG. 5B. All nodes except node 508 have been assigned an initial processing layout.

[0139] Next, as shown in Figure 5C, in accordance with the structure of dataflow graph 500, the initial process layout shown in Figure 5B, and the layout decision rules described herein, a final process layout is determined for nodes 506, 508, 510, and 512 in the backward pass starting from node 514. For example, the final process layout for node 512 is determined based on the initial process layout determined for node 512 and the process layout associated with node 514. The final process layout for node 508 is determined based on the final process layout determined for node 512 (in this example, no initial layout has been determined for node 508). The final process layout for node 506 is determined based on the initial process layout determined for node 506 and the final process layout determined for node 508. The final process layout for node 510 is determined based on the initial process layout determined for node 510 and the final process layout determined for node 512.

[0140] In this example, during the backward pass, the final processing layout of node 512 is selected from the initial processing layout SL1 determined for node 512 during the forward pass and the sequential processing layout SL2 associated with node 514. As shown in FIG. 5C, layout SL1 is determined to be the final processing layout of node 512. The final processing layout of node 508 is determined to be layout SL1 because layout SL1 is the final determined layout of node 512 and node 508 is not associated with any initial processing layout after the forward pass. The final processing layout of node 506 is determined to be PL1 (the initial layout determined for node 506) because PL1 has a greater degree of parallelism than layout SL1 determined to be the final processing layout of node 508. The final processing layout of node 510 is determined to be SL1 because the initial layout determined for node 510 during the forward pass is SL1 and the final layout determined for node 512 during the backward pass is also SL1.

[0141] As shown in Figure 5C, after a processing layout has been determined for each node of the dataflow graph 500, the dataflow graph 500 may be configured to perform one or more repartitioning operations. As described herein, a dataflow graph may be configured to perform a repartitioning operation on data records when adjacent nodes in the dataflow graph are configured to perform data processing operations on data records using processing layouts having different degrees of parallelism. For example, as shown in Figure 5C, the processing layouts of adjacent nodes 506 (PL1) and 508 (SL1) have different degrees of parallelism.

[0142] As described herein, in some embodiments, a dataflow graph may be configured to perform a repartitioning operation by adding a new node to the graph that represents the repartitioning operation. For example, as shown in FIG. 5D, a new node 532 that represents a merge operation may be added to dataflow graph 500 to obtain dataflow graph 530. When data records are processed according to a dataflow graph having node 532, the data records are aggregated after being processed using the processing layout PL1 of node 506, but before being output using the processing layout SL1 of node 508. The aggregation operation may be performed according to the layout SL1 of node 508. In other embodiments, instead of adding a new node to dataflow graph 500 to perform the aggregation operation, one of the existing nodes (e.g., 506 or 508) may be configured to perform the aggregation operation.

[0143] 6 is a block diagram of an illustrative computing environment 600 in which some embodiments of the technology described herein may operate. The environment 600 includes a data processing system 602 configured to access (e.g., read data and / or write data) data stores 610, 612, 614, and 616. Each of the data stores 610, 612, 614, and 616 may store one or more data sets. The data stores may store any suitable type of data in any suitable manner. The data stores may store data as flat text files, as spreadsheets, using a database system (e.g., a relational database system), or in any other suitable manner. In some cases, the data stores may store transaction data. For example, the data stores may store credit card transactions, call log data, or bank transaction data. It should be appreciated that the data processing system 602 may be configured to access any suitable number of data stores of any suitable type (although aspects of the technology described herein are not limited in this respect).

[0144] The data processing system includes a graphical development environment (GDE) 606 that provides an interface for one or more users to create dataflow graphs. Dataflow graphs created using the GDE 606 may be executed using a collaborative system 610, or other suitable execution environment for executing dataflow graphs. Aspects of the graphical development environment and the environment for executing dataflow graphs are described in U.S. Pat. No. 5,966,072, entitled "Executing Computations Represented as Graphs," and U.S. Pat. No. 7,716,630, entitled "Parameter Management for Graph-Based Computations," each of which is incorporated herein by reference in its entirety. Dataflow graphs created using the GDE 606, or obtained in any other suitable manner, may be stored in a dataflow graph storage device 608 that is part of the data processing system 602.

[0145] Data processing system 602 also includes a parallel processing module 604 configured to determine the processing layout of the nodes of a dataflow graph prior to execution of the dataflow graph by collaborating system 610. Parallel processing module 604 may determine the processing layout of the nodes of the dataflow graph using any of the techniques described herein, including, for example, the techniques described with reference to process 200 of FIG.

[0146] Pruned Propagation The inventors have recognized that in some situations, a processing layout determined using the above-described layout propagation technique may result in more computation than necessary. In particular, the inventors have recognized that additional optimizations may be available in situations where at least some of the data processed by nodes in a dataflow graph is not used in downstream processing. The inventors have developed techniques for assigning processing layouts to nodes of a dataflow graph in such situations in a manner that improves overall performance when such dataflow graphs are executed.

[0147] As an example, consider a situation where different data sets accessed by a dataflow graph are stored with different degrees of parallelism. For example, one input data set ("A") may be a file stored in one location, while another input data set ("B") may be stored in four different locations using a distributed file system (e.g., the Hadoop distributed file system). In this case, the layout propagation techniques described herein can propagate the sequential and 4-way parallel layouts to various nodes in the dataflow graph. Then, subject to the layout propagation rules described above that favor layouts with higher degrees of parallelism, the 4-way parallel layout will be selected over the sequential layout for various dataflow graph nodes (e.g., for nodes that represent join or union-all operations on data from two data sources). This is a good thing. However, if no data from input data set "B" is actually used in computing the output of the dataflow graph (e.g., computing the results of the SQL query for which the dataflow graph was generated), then propagating the 4-way parallel layout associated with data set "B" to other parts of the dataflow graph may actually worsen performance.

[0148] More generally, the inventors have recognized that in some embodiments, while data generated by a particular node (or nodes) in a dataflow graph may not be used by downstream nodes in the dataflow graph, the processing layout assigned to those particular nodes may nevertheless be propagated to other nodes in the dataflow graph, which may result in suboptimal performance.

[0149] One reason for the performance degradation is that the resulting dataflow graph has one or more nodes associated with multiple different processing layouts, including processing layouts propagated from nodes (e.g., input and / or output nodes) that were not used during execution. Such nodes may be pruned from the graph, but the processing layouts associated with these nodes may nevertheless be propagated to other parts of the graph. As a result, additional repartitioning operations (e.g., partition bi-key, round robin partition, aggregation, merging) may be introduced into the dataflow graph, which increases the overall amount of processing resources required to execute the dataflow graph. An example of this is described with reference to Figures 8A-8J.

[0150] Another reason for performance degradation is related to the way dataflow graphs are executed. Certain simpler types of dataflow graphs can be executed as micrographs, using a single process on a single computing device (e.g., a micrograph server). This is advantageous as it reduces the start-up time for executing a dataflow graph from seconds to microseconds, an important practical consideration. Aspects of micrographs are described in U.S. Patent No. 9,753,751, filed October 22, 2014, entitled "Dynamically Loading Graph-Based Computations," which is incorporated by reference in its entirety.

[0151] In some embodiments, if the execution of a dataflow graph does not involve inter-process communication, the dataflow graph may be executed as a micrograph. Thus, in some embodiments, a dataflow graph having repartitioning components and / or nodes with different degrees of parallelism cannot be executed as a micrograph. On the other hand, a dataflow graph having nodes with only sequential layouts may be executed as a micrograph. As a result, a data processing system (e.g., data processing system 602) may decide to execute a particular dataflow graph (the entire dataflow graph or a portion of a dataflow graph) as a micrograph based on the processing layouts assigned to the nodes of the dataflow graph (e.g., a portion thereof). If the processing layout (e.g., a parallel processing layout) associated with a node pruned from the dataflow graph is propagated to other nodes in the graph, this may prevent the resulting graph from being executed as a micrograph. For example, a situation may exist where all other dataflow graph nodes have a sequential processing layout, apart from the propagation of the parallel layout from the pruned portion of the graph. In such a situation, propagating the parallel layout from the pruned portion of the graph may prevent the resulting graph from being executed as a micrograph, resulting in poor performance.

[0152] To address the above problems, the inventors have developed an improved processing layout propagation technique in which processing layouts associated with nodes to be pruned (e.g., nodes that output data that is not used by downstream nodes in the graph) are not propagated during processing layout propagation. Thus, the developed technique involves performing layout propagation using information indicating that data that is not used by a particular set of one or more nodes is not used by any node in the dataflow graph downstream from the particular set of nodes.

[0153] In some embodiments, the processing layout is determined in two layout propagation passes: a forward propagation pass and a backward propagation pass. In some embodiments, the technique developed by the inventors involves identifying "unused" or "to-be-pruned" nodes during the forward pass, which are nodes whose outputs are not used by any downstream nodes in the dataflow graph, and preventing the processing layout associated with such "unused" nodes from being propagated to other nodes (during the forward pass, and optionally during the backward pass).

[0154] For example, as shown in the context of the illustrative dataflow graph 700 of FIG. 7, a data processing system may determine that the output of node 704 is not required by any of the downstream nodes in the graph (i.e., nodes 710, 712, and 714). In that sense, node 705 (including nodes 702 and 704) of dataflow graph 700 is unused and may be pruned as part of a dataflow graph optimization performed prior to execution of the graph. In this context, the technique developed by the inventors includes (1) identifying node 704 as an "unused" node and (2) preventing the processing layout PL1 associated with node 704 from being propagated to other nodes in the graph (e.g., nodes 708, 710, and 712) during the forward and backward passes. As a result, the remaining nodes (704, 708, 710, 712, and 714) are all assigned sequential layouts, no repartitioning operations are added, and the resulting graph may be executed as a micrograph.

[0155] As noted above, in some embodiments, a data processing system may determine whether a dataflow graph should be executed as a micrograph based on the process layouts assigned to nodes in the dataflow graph. In some embodiments, this determination is made without considering the process layouts assigned to "unused" nodes. Thus, if a node is flagged as an "unused" node during a forward (or reverse) process layout propagation pass, the process layout associated with that particular node is not used to determine whether the dataflow graph as a whole can be executed as a micrograph.

[0156] A detailed non-limiting example of the above will now be described with reference to the drawings. Figures 8A-8J illustrate an example where the propagation of processing layout from "unused" nodes to other parts of the graph results in the introduction of more repartitioning operations than necessary, thereby increasing the overall amount of processing resources required to execute the dataflow graph and preventing it from being executed as a micrograph. Techniques developed by the inventors to address this issue will then be described with reference to Figures 9A-9D.

[0157] FIG. 8A illustrates an illustrative SQL query 801 processed as a dataflow graph according to some embodiments of the techniques described herein. As shown in FIG. 8A, the SQL query includes: (1) selecting all records from a first data source, called “archive data,” dated before January 1, 2015; (2) selecting all records from a second data source, called “new data,” dated after January 1, 2015; (3) combining all of these records using a “union-all” operation; and (4) selecting from the combined records those having date “D.” As can be easily seen from this query, if the date “D” is before January 1, 2015, then any records from the second data source (the “new data”) are not needed and will not be included in the query results. Similarly, if the date “D” is after January 1, 2015, then any records from the first data source (the “archive data”) are not needed and will not be included in the query results.

[0158] In this example, the "Archived Data" data source is associated with a parallel layout ("PL1"), meaning that data read from this data source is read using this parallel layout. Meanwhile, in this example, the "New Data" data source is associated with a sequential layout ("SL1"), meaning that data read from this data source is read using a sequential layout. The output of the query is written using a sequential data layout. As explained in more detail below, in this example, data from "Archived Data" is not needed to generate the query results (e.g., if date "D" is after January 1, 2015), but the parallel layout ("PL1") associated with the "New Data" data source may be propagated (e.g., using the layout propagation rules described herein) and used to process the data read from the "New Data" source (which is sequential), requiring the introduction of a repartitioning component and impacting overall performance.

[0159] Figure 8B shows an illustrative dataflow graph 800 generated from the SQL query 801 shown in Figure 8A in accordance with some embodiments of the techniques described herein. For example, the SQL query 801 can be used to generate a query plan, and the dataflow graph 800 can be generated from the query plan as described herein, including with reference to the process 200 shown in Figure 2. As shown in Figure 8B, the dataflow graph 800 includes a node 802 for reading data from an "archived data" data source using a parallel processing layout "PL1", a node 804 for reading data from a "new data" data source using a sequential processing layout "SL1", and a node 814 for outputting results of the processing performed by the dataflow graph using the sequential processing layout "SL1". The dataflow graph further includes node 806, which represents a filtering operation performed on the records retrieved from the “Archive Data” data source to select only records dated before January 1, 2015; node 808, which represents a filtering operation performed on the records retrieved from the “New Data” data source to select only records dated on or after January 1, 2015; node 810, which represents a “union all” operation; and node 812, which represents a filtering operation to select records having date “D” (from among the records resulting from the union all operation).

[0160] As shown in FIG. 8B, the process layouts for nodes 806, 808, 810, and 812 have not yet been set. These process layouts may be set according to the layout propagation techniques described herein, as shown in FIGS. 8C and 8D. Specifically, FIG. 8C illustrates a dataflow graph 815 showing process layouts determined for nodes in dataflow graph 800 during a forward layout propagation pass. As can be seen, a parallel layout PL1 has been propagated to node 806, and a sequential process layout SL1 has been propagated to node 808. Next, the parallel layout PL1 is assigned to node 810 by selecting layout PL1 from among two alternatives (PL1 and SL1) based on a process layout decision rule that when selecting a process layout for a node from a group of two or more process layouts (e.g., assigned to the node immediately preceding the node in the dataflow graph), the process layout with the highest degree of parallelism may be selected. Since PL1 has the highest degree of parallelism, this layout is selected for node 810 and is then propagated to node 812 as well.

[0161] After the forward processing layout propagation pass, a backward processing layout propagation pass is performed and the determined layout is shown in dataflow graph 820 of Figure 8D. Notably, during the backward pass, the processing layout of node 808 is updated from sequential ("SL1" in this example) to parallel ("PL1" in this example) because the PL1 layout, propagated from node 810, has a higher degree of parallelism than the sequential layout "SL1".

[0162] 8C-8D show how a process layout is assigned to the unoptimized dataflow graph generated from the SQL query 801 shown in FIG. 8A. However, in some embodiments, the dataflow graph 800 shown in FIG. 8B may be optimized if more information is known about the date of interest "D". Specifically, assume that the SQL query 801 includes the SQL query 831 (instead of its last line) such that data records having a date of May 19, 2007 are requested. In this case, the generated dataflow graph may be optimized by pruning the branch after the union-all operation since records from the "new data" data source are no longer needed. FIG. 8E shows an illustrative graph 830 that may be generated in this situation.

[0163] As shown in FIG. 8E, dataflow graph 830 includes filtering nodes 816 and 818. Node 816 represents a filtering operation to select records having a date of May 19, 2007 from the "Archive Data" data source. Node 818 represents a filtering operation associated with a flag indicating that the results output by this node will not be used by any operations downstream from this node in the dataflow graph. In this example, the flag is a "one-time filtering" flag, which is set to "false" (e.g., in a PostgreSQL implementation, the bottom branch may be placed after a PResultOperator with a "one_time_filter" flag set to "0"). After layout propagation, the assigned processing layout is shown in dataflow graph 835 of FIG. 8F.

[0164] The dataflow graph 835 of FIG. 8F is then optimized as shown in FIG. 8G. The bottom branch is pruned such that nodes 804 and 818 are removed since the results produced by node 818 are not used to generate the results of SQL query 831. With the bottom branch of the graph pruned, node 810 is also removed since there is no need for a union-all operation. Node 817 is added to the graph for performing a repartitioning operation (e.g., an aggregation or merge operation) since processing layout PL1 is assigned to node 816 and processing layout SL1 is assigned to node 814. In this case, processing proceeds efficiently and parallel processing layout PL1 is used to access data from the "archive data" data source.

[0165] However, the situation is different when different branches of the graph are pruned, as will now be described with reference to Figures 8H-8I. Specifically, assume that SQL query 801 includes SQL query 851 (instead of its last line) such that data records having data for June 10, 2018 are requested. In this case, since records from the "archive data" data source are not needed, the generated data flow graph may be optimized by pruning other branches of the graph that are after the union-all operation. Figure 8H shows an illustrative graph 845 that may be generated in this situation.

[0166] As shown in FIG. 8H, dataflow graph 845 includes filtering nodes 822 and 824. Node 824 represents a filtering operation to select records with a date of June 10, 2018 from the "New Data" data source. Node 822 represents a filtering operation associated with a flag indicating that the results output by this node will not be used by any operation downstream from this node in the dataflow graph. In this example, the flag is a "one-time filtering" flag and is set to "false" (e.g., in a PostgreSQL implementation, the bottom branch may be placed after a PResultOperator with a "one_time_filter" flag set to "0"). After layout propagation, the assigned processing layout is shown in dataflow graph 850 of FIG. 8I.

[0167] Dataflow graph 850 of Figure 8I is then optimized as shown in dataflow graph 855 of Figure 8J. The top branch is pruned such that nodes 802 and 822 are removed since the results produced by node 822 are not used to generate results for SQL query 851. Node 810 is also removed since there is no need for a union-all operation since the top branch of the graph has been pruned. Nodes 832 and 834 are added to the graph for performing repartitioning operations (e.g., partition and aggregate operations) since a parallel processing layout PL1 is assigned to node 808 and a sequential processing layout SL1 is assigned to nodes 804-814.

[0168] As can be seen from dataflow graph 855, parallelism is introduced into a graph that reads from a sequential data source and writes to a sequential data sink. So, in this example, an otherwise sequential dataflow graph includes a component (node ​​824) that runs in a parallel processing layout originating from a pruned branch (propagation from the pruned branch). This is inefficient in that it requires the introduction of two repartitioning operations, both of which would not be necessary if node 824 were associated with a sequential processing layout. However, the introduction of these two repartitioning operations has a negative impact on the execution of the graph. First, the graph can no longer be executed by a single process (e.g., a single process running on a microserver). This increases the startup time of the graph from microseconds to seconds. Second, the repartitioning operations take time to execute and therefore consume computational resources (e.g., processors, memory, networking, etc.), resulting in suboptimal overall execution time and allocation of computational resources.

[0169] Although the above examples with reference to Figures 8A-8J include a union-all operation, similar problems may arise in the context of other operations. For example, similar problems may arise in the context of a join operation. For example, a data flow graph may have a node for joining two tables T1 and T2, but data from T1 may not be required to perform any processing following the join. In this example, if T1 is loaded using a parallel processing layout and T2 is loaded using a sequential processing layout, the node associated with T1 may be pruned, but the parallel processing layout may nevertheless propagate and affect other nodes for processing data from T2.

[0170] The inventors recognize that the situation that arises in the above example is undesirable. Specifically, the inventors recognize that it is beneficial to avoid a situation where a parallel layout from a pruned branch of a dataflow graph is propagated out of the branch to the remainder of the graph. If this can be prevented, certain situations (e.g., executing an otherwise sequential graph with parallel components) can be avoided.

[0171] Accordingly, the inventors have developed a technique that includes identifying one or more nodes to be pruned from the dataflow graph (e.g., by identifying one or more nodes whose outputs are not used by any other downstream nodes in the dataflow graph) as part of the processing layout propagation (e.g., during the forward and / or backward processing layout propagation passes) and not propagating the processing layout associated with such nodes. Furthermore, the processing layout associated with such nodes is not used to determine whether the graph can be executed as a micrograph. In this manner, the undesirable effects described above, including with reference to Figures 8A-8J, can be avoided. The technique developed by the inventors to solve this problem is illustrated in Figures 9A-9D.

[0172] 9A shows an illustrative SQL query 900 processed as a dataflow graph in accordance with some embodiments of the techniques described herein. This query, similar to SQL queries 801 and 851, includes (1) selecting all records from a first data source called “archived data” that are dated before January 1, 2015, (2) selecting all records from a second data source called “new data” that are dated on or after January 1, 2015, (3) combining all these records using a “union-all” operation, and (4) selecting from the combined records those that have a date of “June 10, 2018.” Because the date “June 10, 2018” is after January 1, 2015, none of the records from the first data source (the “archived data”) are needed and are not included in the query results.

[0173] Figure 9B illustrates a dataflow graph 910 generated from the SQL query shown in Figure 9A, according to some embodiments of the techniques described herein. For example, the SQL query 900 may be used to generate a query plan, and the dataflow graph 910 may be generated from the query plan, as described herein, including with reference to the process 200 shown in Figure 2. In this example, the structure of the dataflow graph 910 is similar to the structure of the dataflow graph 845 shown in Figure 8H.

[0174] After the dataflow graph 910 is generated, the processing layout is propagated in forward and backward processing layout propagation passes. In this example, during the forward processing layout propagation pass, nodes whose outputs are not used downstream are identified and flagged. In this example, node 822 is identified and flag 901 is set indicating that the data processed by this node is not used downstream. In effect, the branch with nodes 802 and 822 is pruned.

[0175] As shown in Figure 9C, after the forward propagation pass, a sequential processing layout is assigned to node 810. If node 822 had not been flagged (i.e., if it had not been identified as a node whose outputs are not used downstream in dataflow graph 920), the processing layout assigned to node 810 would have been the parallel processing layout PL1 (e.g., following the layout decision rule that the processing layout with the highest degree of parallelism is selected, as shown, for example, in Figure 8C). In this example, the backward propagation pass does not change the processing layout assignment.

[0176] Dataflow graph 920 of Figure 9C is then optimized to obtain dataflow graph 930 of Figure 9D. The top branch is pruned such that nodes 802 and 822 are removed, since the results produced by node 822 are not used to generate results for SQL query 900. Node 810 is also removed, since with the top branch of the graph pruned there is no need for a union-all operation. Importantly, unlike the graph of Figure 8J, no repartitioning operation is added, since all three nodes 802, 808, and 814 are associated with the same processing layout (sequential processing layout SL1).

[0177] As can be seen from this example, dataflow graph 855 shown in Figure 8J and dataflow graph 930 shown in Figure 9D produce the same output. However, because the layouts assigned to the pruned nodes are not allowed to propagate in the examples of Figures 9A-9D but are allowed to propagate in the examples of Figures 8A-8J, dataflow graph 855 has both parallel and sequential layouts, whereas the process layouts assigned to the nodes of dataflow graph 930 are all sequential. As a result, dataflow graph 930 may execute more efficiently since no repartitioning operations are required.

[0178] In addition, as shown in FIG. 10, because dataflow graph 930 does not involve repartitioning operations and does not require inter-process communication, dataflow graph 930 can be executed as a micrograph (e.g., using hardware dedicated to executing micrographs, such as micrograph server 1010).

[0179] In the previous examples described with reference to Figures 9A-9D, "unused" nodes (e.g., node 822) are identified during the forward layout propagation pass. However, in some embodiments, such nodes may be identified during the backward pass, as aspects of the technology described herein are not limited in this respect. For example, if a backward layout propagation pass is performed before a forward propagation pass, then "unused" nodes may be identified during the backward pass.

[0180] 11 illustrates an example of a suitable computing system environment 1100 on which the technology described herein can be implemented. The computing system environment 1100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the technology described herein. Neither should the computing environment 1100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 700.

[0181] The technology described herein can be used with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computers, distributed computing environments that incorporate any of the above systems or devices, and the like.

[0182] A computing environment can execute computer-executable instructions such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The techniques described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0183] With reference to FIG. 11, an exemplary system for implementing the techniques described herein includes a general purpose computing device in the form of a computer 1110. Components of the computer 1110 may include, but are not limited to, a processing unit 1120, a system memory 1130, and a system bus 1121 that couples various system components including the system memory to the processing unit 1120. The system bus 1121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, also known as Mezzanine bus.

[0184] Computer 1110 typically includes a variety of computer readable media. Computer readable media may be any available media that can be accessed by computer 1110, and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store the desired information and that can be accessed by computer 1110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable media.

[0185] The system memory 1130 encompasses computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 1131 and random access memory (RAM) 1132. A basic input / output system 1133 (BIOS), containing the basic routines that help to transfer information between elements within the computer 1110, such as during start-up, is typically stored in ROM 1131. RAM 1132 typically contains data and / or program modules that are immediately available and / or presently being operated on by the processing unit 1120. By way of example, and not limitation, FIG. 11 illustrates operating system 1134, application programs 1135, other program modules 1136, and program data 1137.

[0186] Computer 1110 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, Figure 11 illustrates a hard disk drive 1141 which reads from or writes to non-removable, non-volatile magnetic media, a flash drive 1151 which reads from or writes to removable, non-volatile memory 1152, such as a flash memory, and an optical disk drive 1155 which reads from or writes to a removable, non-volatile optical disk 1156, such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 1141 is typically connected to the system bus 1121 through a non-removable memory interface, such as interface 1140, and the magnetic disk drive 1151 and optical disk drive 1155 are typically connected to the system bus 1121 by a removable memory interface, such as interface 1150.

[0187] The drives and their associated computer storage media, discussed above and illustrated in FIG. 11, provide storage of computer readable instructions, data structures, program modules, and other data for the computer 1110. In FIG. 11, for example, hard disk drive 1141 is illustrated as storing operating system 1144, application programs 1145, other program modules 1146, and program data 1147. Note that these components may be the same as or different from the operating system 1134, application programs 1135, other program modules 1136, and program data 1137. The operating system 1144, application programs 1145, other program modules 1146, and program data 1147 are numbered differently here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 1110 through input devices such as a keyboard 1162 and pointing device 1161, commonly referred to as a mouse, trackball, or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, etc. These and other input devices are often connected to the processing unit 1120 by a user input interface 1160 coupled to the system bus, but may also be connected by other interface and bus structures, such as a parallel port, game port, or universal serial bus (USB). A monitor 1191 or other type of display device is also connected to the system bus 1121 via an interface, such as a video interface 1190. In addition to a monitor, computers may also include other peripheral output devices such as speakers 1197 and printer 1196, which may be connected through an output peripheral interface 1195.

[0188] The computer 1110 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 1180. The remote computer 1180 may be a personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer 1110, although only a memory storage device 1181 is illustrated in FIG. 11. The logical connections depicted in FIG. 11 include a local area network (LAN) 1171 and a wide area network (WAN) 1173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0189] When used in a LAN networking environment, the computer 1110 is connected to the LAN 1171 through a network interface or adapter 1170. When used in a WAN networking environment, the computer 1110 typically includes a modem 1172 or other means for establishing communications over the WAN 1173, such as the Internet. The modem 1172, which may be internal or external, may be connected to the system bus 1121 via the user input interface 1160 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 1110, or portions thereof, may be stored in a remote memory storage device. By way of example, and not limitation, FIG. 11 illustrates remote application programs 1185 as residing on memory device 1181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0190] Having thus described several aspects of at least one embodiment of the present technology, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0191] Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of the technology described herein. Furthermore, advantages of the technology described herein are shown, but it is to be understood that not all embodiments of the technology described herein include all described advantages. Some embodiments may not implement any of the features described herein as advantageous, and in some cases, one or more of the described features may be implemented to obtain further embodiments. Thus, the above description and drawings are merely examples.

[0192] The above embodiments of the techniques described herein may be implemented in any of a number of ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in integrated circuit components, including commercially available integrated circuit components known in the industry by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processors may be implemented in custom circuits, such as ASICs, or semi-custom circuits resulting from the configuration of programmable logic devices. As yet a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of the multiple cores may constitute a processor. However, the processors may be implemented using circuits of any suitable format.

[0193] Further, it should be understood that the computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. Additionally, the computer may be incorporated into devices not generally considered computers, but equipped with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other suitable portable or fixed electronic device.

[0194] A computer may also have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for a visual representation of the output, and a speaker or other sound generating device for an audible representation of the output. Examples of input devices that may be used in a user interface include keyboards and pointing devices such as mice, touch pads, and digitizer tablets. As another example, a computer may receive input information by voice recognition or in other audible formats.

[0195] Such computers may be interconnected by one or more networks of any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0196] Also, the various methods or processes outlined herein may be encoded as software executable for one or more processors using any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine code or intermediate code that runs on a framework or virtual machine.

[0197] In this regard, the technology described herein may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuit configurations in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods for implementing various embodiments of the technology described above. As is evident from the above examples, a computer-readable storage medium can retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such one or more computer-readable storage media may be portable such that one or more programs stored thereon can be loaded into one or more different computers or other processors to implement various aspects of the technology as described above. In this specification, the term "computer-readable storage medium" covers only non-transitory computer-readable media that can be considered to be an article (i.e., an article of manufacture) or a machine. Alternatively or additionally, the techniques described herein may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0198] The terms "program" or "software" are used herein generically to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to perform various aspects of the technology described herein. Additionally, in accordance with certain aspects of the present embodiments, it is to be understood that one or more computer programs that, when executed, perform the methods of the technology described herein need not reside on a single computer or processor, but may be distributed in a modular manner among a number of different computers or processors to perform various aspects of the technology described herein.

[0199] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0200] Also, the data structures may be stored in the computer-readable medium in any suitable form. For ease of illustration, the data structures may be shown with fields related by location within the data structure. Such relationships may likewise be achieved by assigning locations within the computer-readable medium that convey the relationship between the fields to the storage of the fields. However, any suitable mechanism may be used to establish relationships between information in the fields of the data structure, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0201] Various aspects of the technology described herein may be used alone, in combination, or in various arrangements not specifically set forth in the embodiments described above, and therefore are not limited in their application to the details and arrangements of components set forth in the above description or illustrated in the drawings. For example, aspects described in one embodiment can be combined in any manner with aspects described in other embodiments.

[0202] Also, the techniques described herein may be embodied as methods, of which only one example is provided. The acts performed as part of the method may be sequenced in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that shown (which may include performing some acts simultaneously, even though in the illustrative embodiments they are shown as sequential acts).

[0203] Additionally, some actions are described as being performed by a "user." It is to be understood that a "user" need not be a single individual, and that in some embodiments, actions attributed to a "user" may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms.

[0204] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, of itself, imply a priority, precedence, or ordering of one claim element over another, or the chronological order in which acts of a method are performed, but rather is used merely as a label to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms) that distinguishes between the claim elements.

[0205] Also, the phraseology and terminology used herein are for purposes of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items.

Claims

Claim 1 using at least one computer hardware processor to obtain information specifying a data flow graph, the data flow graph including a plurality of nodes and a plurality of edges connecting the plurality of nodes, the plurality of edges representing a flow of data between the plurality of nodes, and the plurality of nodes including a first set of one or more nodes, each node in the first set of nodes representing a respective input data set in a set of one or more input data sets, a second set of one or more nodes, each node in the second set of nodes representing a respective output data set in a set of one or more output data sets, a third set of one or more nodes, each node in the third set of nodes representing at least one respective data processing operation, obtaining, obtaining a first set of one or more processing layouts for the set of input data sets and a second set of one or more processing layouts for the set of output data sets, the first set of processing layouts including processing layouts having different degrees of parallelism, the second set of processing layouts including processing layouts having different degrees of parallelism, or both the first set of processing layouts and the second set of processing layouts including processing layouts having different degrees of parallelism, determining a processing layout of nodes in the third set of nodes using (a) the first set of processing layouts, (b) the second set of processing layouts, (c) one or more layout determination rules including at least one rule for selecting from among processing layouts having different degrees of parallelism, and (d) information indicating that data generated by at least one node in the first set of nodes and / or at least one node in the third set of nodes is not used by any node in the data flow graph downstream from the at least one node, including performing Determining the processing layout is done using two layout propagation paths, In the forward path starting from nodes within the first set of nodes, according to the structure of the data flow graph, the first set of the processing layout, the one or more layout determination rules, and the information indicating that the data generated by the at least one node is not used by any node in the data flow graph downstream from the at least one node are used to determine one or more initial processing layouts of one or more nodes within the third set of nodes; In the reverse path starting from nodes within the second set of nodes, according to the structure of the data flow graph, by using the second set of the processing layout, the one or more initial processing layouts, and the one or more layout determination rules, to determine the processing layout of one or more nodes within the third set of nodes; is performed by further including identifying the at least one node during the forward path, the processing layout associated with the at least one node is not propagated to one or more nodes downstream from the at least one node in the data flow graph during the forward path, and / or the processing layout associated with the at least one node is not propagated to one or more nodes upstream from the at least one node in the data flow graph during the reverse path, After determining the processing layout of each node in the data flow graph, further including executing the data flow graph according to the processing layout determined for each node in the data flow graph, a method. **Claim 2** The method according to claim 1, further including determining whether the data flow graph is executed as a micrograph based on the processing layouts of the nodes within the first set of nodes, the second set of nodes, and the third set of nodes. **Claim 3** Determining whether the data flow graph is executed as a micrograph is Determining to execute the dataflow graph as a micrograph when the processing layouts of the first set of nodes, the second set of nodes, and the third set of nodes have the same degree of parallelism, or, Determining whether the dataflow graph is to be executed as a micrograph, The method of claim 2, including determining to execute the dataflow graph as a micrograph when the processing layouts of the first set of nodes, the second set of nodes, and the third set of nodes, excluding the at least one node, have the same degree of parallelism.

4. The third set of nodes includes a first node, the plurality of edges includes a first edge between the first node and a second node that precedes the first node in the dataflow graph, and determining the one or more initial processing layouts of the one or more nodes within the third set of nodes, The method of claim 1, including determining a first initial processing layout of the first node based on a second initial processing layout determined for the second node.

5. The plurality of edges includes a second edge between the first node and a third node that precedes the first node in the dataflow graph, a third initial processing layout is associated with the third node, and determining the first initial processing layout of the first node, The method of claim 4, including selecting one of the second initial processing layout determined for the second node or the third initial processing layout determined for the third node as the first initial processing layout.

6. The second initial processing layout specifies a first degree of parallelism, the third initial processing layout specifies a second degree of parallelism different from the first degree of parallelism, and the selection is, Selecting the second initial processing layout when the first degree of parallelism is greater than the second degree of parallelism, and Selecting the third initial processing layout when the first degree of parallelism is less than the second degree of parallelism, or, Each of the second initial processing layout and the third initial processing layout specifies a parallel processing layout having the same or different degrees of parallelism, the first edge represents a data flow of a first number of data records, the second edge represents a data flow of a second number of data records, and the selection is selecting the second initial processing layout when the first number of data records is greater than the second number of data records; selecting the third initial processing layout when the first number of data records is less than the second number of data records, the method according to claim 5. **Claim 7** During the determination, a first processing layout is determined for a first node in a third set of nodes, the first processing layout specifying a first degree of parallelism, a second processing layout for a second node immediately preceding the first node in the graph specifies a second degree of parallelism different from the first degree of parallelism, the method further comprising configuring at least one node of the data flow graph to perform at least one repartitioning operation, the method according to claim 1. **Claim 8** During the determination, a first processing layout is determined for a first node in a third set of nodes, the first processing layout specifying a first degree of parallelism, a second processing layout for a second node immediately preceding the first node in the graph specifies a second degree of parallelism different from the first degree of parallelism, the method further comprising adding a new node between the first node and the second node to the data flow graph, the new node representing at least one repartitioning operation, the method according to claim 1. **Claim 9** The determination includes determining a first processing layout for a first node in a third set of nodes, the first node representing a first data processing operation, and determining the first processing layout is determining the degree of parallelism for performing the first data processing operation; identifying a set of one or more computing devices for performing the first data processing operation according to the determined degree of parallelism; The method according to claim 1, comprising

10. Determining the first processing layout includes Determining that a single processor is to be used to perform the first data processing operation, and Identifying a computing device for performing the first data processing operation. The method according to claim 9, comprising

11. Receiving a Structured Query Language (SQL) query, Generating a query plan from the SQL query, and Generating the data flow graph from the generated query plan. The method according to claim 1, further comprising

12. The determination of the processing layout of the nodes in the third set of nodes is In a forward pass starting from the nodes in the first set of nodes, according to the structure of the data flow graph, and using the first set of the processing layout and the one or more layout determination rules, for each specific node of at least some of the nodes in the third set of nodes, the initial processing layout of each specific node is determined such that, for each specific node, the processing layout of the nodes preceding the specific node in the data flow graph is selected as the initial processing layout for the specific node during the forward pass. When there are multiple nodes preceding the specific node in the data flow graph, As indicated by the at least one rule, When the parallel processing layout of one of the plurality of preceding nodes is such that the processing layouts of the other nodes among the plurality of preceding nodes are sequential, the initial processing layout of the specific node is selected during the forward pass, or Determining that the processing layout of the one of the plurality of preceding nodes used to process the maximum number of records is selected as the initial processing layout of the specific node during the forward pass. The method according to claim 1, comprising

13. The method according to claim 12, wherein when the plurality of nodes preceding the specific node in the data flow graph have only sequential processing layouts or only a plurality of parallel processing layouts having the same or different degrees of parallelism from each other, the processing layout of one of the plurality of preceding nodes used to process the maximum number of records is selected as the initial processing layout during the forward pass.

14. The determination of the processing layout of the nodes in the third set of nodes is performed in a reverse pass starting from the nodes in the second set of nodes, and for each specific node of at least some of the nodes in the third set of nodes, according to the structure of the data flow graph, the initial processing layout, and the one or more layout determination rules, the final processing layout of each specific node is as indicated by the at least one rule, the parallel processing layout corresponding to one of the initial processing layout of the specific node or the processing layout of the node subsequent to the specific node is selected as the final processing layout of the specific node during the reverse pass when the other of the initial processing layout of the specific node and the processing layout of the node subsequent to the specific node is sequential, or further includes determining that the processing layout used to process the maximum number of records among the initial processing layout of the specific node and the processing layout of the node subsequent to the specific node is selected as the final processing layout of the specific node during the reverse pass.

15. The method according to claim 14, wherein when both the initial processing layout of the specific node and the processing layout of the subsequent nodes of the specific node have sequential processing layouts, or both have a plurality of parallel processing layouts but with the same or different degrees of parallelism from each other, the processing layout used to process the maximum number of records among the initial processing layout of the specific node and the processing layout of the nodes subsequent to the specific node is selected as the final processing layout of the specific node during the reverse path.

16. When selecting the final processing layout of the specific node during the reverse path, the processing layout associated with the at least one node is ignored, and / or The method according to claim 1, further comprising configuring the data flow graph to perform a repartitioning operation on data processed by adjacent nodes in the data flow graph that have processing layouts with different degrees of parallelism after performing the forward path and / or the reverse path.

17. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the method according to any one of claims 1 to 16.

18. At least one computer hardware processor, and At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform the method according to any one of claims 1 to 16, and A data processing system comprising.

Images