Dataflow graph optimization system and method
Dataflow graph optimization techniques address inefficiencies in automatically generated graphs by reducing redundant operations and enabling parallel processing, resulting in improved computational efficiency and reduced resource usage.
Patent Information
- Application Number
- JP2024075264
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-05-30
- Filing Date
- 2024-05-07
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2039-05-22
Smart Images

Figure 0007948989000001 
Figure 0007948989000002 
Figure 0007948989000003
Abstract
Description
Technical Field
[0001] The present invention relates to a system and method for data flow graph optimization.
Background Art
[0002] A data processing system may process data using one or more computer programs. One or more of the computer programs utilized by the data processing system may be developed as a data flow graph. A data flow graph may include components called "nodes" or "vertices" that represent data processing operations performed on input data, as well as links between components that represent the flow of data. The nodes of a data flow graph may include one or more input nodes representing each input data set, one or more output nodes representing each output data set, and one or more nodes representing data processing operations performed on the data. Techniques for performing computations encoded by data flow graphs are described in U.S. Patent No. 5,966,072 entitled "Executing Computations Expressed as Graphs" and U.S. Patent No. 7,716,630 entitled "Managing Parameters for Graph-Based Computations", each of which is incorporated herein in its entirety by reference.
Summary of the Invention
Means for Solving the Problems
[0003] Some embodiments involve obtaining an initial dataflow graph that is automatically generated by at least one computer hardware processor, and which includes a first plurality of nodes representing a first plurality of data processing operations, and a first plurality of links representing the flow of data between the nodes of the first plurality of nodes, and generating an updated dataflow graph by iteratively applying dataflow graph optimization rules to update the initial dataflow graph, wherein the updated dataflow graph represents a second plurality of data processing operations A data processing system is directed to include a second plurality of nodes representing a first data processing operation and a second plurality of links representing the flow of data between the nodes of the second plurality of nodes, wherein the second plurality of nodes include a node representing a first data processing operation and another node representing a second data processing operation, and executes an updated data flow graph by, at least partially, executing the first data processing operation using a first computer system process and executing the second data processing operation using a second computer system process different from the first computer system process. This includes at least one non-temporary computer-readable storage medium that stores processor-executable instructions.
[0004] Some embodiments are directed to a method that includes using at least one computer hardware processor to perform the following: obtaining an automatically generated initial dataflow graph, the initial dataflow graph comprising a first plurality of nodes representing a first plurality of data processing operations and a first plurality of links representing the flow of data between the nodes of the first plurality of nodes; generating an updated dataflow graph by iteratively applying dataflow graph optimization rules to update the initial dataflow graph, the updated dataflow graph comprising a second plurality of nodes representing a second plurality of data processing operations and a second plurality of links representing the flow of data between the nodes of the second plurality of nodes, the second plurality of nodes comprising a node representing a first data processing operation and another node representing a second data processing operation; and executing the updated dataflow graph by at least partially executing the first data processing operations using a first computer system process and executing the second data processing operations using a second computer system process different from the first computer system process.
[0005] Some embodiments, when executed by at least one computer hardware processor, involve obtaining an automatically generated initial dataflow graph on at least one computer hardware processor, wherein the initial dataflow graph includes a first plurality of nodes representing a first plurality of data processing operations and a first plurality of links representing the flow of data between the nodes of the first plurality of nodes, and generating an updated dataflow graph by iteratively applying dataflow graph optimization rules to update the initial dataflow graph, wherein the updated dataflow graph includes a second plurality of data processing operations A processor-executable instruction is stored on at least one non-temporary computer-readable storage medium, which includes a plurality of nodes and a second plurality of links representing the flow of data between the nodes of a second plurality of nodes, wherein the second plurality of nodes include a node representing a first data processing operation and another node representing a second data processing operation, and executes an updated data flow graph by, at least partially, executing the first data processing operation using a first computer system process and executing the second data processing operation using a second computer system process different from the first computer system process.
[0006] Several embodiments include at least one non-temporary computer-readable storage medium storing processor executable instructions, wherein the processor executable instructions are means for obtaining an automatically generated initial dataflow graph, the initial dataflow graph comprising a first plurality of nodes representing a first plurality of data processing operations and a first plurality of links representing the flow of data between the nodes of the first plurality of nodes, and means for generating an updated dataflow graph by iteratively applying dataflow graph optimization rules to update the initial dataflow graph, the updated dataflow graph comprising a second plurality of Means for generating an updated dataflow graph, which includes a second plurality of nodes representing data processing operations and a second plurality of links representing the flow of data between the nodes of the second plurality of nodes, wherein the second plurality of nodes encompass a node representing a first data processing operation and another node representing a second data processing operation; and means for executing the updated dataflow graph by at least partially executing the first data processing operation using a first computer system process and executing the second data processing operation using a second computer system process different from the first computer system process, directed to at least one non-temporary computer-readable storage medium.
[0007] Some embodiments are directed to a data processing system that includes at least one computer hardware processor and at least one non-temporary computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to: obtain a Structured Query Language (SQL) query; generate a query plan for the SQL query; use the query plan to generate an initial data flow graph, wherein the initial data flow graph includes a first plurality of nodes representing a first plurality of data processing operations; and use at least one data flow graph optimization rule to update the initial data flow graph, thereby generating an updated data flow graph, wherein the updated data flow graph includes a second plurality of nodes representing a second plurality of data processing operations.
[0008] Some embodiments are directed toward a method that includes using at least one computer hardware processor to: obtain a Structured Query Language (SQL) query; generate a query plan for the SQL query; use the query plan to generate an initial dataflow graph, wherein the initial dataflow graph includes a first plurality of nodes representing a first plurality of data processing operations; and use at least one dataflow graph optimization rule to update the initial dataflow graph, thereby generating an updated dataflow graph, wherein the updated dataflow graph includes a second plurality of nodes representing a second plurality of data processing operations.
[0009] Several embodiments are directed to at least one non-temporary computer-readable storage medium that stores processor-executable instructions causing at least one computer hardware processor to: obtain a Structured Query Language (SQL) query; generate a query plan for the SQL query; use the query plan to generate an initial dataflow graph, wherein the initial dataflow graph includes a first plurality of nodes representing a first plurality of data processing operations; and use at least one dataflow graph optimization rule to update the initial dataflow graph, thereby generating an updated dataflow graph, wherein the updated dataflow graph includes a second plurality of nodes representing a second plurality of data processing operations.
[0010] Several embodiments include at least one non-temporary computer-readable storage medium storing processor-executable instructions, the processor-executable instructions being directed to at least one non-temporary computer-readable storage medium which includes means for obtaining a Structured Query Language (SQL) query, means for generating a query plan for the SQL query, means for generating an initial dataflow graph using the query plan, wherein the initial dataflow graph includes a first plurality of nodes representing a first plurality of data processing operations, and means for generating an updated dataflow graph by using at least one dataflow graph optimization rule to update the initial dataflow graph, wherein the updated dataflow graph includes a second plurality of nodes representing a second plurality of data processing operations.
[0011] The above is a non-limiting summary of the present invention as defined by the attached claims.
[0012] Various aspects and embodiments will be described with reference to the following drawings. It should be understood that these drawings are not necessarily drawn to a consistent scale. Items appearing in multiple drawings are indicated by the same or similar reference numbers in all drawings in which they appear. [Brief explanation of the drawing]
[0013] [Figure 1A] This is a block diagram of a computing environment for illustration purposes in which several embodiments of the technology described herein may operate. [Figure 1B] This is a flowchart illustrating the process of automatically generating a data flow graph from an input Structured Query Language (SQL) query, according to several embodiments of the technology described herein. [Figure 2] This is a flowchart of Process 200 for illustration, which automatically generates a data flow graph from an input SQL query according to several embodiments of the technology described herein. [Figure 3A] This specification demonstrates how to apply optimization rules to a data flow graph for illustrative purposes in order to eliminate one or more redundant data processing operations in some embodiments of the technology described herein. [Figure 3B] In some embodiments of the techniques described herein, we demonstrate how to change the order of exchange data processing operations to facilitate the application of optimization rules to data flow graphs for alternative explanations. [Figure 3C] This specification demonstrates how, in some embodiments of the technology described herein, optimization rules are applied to a dataflow graph to eliminate one or more redundant data processing operations. [Figure 3D] This specification demonstrates how some embodiments of the technology described herein apply optimization rules to a dataflow graph to eliminate one or more redundant data processing operations. [Figure 4A]This specification demonstrates how to apply optimization rules to a dataflow graph for explanation in order to perform intensity reduction optimization using several embodiments of the technology described herein. [Figure 4B] This specification demonstrates how to apply optimization rules to a dataflow graph for explanation in order to perform a different intensity reduction optimization using several embodiments of the techniques described herein. [Figure 5A] This specification demonstrates how to apply optimization rules to a dataflow graph for explanation in order to perform integrated operation optimization using several embodiments of the technology described herein. [Figure 5B] This specification demonstrates how to apply optimization rules to a dataflow graph to perform integrated operation optimization using several embodiments of the technology described herein. [Figure 5C] This specification demonstrates how some embodiments of the technology described herein apply optimization rules to a dataflow graph to perform integrated operation optimization, as further explained below. [Figure 5D] This specification demonstrates how some embodiments of the technology described herein apply optimization rules to a dataflow graph to perform integrated operation optimization, as further explained below. [Figure 6] This specification demonstrates how some embodiments of the techniques described herein apply optimization rules to a dataflow graph for illustrative purposes to eliminate unreferenced data processing behaviors. [Figure 7] This specification demonstrates how to apply optimization rules to a data flow graph for explanation in order to perform width reduction optimization according to several embodiments of the technology described herein. [Figure 8A] This specification demonstrates how to apply optimization rules to a dataflow graph for explanation in order to perform deduplication optimization according to several embodiments of the technology described herein. [Figure 8B] This specification demonstrates how to apply optimization rules to a dataflow graph for deduplication optimization using zippering, according to several embodiments of the technology described herein. [Figure 9]shows that optimization rules are applied to an illustrative data flow graph to perform sequential-parallel optimization, in accordance with some embodiments of the techniques described in this specification. [Figure 10A] shows an initial data flow graph, in accordance with some embodiments of the techniques described in this specification. [Figure 10B] shows an updated data flow graph obtained by iteratively applying optimization rules to the initial data flow graph shown in FIG. 10A, in accordance with some embodiments of the techniques described in this specification. [Figure 10C] shows another representation of the initial data flow graph of FIG. 10A, in accordance with some embodiments of the techniques described in this specification. [Figure 10D] shows another representation of the updated data flow graph of FIG. 10B, in accordance with some embodiments of the techniques described in this specification. [Figure 11] is a block diagram of an illustrative computing system environment that can be used to implement some embodiments of the techniques described in this specification. [Detailed Description of the Invention]
[0014] Aspects of the techniques described in this specification relate to improving the speed, throughput, and accuracy of data processing systems by improving upon conventional techniques for performing data processing operations using data flow graphs.
[0015] Some data processing systems use data flow graphs to process data. Often, data flow graphs are automatically generated rather than manually specified. For example, some data processing systems can automatically generate data flow graphs from Structured Query Language (SQL) queries. In such cases, a user or computer program can provide the data processing system with an input SQL query, which can then generate a data flow graph from the SQL query and execute the SQL query by running the generated data flow graph. As another example, a data processing system may receive a representation of an input query (not the SQL query itself) from a user or computer program and automatically generate a data flow graph from the query representation. Yet another example is when a data processing system receives an input data flow graph from another data processing system. An input data flow graph may not be suitable for execution in this data processing system (even if it is suitable for execution in another data processing system). Therefore, the data processing system generates a new data flow graph from the input data flow graph that is suitable for execution in its own system.
[0016] The inventors recognized that conventional techniques for automatically generating dataflow graphs (e.g., from SQL queries, other query expressions, or input dataflow graphs as described above) can be improved. While automatically generated dataflow graphs can be executed in data processing systems, conventional techniques for automatically generating dataflow graphs result in dataflow graphs that require significant computational resources (e.g., processor resources, memory resources, network resources, etc.) and considerable time for execution. For example, an automatically generated dataflow graph may (1) include nodes representing redundant data processing operations, (2) require performing data processing operations whose results will not be used later, (3) require sequential processing unnecessarily when parallel processing is possible, (4) apply data processing operations to more data than is required to obtain the desired result, (5) generate computations across multiple nodes, which significantly increases the computational cost of performing computations where data processing at each dataflow graph node is performed by a dedicated thread in a computer program, a dedicated computer program (e.g., a process in an operating system), and / or a dedicated computing device, (6) require performing stronger data processing operations that require more computation (e.g., sort operations, rollup operations, etc.) when weaker data processing operations that require less computation (e.g., sort-within-groups operations, rollup-within-groups operations, etc.) would suffice, and / or (7) require the duplication of processing results.
[0017] The inventors have further recognized that conventional automated techniques for generating dataflow graphs can produce dataflow graphs that require significant computational resources and considerable time to execute, even if some conventional optimization techniques are used as part of the dataflow graph generation process. For example, a dataflow graph can be generated from an SQL query by generating a query plan from the SQL query, and then generating a dataflow graph from the generated query plan. However, even if the query plan generation involves certain optimizations, the resulting dataflow graph (generated from the query plan) can still require significant computational resources. In fact, conventional techniques for generating dataflow graphs from query plans often introduce inefficiencies and are not sophisticated enough to produce dataflow graphs that can be executed in a computationally efficient manner.
[0018] The inventors recognized that if automatically generated dataflow graphs were further processed and optimized to reduce the amount of computing resources used to execute the generated dataflow graphs, the performance of the data processing system would be improved. Some of the dataflow graph optimization techniques described herein were developed by the inventors for this purpose. The dataflow graph optimization techniques described herein improve the performance of the data processing system (e.g., throughput, speed, accuracy, etc.) by reducing the amount of computing resources (e.g., processor resources, memory resources, network resources, etc.) used to execute the dataflow graphs generated by at least partially using dataflow graph optimization techniques.
[0019] Another advantage of the dataflow graph optimization techniques described herein is that the presence of a dataflow graph optimizer as part of the data processing system allows developers of other data processing system components and / or users of the data processing system to rely on the dataflow graph optimizer rather than attempting ad-hoc optimizations as part of their own work. This not only helps reduce the amount of work that must be done by such developers and / or users, but also prevents them from unintentionally introducing errors into the data processing system, which also, of course, improves the data processing system by reducing the number of errors.
[0020] It should be understood that the dataflow graph optimization techniques described herein can (but not always) produce an "optimal" dataflow graph. More precisely, optimization techniques generally attempt to improve the performance of a data processing system by modifying the dataflow graph before it is executed in order to improve the computational efficiency of its execution.
[0021] Some of the embodiments described herein address all of the above-mentioned problems that the inventors have recognized with respect to prior art for automatically generating dataflow graphs. However, not all embodiments described herein address each of these problems, and some embodiments may not address any of them. Therefore, it should be understood that the embodiments of the technology described herein are not limited to addressing all or any of the above-mentioned problems of prior art for automatically generating dataflow graphs. For example, some embodiments of the technology described herein can be applied to optimize manually specified dataflow graphs (for such dataflow graphs may also contain inefficiencies and require more computing resources than necessary).
[0022] Accordingly, some embodiments provide novel techniques for automatically generating dataflow graphs from SQL queries and / or other inputs. Examples of such other inputs are provided herein. In some embodiments, a data processing system may (1) obtain a Structured Query Language (SQL) query, (2) generate a query plan for the SQL query, (3) generate an initial dataflow graph using the query plan, and (4) generate an updated dataflow graph using at least one dataflow graph optimization rule to update the initial dataflow graph.
[0023] In some embodiments, the updated dataflow graph may be saved (for example, to non-volatile memory) for later use. Additionally or alternatively, in some embodiments, the updated dataflow graph may be executed by a data processing system. Prior to execution, the data processing system may assign a processing layout to each of one or more nodes of the updated dataflow graph.
[0024] In some embodiments, the initial dataflow graph may contain a first set of nodes representing each of several data processing operations that would be performed if the initial dataflow graph were executed by the data processing system. The updated dataflow graph may contain a second set of nodes representing each of several data processing operations that would be performed if the updated dataflow graph were executed by the data processing system. In some embodiments, the second set of nodes has fewer nodes than the first set of nodes. In such embodiments, the number of nodes in the updated dataflow graph is less than the number of nodes in the initial dataflow graph. The reduction in the number of nodes in the graph can reduce the amount of computing resources required to run the updated dataflow graph compared to the computing resources that would have been used by the data processing system if the initial dataflow graph were run instead.
[0025] In some embodiments, a data processing system may generate an updated data flow graph from an initial data flow graph by applying one or more data flow graph optimization rules (examples of which are provided herein) to one or more portions of the initial data flow graph. The optimization rules may be applied iteratively. For example, in some embodiments, a data processing system may update an initial data flow graph by (1) selecting a first optimization rule, (2) identifying a first portion of the initial data flow graph to which the first optimization rule applies, and (3) applying the first optimization rule to the first portion of the initial data flow graph. Subsequently, the data processing system may continue updating the initial data flow graph by (1) selecting a second optimization rule different from the first optimization rule, (2) identifying a second portion of the initial data flow graph to which the second optimization rule applies, and (3) applying the second optimization rule to the second portion of the initial data flow graph.
[0026] There are several ways in which optimization rules can be applied to an updated initial dataflow graph. For example, in some embodiments, for each particular optimization rule, the data processing system may identify the portion of the dataflow graph to which the particular optimization rule can be applied and apply the optimization rule to the identified portion. As another example, in some embodiments, for each particular portion of the dataflow graph, the data processing system may identify optimization rules that can be applied to that portion and apply the identified optimization rules to that portion. In such embodiments, the initial dataflow graph may be topologically sorted, and the topologically sorted graph may be traversed (e.g., from left to right) to identify the particular portions to which the optimization rules can be applied.
[0027] In some embodiments, a data processing system may use a dataflow subgraph pattern matching language to identify one or more portions of an initial dataflow graph to which one or more optimization rules apply. The dataflow subgraph pattern matching language may encompass one or more expressions representing each pattern identified in the dataflow graph. Examples of such expressions are provided herein.
[0028] In some embodiments, to identify the portion of the initial dataflow graph to which a particular optimization rule applies, the data processing system may decide whether to swap one or more nodes with one or more other nodes. That is, the data processing system may decide whether the order in which one or more nodes appear in the dataflow graph can be changed without altering the processing results. This is a beneficial feature because, when multiple swapped nodes exist, changing the order of at least some of the swapped nodes can make the optimization rule applicable to parts of the graph to which the rule would otherwise not be applicable.
[0029] For example, an optimization rule may involve identifying two adjacent nodes in the initial dataflow graph that represent sorting operations, such that the second sorting operation neutralizes the effect of the first operation so that the first operation should be withdrawn (see, for example, the examples shown in Figures 3B and 3C). By definition, such an optimization rule does not apply to a dataflow graph that does not have adjacent nodes representing sorting operations. However, if the first node representing the first sorting operation is to be swapped with one or more other nodes, it may be possible to change the order of the first node with at least one of the one or more other nodes so that the first node representing the first sorting operation is placed adjacent to the second node representing the second sorting operation. As a result of swapping nodes in this way, an optimization rule that eliminates a redundant first sorting operation may be applied to the dataflow graph. Thus, in some embodiments, identifying a first portion of the initial dataflow graph may involve identifying a first node that represents a sorting data processing operation to be swapped with a data processing operation represented by a second node connected to the first node.
[0030] In some embodiments, one or more of several types of optimization rules may be applied when generating an updated dataflow graph from an initial dataflow graph. For example, but not limited to, applying optimization rules to an initial dataflow graph may involve removing one or more redundant data processing operations, removing one or more unreferenced data processing operations, performing one or more intensity reduction optimizations, one or more integrated operation optimizations, one or more width reduction optimizations, and / or one or more deduplication optimizations.
[0031] In some embodiments, optimization rules may be embodied in program code that, when executed, causes the corresponding optimization to be performed on the dataflow graph. For example, an optimization rule for eliminating redundancy may be embodied in program code that, when executed, results in the removal (from the dataflow graph to which the rule applies) of at least one node representing a data processing operation determined to be redundant. The program code can be written in any programming language (the aspects of the technology described herein are not limited in this respect).
[0032] As yet another example, an optimization rule that removes one or more unreferenced data processing operations can be embodied in program code that, when executed, results in the removal (from the data flow graph to which the rule applies) of at least one node representing a data processing operation whose results are not referenced and / or used (e.g., a sorted data processing operation that is not referenced because the order resulting from sorting is not needed or relied upon in subsequent processing).
[0033] As yet another example, an optimization rule that reduces strength can be embodied in program code such that, when executed, it results in replacing a first node representing a first data processing operation (e.g., a node representing a sort data processing operation) with a second node representing a second data processing operation of a weaker type than the first data processing operation (e.g., a node representing an intra-group sort data processing operation) in the data flow graph to which the rule is applied.
[0034] As yet another example, an optimization rule that performs integrated operation optimization can be embodied in program code that, when executed, results in replacing multiple nodes representing multiple operations (in the data flow graph to which the rule applies) with a single node representing the integration of those operations.
[0035] As yet another example, an optimization rule that performs width reduction optimization may be embodied in program code such that, when executed, it results in the deletion of some data (e.g., one or more data columns) in a particular part of the graph prior to the execution of subsequent operations, because that data (i.e., the data to be deleted) is not used in subsequent operations and does not need to be propagated as part of the process. As yet another example, a node in a dataflow graph may be configured to perform several calculations, some of which may not be used. Thus, in some embodiments, an optimization rule that performs width reduction optimization may be embodied in program code such that, when executed, a particular node is replaced with another node configured to perform only calculations whose results are used, and the unnecessary calculations are no longer performed.
[0036] As yet another example, an optimization rule that performs deduplication optimization can be embodied in program code such that, when executed, it results in the merging of multiple different branches of the dataflow graph to which the rule applies.
[0037] It should be understood that the techniques described above and further detailed below can be implemented in any of many ways (these techniques are not limited to any particular way of implementation of the embodiments). Examples of the details of the embodiments are provided herein for illustrative purposes only. Furthermore, the techniques disclosed herein can be used individually or in any appropriate combination (the aspects of the techniques described herein are not limited to the use of any particular technique or combination of techniques).
[0038] Figure 1A is a diagram of a data processing system 100 for illustrative purposes, according to several embodiments of the technology described herein. As shown in Figure 1A, the data processing system 100 comprises a query input module 104, a query plan generator 106, a data flow graph generator 108, a graph optimizer 110, a layout assignment module 112, and a graph execution engine 115.
[0039] The data processing system 100 is configured to access (e.g., read data from and / or write data to) data storage devices 102-1, 102-2, ... and 102-n. Each of the data storage devices 102-1, 102-2, ... and 102-n may store one or more datasets. The data storage devices may store any appropriate type of data in any appropriate manner. The data storage devices may store data as flat text files, as spreadsheets, using a database system (e.g., a relational database system), or in any appropriate manner. In some cases, the data storage devices may store transaction data. For example, the data storage devices may store credit card payment data, call log data, or bank transaction data. It should be understood that the data processing system 100 may be configured to access any appropriate number of any appropriate type of data storage devices (the aspects of the technology described herein are not limited in this respect). Data storage devices from which the data processing system 100 may read data may be referred to as data sources. A data storage device that can be configured for a data processing system 100 to write data to may be called a data sink.
[0040] In some embodiments, the data storage devices 102-1, 102-2, ... 102-n may be of the same type (for example, all may be relational databases) or of different types (for example, one may be a relational database, while another may be a data storage device that stores data in flat files. The data storage devices may be SQL Server data storage devices, Oracle data storage devices, Teradata data storage devices, flat file data storage devices, multi-file data storage devices, HADOOP data storage devices, DB2 data storage devices, Microsoft SQL Server data storage devices, Informix data storage devices, SAP data storage devices, MongoDB data storage devices, metadata data stores, and / or other appropriate types of data storage devices (the aspects of the technology described herein are not limited in this respect).
[0041] In some embodiments, the query input module 104 may be configured to receive input SQL queries. In some embodiments, the query input module 104 may be configured to receive input SQL queries from a user. For example, the query input module 104 may be configured to generate a graphical user interface through which a user can input SQL queries. As another example, the query input module 104 may be configured to receive information provided by the user using the graphical user interface (information not necessarily generated by the query input module 104 itself). In some embodiments, the query input module 104 may be configured to receive input SQL queries from another computer program. For example, the query input module 104 may be configured to expose an application programming interface (API) through which input SQL queries can be provided (e.g., the Open Database Connectivity (ODBC) API and the Java Database Connectivity (JDBC) API), access SQL queries in response to a notification that an SQL query should be accessed, or otherwise receive input SQL queries from another computer program in an appropriate manner.
[0042] SQL queries received by the query input module 104 may involve reading data from a single data storage device and / or writing data to a single data storage device. Alternatively, SQL queries received by the query input module 104 may involve reading data from multiple data storage devices and / or writing data to multiple data storage devices. When the multiple data storage devices are of different types, the SQL query may be called a federated SQL query. In some embodiments, the SQL query may involve reading data from a federated database and / or writing data to a federated database.
[0043] In some embodiments, the query plan generator 106 is configured to generate a query plan from an SQL query received by the query input module 104. The generated query plan may identify one or more data processing operations that should be performed if the SQL query were executed. The generated query plan may further specify the order in which the identified data processing operations should be performed. Thus, the generated query plan may represent a sequence of data processing operations that should be performed to execute the SQL query received by the query input module 104. The query plan generator 106 may be configured to generate a query plan in any appropriate manner. For example, in some embodiments, the query plan generator 106 may implement any of the techniques for generating query plans described in U.S. Patent No. 9,116,955, titled "Managing Data Queries," which is incorporated herein by reference in whole.
[0044] In some embodiments, the data flow graph generator 108 is configured to generate an initial data flow graph from a query plan generated by the query plan generator 106. The data flow graph generator 108 may be configured to generate an initial data flow graph from a query plan in any appropriate manner. For example, in some embodiments, the data flow graph generator 108 may implement any of the query plan generation techniques described in U.S. Patent No. 9,116,955, titled "Managing Data Queries," which is incorporated herein by reference in whole.
[0045] In some embodiments, a dataflow graph may include components called “nodes” or “points” that represent data processing operations to be performed on input data, and links between components that represent the flow of data. A node in a dataflow graph may include one or more input nodes representing each input dataset, one or more output nodes representing each output dataset, and one or more nodes representing data processing operations to be performed on the data. In some embodiments, input nodes may represent federated databases or other types of databases. Similarly, in some embodiments, output nodes may represent federated databases or other types of databases.
[0046] In some embodiments, multiple different data processing operations, represented by multiple different nodes in a dataflow graph, may be executed using different computer system processes. For example, a dataflow graph may include a first node representing a first data processing operation (e.g., a "sort" operation) and a second node representing a second data processing operation (e.g., a "join" operation) that is different from the first data processing operation. In some embodiments, the first computer system process may be used to execute the first data processing operation, and a second computer system process different from the first may be used to execute the second data processing operation. In some embodiments, the first and second computer system processes may run on the same computing device and, for example, be managed by the same operating system. In other embodiments, the first and second computer system processes may run on different computing devices.
[0047] In some embodiments, a computer system process used to perform data processing operations represented by nodes in a dataflow graph may be an instance of a computer program configured to execute processor-executable instructions that encode the data processing operations. The computer system process may be a single-threaded process or a multi-threaded process. The computer system process may be associated with one or more computer system resources that include, but are not limited to, processor-executable instructions that represent encoding the data processing operations, memory (e.g., areas of physical and / or virtual memory holding executable code, process-specific input and / or output data, call stacks, compute heaps, and / or other data), process identifiers (e.g., used by the operating system to identify the computer system process), security attributes (e.g., permissions indicating the ownership of one or more processes and / or operations that the computer system process is permitted to perform), and / or information specifying the state of the computer system process.
[0048] In some embodiments, an initial dataflow graph may be generated from a query plan by generating an initial dataflow graph that at least partially encompasses each node of at least a subset (e.g., some or all) of the data processing operations identified in the query plan. Subsequently, links connecting the nodes in the initial dataflow graph may be generated using the order of data processing operations specified in the query plan. For example, if the generated query plan indicates that a first data processing operation takes place before a second data processing operation, the generated initial dataflow graph may have a first node (representing the first data processing operation) and a second node (representing the second data processing operation), and one or more links specifying the path from the first node to the second node.
[0049] In some embodiments, generating an initial dataflow graph from a query plan involves adding one or more nodes to the graph that represent input and / or output data sources. For example, generating an initial dataflow graph may involve adding an input node for each data source from which data records should be read during the execution of an SQL query. Each input node may be configured with parameter values associated with its respective data source. These values may indicate how to access data records in the data source. As another example, generating an initial dataflow graph may involve adding an output node for each data sink to which data records should be written during the execution of an SQL query. Each output node may be configured with parameter values associated with its respective data sink. These values may indicate how to write data records to the data source. In some embodiments, the initial dataflow graph may be runnable by a graph execution engine. In other embodiments, the initial dataflow graph may not be runnable by a graph execution engine.
[0050] In some embodiments, the graph optimizer 110 is configured to generate an updated dataflow graph using one or more dataflow graph optimization rules to update the initial dataflow graph generated by the dataflow graph generator 108. The graph optimizer 110 may be configured to apply one or more of the numerous types of optimization rules described herein to the initial dataflow graph. For example, the graph optimizer 110 may be configured to update the initial dataflow graph by removing one or more redundant data processing operations, removing one or more unreferenced data processing operations, performing one or more intensity reduction optimizations, performing one or more integrated operation optimizations, performing one or more width reduction optimizations, and / or performing one or more deduplication optimizations. The graph optimizer 110 may be configured to operate in any appropriate manner, for example, according to process 200 for the description described with reference to Figure 2, or one or more other forms thereof.
[0051] In some embodiments, the layout assignment module 112 may determine the processing layout for each of the one or more data processing operations represented by each node in the updated dataflow graph generated by the graph optimizer 110. The processing layout for a data processing operation can specify how many computing devices should be used to perform the data processing operation and can identify the specific computing devices that should be used to perform the data processing operation. Thus, in some embodiments, the layout assignment module 112 may determine, for each of the one or more nodes in the updated dataflow graph, whether the data processing operation should be performed using a single device (e.g., a single processor, a single virtual machine, etc.) or multiple devices (e.g., multiple processors, multiple virtual machines, etc.), and which devices should be used. In some embodiments, the layout assignment module may assign different degrees of parallelism to multiple different nodes in the updated dataflow graph. Therefore, it should be understood that different processing layouts may be assigned to multiple different data processing operations performed during the execution of the updated dataflow graph generated by the graph optimizer 110.
[0052] In some embodiments, the updated dataflow graph may contain multiple (e.g., two or more) nodes representing multiple different data processing operations, and multiple different processes may be used to perform these data processing operations. For example, one or more computer system processes may be used to perform the data processing operations represented by a first node (for example, multiple computer system processes may be used if the data processing operations are parallelized), and one or more other computer system processes may be used to perform data processing operations represented by a second node in the updated dataflow graph, which is different from the first node.
[0053] In some embodiments, the graph execution engine 115 is configured to execute one or more dataflow graphs. For example, in some embodiments where an initial dataflow graph is runnable, the graph execution engine 115 may be configured to execute any initial dataflow graph generated by the dataflow graph generator 108. As another example, the graph execution engine 115 may be configured to execute any updated dataflow graph generated by the graph optimizer 110. The graph execution engine may include a collaborative system or other appropriate execution environment for executing dataflow graphs. Aspects of the environment for developing and executing dataflow graphs are described in U.S. Patent No. 5,966,072, titled “Executing Computations Expressed as Graphs,” and U.S. Patent No. 7,716,630, titled “Managing Parameters for Graph-Based Computations,” which are incorporated herein by reference in their entirety.
[0054] Figure 1B is a flowchart of explanatory process 120, which automatically generates a data flow graph from an input Structured Query Language (SQL) query according to several embodiments of the technology described herein. Process 120 can be performed by any suitable data processing system, for example, by the data processing system 100 described with reference to Figure 1A.
[0055] Process 120 is initiated in Act 122, when an SQL query is received. The SQL query may be received using the query input module 104. This can be done in any appropriate manner, including any of the methods described with reference to Act 202 of Process 200.
[0056] Next, process 120 proceeds to act 124, in which a query plan is generated from the SQL query received in act 122. The query plan may be generated using the query plan generator 106. This can be done in any appropriate manner, including any of the methods described with reference to act 204 of process 200.
[0057] Next, process 120 proceeds to act 126, in which an initial data flow graph is generated from the query plan obtained in act 124. The initial query plan may be generated by the data flow graph generator 108. This can be done in any appropriate manner, including any of the methods described with reference to act 206 of process 200.
[0058] Next, process 120 proceeds to act 128, in which an updated dataflow graph is generated from the initial dataflow graph by applying one or more optimization rules to the initial dataflow graph. The updated dataflow graph may be generated by the graph optimizer 110. This can be done in any appropriate manner, including any of the methods described with reference to act 207 of process 200.
[0059] The updated dataflow graph may be saved for later use or executed by a data processing system. Prior to execution, a processing layout may be assigned to one or more data processing operations represented by nodes in the updated dataflow graph. The processing layout may be assigned to data processing operations by the layout assignment module 112.
[0060] Figure 2 is a flowchart of an illustrative process 200 that automatically generates a data flow graph from an input SQL query, according to several embodiments of the technology described herein. Process 200 can be performed using any suitable data processing system, including, for example, the data processing system 100 described with reference to Figure 1A.
[0061] Process 200 is initiated in Act 202, in which an SQL query is received. In some embodiments, the SQL query may be received by the data processing system executing Process 200 as a result of a user providing the SQL query as input to the data processing system. The user may input the SQL query using a graphical user interface or any other appropriate type of interface. In other embodiments, the SQL query may be provided to the data processing system by another computer program. For example, the SQL query may be provided by a computer program configured to cause the data processing system to execute one or more SQL queries (each SQL query may be specified by a user or automatically generated). The SQL query may be of any appropriate type and may be provided in any appropriate format (the aspects of the technology described herein are not limited in this respect).
[0062] Next, process 200 proceeds to act 204, in which a query plan is generated from the SQL query received in act 202. The generated query plan may identify one or more data processing operations that will be performed when the SQL query is executed. The generated query plan may further specify the order in which the identified data processing operations are performed. Thus, the generated query plan may represent the sequence of data processing operations to be performed to execute the SQL query received in act 202. The generated query plan may be generated using any appropriate type of query plan generator (e.g., query plan generator 106). Several exemplary techniques for generating query plans are described in U.S. Patent No. 9,116,955, titled "Data Query Management," which is incorporated herein by reference in whole.
[0063] Next, process 200 proceeds to act 206. An initial dataflow graph is generated from the query plan generated in act 204 using the SQL query received in act 202. In some embodiments, the initial dataflow graph may be generated from the query plan by generating the initial dataflow graph to include, at least partially, each node of at least a subset (e.g., some or all) of the data processing operations identified in the query plan. In some embodiments, a single node in the query plan may result in the inclusion of multiple nodes in the initial dataflow graph. Subsequently, links connecting the nodes in the initial dataflow graph may be generated using the order of data processing operations specified in the query plan. For example, if the generated query plan indicates that a first data processing operation takes place before a second data processing operation, the generated initial dataflow graph may have a first node (representing the first data processing operation) and a second node (representing the second data processing operation), and one or more links specifying the path from the first node to the second node.
[0064] In some embodiments, generating an initial dataflow graph from a query plan involves adding one or more nodes to a graph representing input and / or output data sources. For example, generating an initial dataflow graph may involve adding an input node for each data source from which data records are read during the execution of an SQL query. Each input node may be configured with parameter values associated with its respective data source. These values may indicate how to access data records in the data source. As another example, generating an initial dataflow graph may involve adding an output node for each data sink to which data records are written during the execution of an SQL query. Each output node may be configured with parameter values associated with its respective data sink. These values may indicate how to write data records to the data source.
[0065] It should be noted that the initial dataflow graph generated in Act 206 is different from the query plan generated in Act 204. While the dataflow graph can be executed by a graph execution engine (e.g., graph execution engine 115), the query plan cannot be executed by the graph execution engine (it is an intermediate representation used to generate the dataflow graph, which is then executed by the graph execution engine to execute SQL queries). The query plan is not executable and requires further processing to generate an execution strategy, even under the context of a relational database management system. In contrast, the dataflow graph is executable by the graph execution engine to execute SQL queries. In addition, even after further processing by the relational database system, the resulting execution strategy does not enable reading and / or writing data to other types of data sources and / or data sinks, although the dataflow graph is not limited in this respect.
[0066] In some embodiments, the initial dataflow graph generated in Act 206 may include nodes representing data processing operations that are not present in the query plan generated in Act 204. Conversely, in some embodiments, the initial dataflow graph generated in Act 206 may not include nodes representing data processing operations that are present in the query plan generated in Act 204. Such situations can arise due to various optimizations that may be performed during the process of generating the dataflow graph from the query plan. In some embodiments, the initial dataflow graph generated in Act 206 may include nodes representing data processing operations other than database operations performed on a database computer system (e.g., a relational database management system).
[0067] In some embodiments, the query plan and the dataflow graph may be implemented using different types of data structures. For example, in some embodiments, the query plan may be implemented using a directed graph (e.g., a tree such as a binary tree) where each node has a single parent node, while the dataflow graph may be implemented using a directed acyclic graph where at least one node may have multiple parent nodes.
[0068] Next, process 200 proceeds to act 207, where the initial dataflow graph is updated to obtain the updated dataflow graph. This can be done in any of several ways. For example, in the illustrated embodiment, the dataflow graph optimization rule is selected in act 208. Next, the data processing system running process 200 identifies the portion of the initial dataflow graph to which the optimization rule identified in act 208 applies. In act 212, the selected optimization rule is applied to the identified portion of the graph. Next, process 200 proceeds to decision block 214, where it is determined whether there is an optimization rule that applies to at least one more portion of the dataflow graph. If it is determined that there is an optimization rule that applies to at least one more portion of the graph (for example, the optimization rule selected in act 208 may be applied to a different portion of the graph than the portion identified in act 210, or a different optimization rule may be selected overall), process 200 returns to act 208. Otherwise, process 200 proceeds to act 216.
[0069] In some embodiments, for each specific optimization rule selected in Act 208, the data processing system may identify the portion of the dataflow graph to which the selected optimization rule is applicable and apply the optimization rule to the identified portion. Once all such portions have been identified, different optimization rules may be selected. However, previously applied optimization rules may also be selected so that the application of the same optimization rule to the dataflow graph may be considered multiple times (this may result in a more optimized dataflow graph compared to a method in which optimization rules are greedily selected and not used after being applied once). Optimization rules may be selected in any appropriate order (the aspects of the technique described herein are not limited in this respect). For example, after deduplication optimization, any nodes representing redundant operation may be removed, and any empty nodes may be removed. After removing empty nodes, width reduction optimization may be performed, etc.
[0070] In some embodiments, the order of acts 208 and 210 may be changed. In such embodiments, the data processing system may first identify a portion of the dataflow graph and then select an optimization rule that can be applied to the identified portion of the dataflow graph. In such embodiments, the initial dataflow graph may be topologically sorted, and the topologically sorted graph may be traversed (e.g., from left to right) to identify specific portions to which the optimization rule can be applied.
[0071] In some embodiments, a data processing system may use a dataflow subgraph pattern matching language to identify one or more portions of an initial dataflow graph to which one or more optimization rules are applied. The dataflow subgraph pattern matching language may contain one or more expressions for identifying a particular type of subgraph in the dataflow graph. In some embodiments, a data processing system executing process 200 may be configured to use expressions of the subgraph pattern matching language to identify portions of the dataflow graph to which one or more optimization rules are applied. Certain expressions may facilitate the identification of one or more portions to which a particular one or more optimization rules are applied. In some embodiments, if a dataflow graph optimizer (e.g., graph optimizer 110) is configured with one or more new optimization rules, the graph optimizer may be configured with one or more new expressions written in the subgraph pattern matching language to facilitate the identification of portions of the dataflow graph to which the new optimization rules may be applied.
[0072] For example, a pattern matching language may contain an expression for identifying a series of nodes of at least a threshold length (e.g., at least 2, 3, 4, 5, etc.) that represent each series of computations that can be integrated using integrated behavior optimization rules and represented by a single node in the graph. Identifying such patterns can facilitate the application of integrated behavior optimization rules, which are further described below, including references in Figures 5A-5D. One non-restrictive example of such an expression is "A→B→C→D", which can help identify a series of four consecutive data processing operations that can be integrated.
[0073] As another example, a pattern matching language may contain expressions for identifying parts of a dataflow graph where certain types of nodes are interchangeable with other nodes. This can facilitate the application of multiple different types of optimization rules to a dataflow graph. When a data processing system determines that the order of one or more nodes in a dataflow graph can be changed without altering the processing results, this allows the data processing system to consider changes to the structure of the dataflow graph (allowed by the degrees of freedom gained by the swap operation) in order to identify parts where optimization rules can be applied. As a result of considering changes based on swaps, one or more optimization rules may become applicable to parts of the graph where these rules would otherwise not have been applicable.
[0074] For example, an optimization rule might involve identifying two adjacent nodes in the initial dataflow graph that represent each sorting operation, such that the second sorting operation negates the effect of the first operation so that the first operation should be withdrawn (see, for example, the examples shown in Figures 3B and 3C). By definition, such an optimization rule does not apply to a dataflow graph that does not have adjacent nodes representing sorting operations. However, if the first node representing the first sorting operation is swapped with one or more other nodes, it may be possible to rearrange the order of the first node with at least one of the other nodes so that the first node representing the first sorting operation is adjacent to the second node representing the second sorting operation. As a result of swapping nodes in this way, an optimization rule that eliminates a redundant first sorting operation may be applied to the dataflow graph.
[0075] Accordingly, in some embodiments, the subgraph matching language may include one or more expressions for identifying subgraphs of a dataflow graph in situations where the order of nodes in the dataflow graph may be changed. For example, the expression "A*→(...)→B" (where A and B can each be any appropriate data processing operation such as sorting or merging) may be used to find a portion of a dataflow graph having node "A" (i.e., a node representing operation "A") and node B (representing operation B), and one or more nodes between node A and node B that are swapped with node A (for example, the result of the processing performed by these nodes does not change if the order of the nodes is changed). Once such a portion is identified, the dataflow graph may be modified by moving node A so that it is adjacent to node B in order to obtain the portion "AB". In a particular example, suppose the dataflow graph has nodes ACDB, and if operation A is swapped with operations C and D, the dataflow graph may be modified to become "CDAB". The data processing system may then consider whether the optimization rule applies to the portion "AB". For example, if operation A is a sort and operation B is a sort, the data processing system can attempt to determine whether these two sorts can be replaced by a single sort, as in the example in Figure 5B.
[0076] As another example, the formula "A → (...) → B*" may be used to find a portion of a dataflow graph that has node A, a second node B, and one or more nodes between these nodes that are swapped with node B. In a particular example, if the dataflow graph has nodes ACDB, and if operation B is swapped with operations C and D, the dataflow graph may be modified to "ABCD". The data processing system may then consider whether the optimization rule applies to portion "AB".
[0077] As another example, the expression "A → (...) → B**" may be used to find a portion of a dataflow graph that has node A, node B, and one or more nodes between node A and node B (e.g., C and D) that are not swapped with node B. In this case, the system may attempt a "forced" swap (where nodes C and D are moved to the left of node A, if possible). As a particular example, suppose the dataflow graph has node ACEDB, and operation B is swapped with operation E, but not with operations C and D, then the dataflow graph may be modified to "CDABE" (B is swapped with E, but C and D are moved to the left of A).
[0078] As yet another example, the expression "A**→(...)→B" may be used to find a portion of a dataflow graph that has node A, node B, and one or more nodes between node A and node B that are not exchanged with node A (e.g., C and D). In this case, the system may attempt a "forced" exchange (where nodes C and D are moved to the right of node B, if possible). As a particular example, suppose the dataflow graph has node ACEDB, and operation A is exchanged with operation E, but not with operations C and D, then the dataflow graph may be modified to "EABCD" (node A is exchanged with E, but C and D are moved to the right of B).
[0079] It should be understood that the above examples of expressions for the subgraph matching language are for illustrative purposes only. In some embodiments, one or more other expressions may be part of the subgraph matching language in addition to, or instead of, the above examples.
[0080] In some embodiments, when generating an updated dataflow graph from the initial dataflow graph in Act 207, one or more of several types of optimization rules may be applied. For example, applying optimization rules to the initial dataflow graph may involve removing one or more redundant data processing operations, removing one or more unreferenced data processing operations, performing one or more intensity reduction optimizations, performing one or more integrated operation optimizations, performing one or more width reduction optimizations, and / or one or more deduplication optimizations.
[0081] In some embodiments, optimization rules may be embodied in program code that, when executed, causes the corresponding optimization to be performed on the dataflow graph. For example, an optimization rule for eliminating redundancy may be embodied in program code that, when executed, results in the removal (from the dataflow graph to which the rule applies) of at least one node representing a data processing operation determined to be redundant. Examples of applying optimization rules to a dataflow graph to eliminate one or more redundant data processing operations are shown in Figures 3A–3C, as will be described in more detail below.
[0082] As another example, an optimization rule for reducing intensity can be embodied in program code that, when executed, results in replacing a first node representing a first data processing operation (e.g., a node representing a sort data processing operation) with a second node representing a second data processing operation of a weaker type than the first data processing operation (e.g., a node representing an intra-group sort data processing operation) in the data flow graph to which the rule is applied. An example of applying an optimization rule to a data flow graph to perform intensity reduction optimization is shown in Figures 4A and 4B, as will be described in more detail below.
[0083] As another example, optimization rules for integrated behavior optimization can be embodied in program code that, when executed, result in replacing multiple nodes representing multiple actions (in the dataflow graph to which the rule is applied) with a single node representing the integration of these multiple actions. An example of applying optimization rules to a dataflow graph for integrated behavior optimization is shown in Figures 5A to 5D, as described in more detail below.
[0084] As yet another example, an optimization rule to remove one or more unreferenced data processing operations can be embodied in program code that, when executed, results in the removal (from the data flow graph to which the rule applies) of at least one node representing a data processing operation whose results are not referenced and / or used (for example, a sorted data processing operation that is not referenced because the order resulting from sorting is not needed or relied upon in subsequent processing). An example of applying such an optimization rule to a data flow graph is shown in Figure 6, as described in more detail below.
[0085] As another example, an optimization rule for width reduction optimization, when executed, can be embodied in program code that causes the deletion of some data (e.g., one or more data columns, data rows, etc.) in a specific part of the graph prior to the execution of subsequent operations, because that data (i.e., the data to be deleted) is not used in subsequent operations and does not need to be propagated as part of the process. An example of applying such an optimization rule to a dataflow graph is shown in Figure 7, which is described in more detail below.
[0086] As another example, an optimization rule for deduplication optimization can be embodied in program code that, when executed, causes multiple different branches of the dataflow graph to which the rule applies to be merged. An example of applying such an optimization rule to a dataflow graph is shown in Figures 8A and 8B, as will be described in more detail below.
[0087] As another example, optimization rules for sequential-parallel optimization can be implemented in program code that, when executed, causes sequential processes to be performed in parallel. An example of applying such optimization rules to a dataflow graph is shown in Figure 9, which is described in more detail below.
[0088] Please understand that the above optimization rules and optimizations are non-limiting examples for illustrative purposes. As part of process 200, one or more other optimization rules and / or optimizations may be applied to the initial data flow graph instead of, or in addition to, the above optimization rules and / or optimizations.
[0089] Next, process 200 proceeds to act 216, which outputs the updated dataflow graph. In some embodiments, in act 216, the updated dataflow graph may be saved (for example, to non-volatile memory) for later use.
[0090] In addition to, or instead of, being saved, the updated dataflow graph may be executed. In some embodiments in which the updated dataflow graph is executed, in act 218 of process 200, a processing layout is assigned to one or more nodes of the updated dataflow graph. The processing layout of a node representing a data processing operation can specify how many computing devices should be used to perform that data processing operation, and can identify the specific computing devices that should be used to perform that data processing operation. This can be done in any appropriate manner, including using any of the layout assignment techniques described in U.S. Patent Application No. 15 / 939,829, filed March 29, 2018, entitled "Systems and Methods for Performing Data Processing Operations Using Variable Level Parallelism," which is incorporated herein by reference in whole. In some embodiments, the decision of whether a node will be processed using a single computing device or multiple computing devices (e.g., whether parallel processing should be applied and what level of parallelism should be used) may be made in advance (e.g., during Act 207) (the specific computing device to be used for the computation is allocated in Act 218).
[0091] After the processing layout is assigned in Act 218, the updated data flow graph can be executed. For example, if process 200 is performed by a data processing system, the data processing system 100 can execute the updated data flow graph using the graph execution engine 115. In some embodiments, the updated data flow graph generated in Act 207 can be executed as soon as it is generated and without user input. In other embodiments, the updated data flow graph can be generated in Act 207, but its execution can only be started in response to a command to execute it, which can be provided by the user via an interface or by another computer program (e.g., via an API call).
[0092] It should be understood that process 200 is for illustrative purposes only and that multiple variations exist. For example, in some embodiments, the optional acts 218 and 220 may be omitted, and process 200 may complete after the updated dataflow graph has been generated and saved. In another example, process 200 may be used to optimize a dataflow graph provided from another source (e.g., another data processing system) rather than a dataflow graph generated from an input SQL query, as in the illustrated embodiment. In such embodiments, acts 202-204 may be omitted, and the initial dataflow graph may be generated in act 206 from a dataflow graph provided from another source. Such generation may involve transforming the received dataflow graph into a dataflow graph configured for the data processing system running process 200.
[0093] Referring to Figures 3A-39, an example illustrating the application of optimization rules to a dataflow graph is provided below. Each dataflow graph shown in these figures may be a subgraph of a larger dataflow graph that is being optimized (for example, as part of Act 207 of Process 200). For example, each of the one or more dataflow graphs shown in these figures may be a subgraph of an initial dataflow graph generated in Act 206 of Process 200, and / or a subgraph of one or more dataflow graphs obtained by transforming the initial dataflow graph as part of Act 207.
[0094] Figure 3A illustrates the application of optimization rules to the explanatory dataflow graph 300 to eliminate one or more redundant data processing operations. As shown in Figure 3A, the dataflow graph 300 includes a node 302 representing a repartitioning data processing operation (partitioning data for parallel processing on multiple different computing devices), followed by a node 304 representing a serialization operation (operating to consolidate all data for sequential processing by a single computing device). Since the effect of repartitioning is negated by the subsequent serialization command, it is not necessary to perform the repartitioning data processing operation. Therefore, the dataflow graph optimizer (e.g., the graph optimizer 110 in the data processing system 100) may remove the node 302 representing the repartitioning operation. As a result, portion 300 of the dataflow graph is transformed into portion 305.
[0095] Figure 3B illustrates how, in some embodiments of the technology described herein, the order of swapped data processing operations can be altered to facilitate the application of optimization rules to a data flow graph for a different explanation. As shown in Figure 3B, the data flow graph 310 includes a node 311 representing a sort data processing operation (sorting is performed with respect to key A), followed by one or more nodes (not shown), and then a node 312 representing another sort data processing operation (sorting is performed with respect to key B). In this example, determining whether the sort operation represented by node 311 is swapped with the data processing operation represented by the node between node 311 and node 312 can facilitate the application of one or more optimization rules to the data flow graph 310. For example, if the sort operation represented by node 311 is swapped with the data processing operation represented by the node between node 311 and node 312, altering the order of the nodes allows nodes 311 and 312 to be placed adjacent to each other, as shown in the data flow graph 313. This then allows us to consider whether one or more optimization rules can be applied to the resulting graph 313 (these rules may not have been applicable to the dataflow graph 310 before the sorting operation represented by node 311 changed the order in which they appear in the graph). For example, in this case, the effect of sorting with respect to key A (node 311) is negated by the subsequent re-sorting with respect to key B (node 312). Therefore, the sorting operation represented by node 311 is unnecessary and can be removed, as shown in Figure 3C, resulting in the dataflow graph 314.
[0096] Figure 3D illustrates the application of optimization rules to dataflow graph 320 for alternative explanation, in order to eliminate redundant data processing operations. As shown in Figure 3D, dataflow graph 320 includes node 322, which represents a sort operation with respect to key A. However, if the data is already sorted with respect to the same key, the dataflow graph optimizer can remove node 322, which represents the sort operation with respect to key A, resulting in dataflow graph 323. In all the examples shown in Figures 3A-3D, the removal of redundant nodes results in a dataflow graph whose execution consumes fewer computing resources than if these nodes were not removed.
[0097] Figure 4A illustrates how to perform de-intensity optimization and apply optimization rules to the explanatory dataflow graph 400 to obtain the dataflow graph 402. As shown in Figure 4A, the dataflow graph 400 contains a node 401 that represents a sorting operation where incoming data is sorted with respect to a primary key A (e.g., sorted by last name) and then sorted with respect to a secondary key B (e.g., sorted by first name for people with the same last name). However, if the graph optimizer detects that the incoming data to node 401 is already sorted with respect to key A (e.g., already sorted by last name), the graph optimizer performs de-intensity optimization by replacing this sorting operation with an in-group sorting operation (represented by node 403 in graph 402). This reduces the intensity of the sorting operation, but still obtains the same result and avoids unnecessary calculations.
[0098] Figure 4B shows that another intensity reduction optimization is performed and the optimization rule is applied to the explanatory dataflow graph 410 to obtain the dataflow graph 412. As shown in Figure 4B, the dataflow graph 410 contains node 411, which represents a rollup operation performed with respect to primary key A and secondary key B. However, if the graph optimizer detects that the data coming into node 411 is already sorted with respect to key A, the optimizer performs an intensity reduction optimization by replacing the rollup operation with a grouped rollup operation (represented by node 413 in graph 412), thereby reducing the intensity of the rollup operation but obtaining the same result and avoiding unnecessary calculations.
[0099] Figure 5A illustrates the application of optimization rules to the explanatory dataflow graph 500 in order to perform integrated operation optimization. As shown in Figure 5A, the dataflow graph 500 contains a sequence of nodes 502, 504, and 506, each representing its own computation. In some embodiments, during the execution of the dataflow graph, data processing operations represented by separate nodes may be performed by multiple different processes running on one or more computing devices. The graph optimizer can perform integrated operation optimization by replacing the sequence of three nodes with a single node (e.g., node 508 in dataflow graph 505) so that all operations are performed by a single process running on a single computing device, thereby reducing the overhead of inter-process (and potentially inter-device) communication.
[0100] Figure 5B illustrates the application of optimization rules to a dataflow graph 510 for alternative explanation in order to perform integrated operation optimization. As shown in Figure 5B, the dataflow graph 510 includes a node 512 representing a join operation on datasets A and B using keys A1 and B1, followed by a node 514 representing the output of the join operation represented by node 512 and a join operation on dataset C using keys A1 and C1. In this example, the graph optimizer can perform integrated operation optimization by replacing the two separate nodes representing each join operation with a single node representing the join operation on datasets A, B, and C using keys A1, B1, and C1, as shown by node 516 in the dataflow graph 515. In this way, the join process is performed by a single process running on a single computing device, which reduces the overhead of inter-process (and potentially inter-device) communication.
[0101] Figure 5C shows how optimization rules are applied to a dataflow graph 520 for alternative explanation in order to perform integrated operation optimization. As shown in Figure 5C, the dataflow graph 520 includes node 522 representing a filtering operation for key A, followed by node 524 representing another filtering operation for key B. In this example, the graph optimizer can perform integrated operation optimization by replacing the two separate nodes 522 and 524, each representing a different filtering operation, with a single node representing a filtering operation that filters for both key A and key B, as shown using node 526 in the dataflow graph 525. In this way, filtering is performed by a single process running on a single computing device, which reduces the overhead of inter-process (and potentially inter-device) communication.
[0102] Figure 5D shows how optimization rules are applied to a dataflow graph 530 for alternative explanation in order to perform integrated operation optimization. As shown in Figure 5D, the dataflow graph 530 includes node 532 representing a filtering operation with respect to key A, followed by node 534 representing a roll-up operation. In this example, the graph optimizer can perform integrated operation optimization by replacing the two separate nodes 522 and 524 with a single node representing a roll-up operation that processes the selected input with respect to key A, as shown using node 536 in the dataflow graph 535. In this way, a single process running on a single device can perform the equivalent of both the roll-up and filtering operations, which reduces the overhead of inter-process (and potentially inter-device) communication.
[0103] Figure 6 illustrates the application of optimization rules to the explanatory dataflow graph 600 to eliminate unnecessary data processing operations. As shown in Figure 6, the dataflow graph 600 includes node 602 representing a sort operation on key A, followed by node 604 representing a reformatting command, and then node 606 representing an unordered write command. If the write command is unordered, and therefore the order imposed on the data by the sort operation represented by node 602 is not preserved by the write operation, the graph optimizer can remove the node representing the sort operation, resulting in dataflow graph 605.
[0104] Figure 7 illustrates the application of optimization rules to an illustrative dataflow graph 700 to perform width reduction optimization according to several embodiments of the technology described herein. As shown in Figure 7, the dataflow graph 700 includes a node 702 that sets the value of data column A as a logical "or" of the data stored in columns B and C, followed by a node 704 representing a filtering operation on data column A provided by node 702, and a node 706 representing a sorting operation using key D. After the filtering operation, data column A may not be used in downstream calculations. Therefore, in some embodiments, the graph optimizer can perform width reduction optimization by removing the data in data column A so that it does not propagate further (for example, by introducing a node to remove this data from the data moving along the links between nodes, as shown using node 708 in the dataflow graph 705). This reduces the amount of computing resources required to carry the data through subsequent calculations downstream of node 704 in the dataflow graph 705 (for example, by reducing the network resources, memory resources, and processing resources used).
[0105] Figure 8A illustrates the application of optimization rules to the explanatory dataflow graph 800 for deduplication optimization. As shown in Figure 8A, the dataflow graph 800 includes nodes 802 and 804, both representing read operations from the same underlying file (A.dat). The graph optimizer can perform deduplication optimization by replacing both such nodes with a single node (for example, as shown by node 806 in the dataflow graph 805) so that the data is read only once. By not reading the same data twice, even if the data is later used in multiple different parts of the dataflow graph 805, the amount of computing resources used to access the data is reduced.
[0106] Figure 8B shows how to apply optimization rules to the explanatory dataflow graph 811 to perform deduplication optimization using zippering. As shown in Figure 8B, the dataflow graph 811 contains node 810, which represents a data processing operation that reads data from a file called "A.dat". This data is then processed by two different branches. The first branch contains node 812 (which extracts the data column "Af" and adds "1" to the data in that column) and node 814 (which sorts the data with respect to the column "Ak"). The second branch contains node 816 (which extracts the data column "Ag" and multiplies the data in that column by "3") and node 818 (which sorts the data with respect to the column "Ak"). In the example in Figure 8A, deduplication was involved in removing nodes that performed the same operation (reading the same data from the same file). However, in the example in Figure 8B, the operations performed by the different branches of graph 811 are not identical. However, the processes performed on these different branches are similar enough to be folded into a single path. This folding can be done by merging the paths from left to right (similar to how a zipper connects its two sides).
[0107] In this example, the graph optimizer may modify graph 811 so that the computations represented by nodes 812 and 816 are performed sequentially (rather than in parallel). The result is shown in dataflow graph 821. Next, the graph optimizer may modify graph 821 so that nodes 826 and 828, representing their respective sorting operations, are consolidated into a single node 836. The result is shown in dataflow graph 831. As can be seen, the number of nodes in graph 831 is reduced compared to graph 811, allowing the same data to be processed in the same location, thereby reducing the computational resources required. In addition, further optimizations can be applied to the resulting graph 831, for example, by consolidating the operations represented by nodes 832 and 834 into a single node.
[0108] Figure 9 illustrates the application of optimization rules to the explanatory dataflow graph 900 for sequential-parallel optimization. As shown in Figure 9, the dataflow graph 900 includes node 902 representing serialization operations applied to data previously processed across k computing devices, and node 904 representing sorting (with respect to key A). In some embodiments, the dataflow graph optimizer may modify the dataflow graph so that one or more operations applied to the serialized data are instead applied in a parallelized manner. In this example, the dataflow graph optimizer may modify graph 900 to remove the serialization operation and allow sorting to be applied in a parallel form with respect to k, as shown using node 906 in dataflow graph 905. The results of k parallel sorts can then be merged using a merge operation, as shown using node 908 in dataflow graph 905.
[0109] Figure 10A shows an initial dataflow graph 1000 according to several embodiments of the technology described herein. Figure 10B shows an updated dataflow graph 1050 obtained by iteratively applying optimization rules to the initial dataflow graph shown in Figure 10A, according to several embodiments of the technology described herein. As can be seen by comparing the initial dataflow graph 1000 in Figure 10A and the updated dataflow graph 1050 in Figure 10B, the updated dataflow graph has fewer nodes and links than the initial dataflow graph and can be executed more efficiently than the initial dataflow graph. Several optimizations applied to the initial dataflow graph 1000 according to several embodiments of the technology described herein are detailed below.
[0110] In the example illustrating Figures 10A and 10B, multiple integrated operation optimizations are applied to the initial dataflow graph 1000 in Figure 10A to generate the updated dataflow graph 1050 in Figure 10B. For example, a data processing operation represented by node 1002 ("expr" - which may also be an expression for performing any appropriate calculation on the data flowing through node 1002, in this case formatting the data for output) and a write data processing operation represented by node 1004 ("write_file") are integrated using integrated operation optimizations during the generation of the updated dataflow graph 1050, which includes node 1054 configured to perform both of these data processing operations. As another example, a data processing operation represented by node 1006 ("expr") and a filter data processing operation represented by node 1008 ("filter") are integrated using integrated operation optimizations during the generation of the updated dataflow graph 1050, which includes node 1056 configured to perform both of these data processing operations. As yet another example, a data processing operation ("expr") represented by node 1010 and a write data processing operation ("write_file") represented by node 1012 are integrated using integrated operation optimization during the generation of an updated dataflow graph 1050, which includes node 1058 configured to perform both of these data processing operations. As described herein, in some embodiments, the data processing operations associated with a single dataflow graph node are performed by a single process running on a single computing device, so node integration reduces the overhead of inter-process (and potentially inter-device) communication, which improves the performance of the data processing system.
[0111] In the example illustrating Figures 10A and 10B, several optimizations to reduce redundancy are applied to the initial dataflow graph 1000 in Figure 10A in order to generate the updated dataflow graph 1050 in Figure 10B. For example, the sorting operations represented by nodes 1030, 1032, and 1034 in the initial dataflow graph 1000 are removed and have no equivalents in the updated dataflow graph 1050. These sorting operations are removed because the data entering these nodes has already been sorted by sorting operations applied by nodes 1022, 1026, and 1028, respectively. The sorted order applied by the processing at nodes 1022, 1026, and 1028 is maintained by subsequent data processing operations (e.g., sorted rollup operations), and the sorted order of the data is preserved so that further sorting at nodes 1030, 1032, and 1034 is unnecessary. Therefore, nodes 1030, 1032, and 1034 are removed during the generation of the updated dataflow graph 1050. On the other hand, the sorting operations represented by nodes 1022, 1026, and 1028 are retained. For example, nodes 1022 and 1026 in the initial dataflow graph 1000 correspond to nodes 1070 and 1072 in the updated dataflow graph 1050. In contrast, the sorting operation represented by node 1036 is retained because it is not redundant since the preceding complete outer merge join operation does not retain a sorted state (node 1074 is the corresponding node in the updated dataflow graph 1050).
[0112] Figures 10A and 10B also show another example of removing unnecessary data processing operations. As shown in Figure 10A, the initial data flow graph 1000 eventually includes a node 1024 representing a sort operation, followed by a node 1004 representing an unordered write command. If the write commands are unordered, and therefore the order imposed on the data by the sort operation represented by node 1024 is not preserved by the write operation, then node 1024 representing the sort operation can be removed. As can be seen from Figure 10B, there is no sort operation corresponding to node 1024 (this operation has been removed).
[0113] As yet another example of removing unnecessary data processing operations, the initial dataflow graph 1000 includes layout and partition operations represented by nodes 1040, 1045, 1046, and 1047 (each of which is redundant as each is preceded by its respective layout and partition node, and the layout would otherwise not change). Therefore, these nodes are removed from the initial dataflow graph 1000 (the corresponding nodes do not exist in the dataflow graph 1050). In the related optimization applied to the initial dataflow graph 1000, the layout and partition operation represented by node 1048 is replaced by the aggregation operation represented by node 1088. In contrast, the layout and partition data processing operation represented by node 1044 is retained because it is not redundant, and it is responsible for establishing the final partitioning and layout, resulting in node 1080 (partition_by_key) in the final graph 1050.
[0114] Figures 10A and 10B also show an example of width reduction optimization. As can be seen from the introduction of node 1064 ("reformat_implicit") shown in the updated data flow graph 1050 in Figure 10B, the number of data columns processed using the read data processing operation represented by node 1014 is reduced. In this example, the graph optimizer detects that some columns of the read data will not be used later by determining which columns will be written out by the write operation represented by node 1012.
[0115] Figure 10C shows another view of the initial dataflow graph 1000 in Figure 10A, according to several embodiments of the technology described herein. In this view, the labels of the dataflow graph nodes are replaced with their acronyms. Similarly, Figure 10D shows another view of the updated dataflow graph in Figure 10B (in this view, the labels of the dataflow graph nodes are replaced with their acronyms).
[0116] Figure 11 shows an example of a suitable computing system environment 1100 in which the technologies described herein may be implemented. Computing system environment 1100 is merely an example of a suitable computing environment and is not intended to imply any limitation on the scope of use or functionality of the technologies described herein. Computing environment 1100 should not be construed as having any dependence or requirements on any one or combination of the components illustrated in the exemplary operating environment 1100.
[0117] The technologies described herein can be used in conjunction with a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the technologies described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computers, and distributed computing environments encompassing any of the above systems or devices.
[0118] A computing environment can execute computer executable instructions, such as program modules. Generally, program modules include routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. The techniques described herein may be executed in a distributed computing environment in which tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may reside in both local and remote computer storage media, including memory storage devices.
[0119] Referring to Figure 11, an exemplary system implementing the techniques described herein encompasses a general-purpose computing device in the form of a computer 1110. The components of computer 1110 may include, but are not limited to, a processing unit 1120, system memory 1130, and a system bus 1121 that connects various system components, including system memory, to the processing unit 1120. The system bus 1121 may be any of several types of bus structures that include a memory bus or memory controller, peripheral bus, and local bus, using any of various bus architectures. For example, and not limited to, such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Extended ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, also known as a mezzanine bus.
[0120] Computer 1110 generally encompasses a variety of computer-readable media. Computer-readable media may be any available media accessible by computer 1110 and include both volatile and non-volatile media, removable and non-removable media. For example, and not limited to, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store desired information and can be accessed by computer 1110. Communication media generally encompass all information distribution media that embody computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transport mechanisms. The term “modulated data signal” means a signal that has one or more of its characteristic sets, or that has been modified to encode information in a signal. By example, and not limited to, communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the above shall also be included within the scope of computer-readable media.
[0121] The system memory 1130 includes computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 1131 and random-access memory (RAM) 1132. The basic input / output system 1133 (BIOS), which contains basic routines that help transfer information between elements within the computer 1110, such as during startup, is generally stored in the ROM 1131. The RAM 1132 generally contains readily available data and / or program modules currently being manipulated by the processing unit 1120. As an example, and not limited to, Figure 11 illustrates an operating system 1134, an application program 1135, other program modules 1036, and program data 1137.
[0122] Computer 1110 may also include other removable / non-removable, volatile / non-volatile computer storage media. As just one example, Figure 11 illustrates a hard disk drive 1141 that reads or writes to a non-removable, non-volatile magnetic medium, a flash drive 1151 that reads or writes to removable, non-volatile memory 1152 such as flash memory, and an optical disk drive 1155 that reads or writes to removable, non-volatile optical disk 1156 such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital multipurpose disks, digital videotapes, solid RAM, solid ROM, and the like. The hard disk drive 1141 is generally connected to the system bus 1121 through a non-removable memory interface such as interface 1140, and the magnetic disk drive 1151 and optical disk drive 1155 are generally connected to the system bus 1121 through a removable memory interface such as interface 1150.
[0123] The drives described above and illustrated in Figure 11, and the computer storage media associated with them, provide storage for computer-readable instructions, data structures, program modules, and other data of the computer 1110. In Figure 11, for example, the hard disk drive 1141 is illustrated as storing the operating system 1144, application programs 1145, other program modules 1146, and program data 1147. Note that these components may be the same as or different from the operating system 1134, application programs 1135, other program modules 1136, and program data 1137. The operating system 1144, application programs 1145, other program modules 1146, and program data 1147 are given different numbers here to illustrate that they are at least different copies. The user can input commands and information into the computer 1110 using input devices such as the keyboard 1162 and a pointing device 1161, commonly called a mouse, trackball, or touchpad. Other input devices (not shown) may include microphones, joysticks, gamepads, satellite dishes, scanners, and the like. These and other input devices are often connected to the processing unit 1120 by a user input interface 1160 coupled to the system bus, but may also be connected by other interfaces and bus structures such as parallel ports, game ports, or Universal Serial Bus (USB). A monitor 1191 or other type of display device is also connected to the system bus 1121 via an interface such as a video interface 1190. In addition to the monitor, the computer may also include other peripheral output devices such as speakers 1197 and printers 1196, which can be connected through an output peripheral interface 1195.
[0124] Computer 1110 can operate in a networked environment using logical connections to one or more remote computers, such as remote computers 1180. The remote computers 1180 may be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and generally, although only the memory storage device 1181 is illustrated in Figure 11, many or all of the above-mentioned elements in relation to computer 1110 are included. The logical connections depicted in Figure 11 include local area networks (LANs) 1171 and wide area networks (WANs) 1173, but may also include other networks. Such networking environments are common in offices, enterprise-scale computer networks, intranets, and the internet.
[0125] When used in a LAN networking environment, computer 1110 is connected to LAN 1171 via a network interface or adapter 1170. When used in a WAN networking environment, computer 1110 generally includes a modem 1172 or other means for establishing communication over WAN 1173, such as the Internet. The modem 1172, which may be internal or external, may be connected to the system bus 1121 via a user input interface 1160 or other appropriate mechanism. In a networked environment, program modules, or parts thereof, written in relation to computer 1110 may be stored in a remote memory storage device. As an example, and not limited to, Figure 11 illustrates a remote application program 1185 residing in memory device 1181. The network connections shown are illustrative, and it will be understood that other means may be used to establish communication links between computers.
[0126] Although several aspects of at least one embodiment of the present invention have been described above, it will be understood that various changes, modifications, and improvements are readily conceivable to those skilled in the art.
[0127] Such changes, modifications, and improvements are intended to be part of this disclosure and to be within the spirit and scope of the invention. Furthermore, while advantages of the invention are shown, it should be understood that not all embodiments of the technology described herein encompass all described advantages. Some embodiments may not implement any of the features described herein as advantageous, and in some cases, one or more of the described features may be implemented to obtain further embodiments. Accordingly, the above description and drawings are merely examples.
[0128] The above embodiments of the technology described herein may be implemented in any of many ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code can run on any suitable processor or group of processors, whether provided on a single computer or distributed across multiple computers. Such processors may be implemented as integrated circuits and have one or more processors in an integrated circuit component that includes commercially available integrated circuit components known in the industry by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processor may be implemented in custom circuitry such as ASICs, or in semi-custom circuitry resulting from the configuration of programmable logic devices. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercial, semi-custom, or custom. As one specific example, some commercial microprocessors have multiple cores such that one or a subset of multiple cores can constitute a processor. However, the processor can be implemented using circuitry of any suitable format.
[0129] Furthermore, it is understood that a computer may be embodied in any of many forms, such as a rack-mount computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computer may be incorporated into a device that is not generally considered a computer but has appropriate processing power, including a personal digital assistant (PDA), a smartphone, or any other appropriate portable or fixed electronic device.
[0130] Furthermore, a computer may have one or more input and output devices. These devices can be used, in particular, to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for a visual representation of the output, and a speaker or other sound-generating device for an audible representation of the output. Examples of input devices that can be used for the user interface include a keyboard, as well as pointing devices such as a mouse, touchpad, and digitizer tablet. As another example, a computer may receive input information by speech recognition or in other audible formats.
[0131] Such computers can be interconnected by one or more networks of any appropriate form, encompassing a corporate network or a local area network or wide area network such as the Internet. Such networks may be based on any appropriate technology, may operate according to any appropriate protocol, and may encompass wireless networks, wired networks, or fiber optic networks.
[0132] Furthermore, the various methods or processes outlined herein may be encoded as software executable for one or more processors using any one of various operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code to be executed against a framework or virtual machine.
[0133] In this regard, the present invention may be embodied as a computer-readable storage medium (or more computer-readable media) (e.g., computer memory, one or more floppy disks, compact discs (CDs), optical discs, digital video discs (DVDs), magnetic tape, flash memory, field-programmable gate arrays or circuit configurations in other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods for carrying out the various embodiments of the present invention described above. As is evident from the above examples, the computer-readable storage medium can retain information for a time sufficient to provide computer-executable instructions in a non-temporary form. Such one or more computer-readable storage media may be portable so that one or more programs stored therein can be loaded onto one or more different computers or other processors to carry out the various embodiments of the present invention described above. In this specification, the term “computer-readable storage medium” encompasses only non-temporary computer-readable media that can be considered as a product (i.e., manufactured goods) or machine. Alternatively or additionally, the present invention may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.
[0134] The terms “program” or “software” are used herein collectively to refer to any type of computer code or set of computer executable instructions that can be used to program a computer or other processor to implement various aspects of the present invention as described above. In addition, according to certain aspects of this embodiment, it will be understood that, when executed, one or more computer programs that perform the methods of the present invention do not need to reside on a single computer or processor, but may be distributed modularly across a number of different computers or processors to implement various aspects of the present invention.
[0135] Computer executable instructions can take many forms, such as program modules, which are executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Generally, the functionality of program modules may be combined or distributed as desired in various embodiments.
[0136] Furthermore, the data structure may be stored in a computer-readable medium in any appropriate form. For the sake of simplicity in illustration, the data structure may be shown having fields that are related by location within the data structure. Such relationships can also be achieved by assigning locations in the computer-readable medium that convey the relationships between fields to the storage of the fields. However, relationships between information in the fields of the data structure may be established using any appropriate mechanism, including the use of pointers, tags, or other mechanisms for establishing relationships between data elements.
[0137] Various aspects of the present invention may be used individually, in combination, or in various arrangements not specifically described in the embodiments described above, and therefore, in their application, are not limited to the details and arrangements of components described in the above description or illustrated in the drawings. For example, an aspect described in one embodiment may be combined in any way with an aspect described in another embodiment.
[0138] Furthermore, the present invention may be embodied as one example provided. The acts performed as part of this method may be ordered in any appropriate manner. Thus, embodiments may be constructed in which the acts are performed in a different order than that shown (this may include performing several acts simultaneously, even if they are shown as sequential acts in the explanatory embodiments).
[0139] Furthermore, certain actions are described as being performed by a “User.” It is understood that a “User” does not necessarily have to be a single individual, and in some embodiments, actions attributed to a “User” may be performed by a team of multiple individuals and / or by an individual in combination with computer-aided tools or other mechanisms.
[0140] The use of ordinal terms such as “first,” “second,” and “third” in a claim to modify a claim element does not in itself imply a temporal order in which one claim element takes precedence, precedence, or sequence or method of acting over another claim element, and these claim elements are distinguished by being used merely as labels to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms).
[0141] Furthermore, the expressions and terms used herein are for illustrative purposes only and are not intended to be limiting. The use herein of “including,” “comprising,” “having,” “containing,” “involving,” and their variations means that they encompass the items listed thereafter and their equivalents, as well as any additional items. [Explanation of symbols]
[0142] 100 Data Processing Systems 102-1 Data Storage Devices 102-2 Data Storage Devices 104 Query Input Module 106 Query Plan Generator 108 Dataflow Graph Generator 110 Graph Optimizer 112 modules 115 Graph Execution Engine
Claims
1. A method that is performed by at least one computer hardware processor, A step of obtaining an initial data flow graph including a first plurality of nodes representing a first plurality of data processing operations and a first plurality of links representing the flow of data between the nodes of the first plurality of nodes, A step of generating an updated dataflow graph by iteratively applying one or more dataflow graph optimization rules to update the initial dataflow graph, wherein the updated dataflow graph includes a second plurality of nodes representing a second plurality of data processing operations, and a second plurality of links representing the flow of data between the nodes of the second plurality of nodes. Includes, The aforementioned iterative application is performed using a dataflow graph pattern matching language that defines a plurality of expressions, including a first expression, which represents each pattern identified in the dataflow graph. The aforementioned repeated application is Identifying a first portion of the initial dataflow graph to which a first optimization rule is applied using the first expression of the dataflow graph pattern matching language, wherein the first expression includes a plurality of symbols associated with each node in the dataflow graph, each representing a data processing operation, and the order of the symbols representing the order of the data processing operations in the dataflow graph. Applying the first optimization rule to the first portion of the initial data flow graph Includes, The first optimization rule includes an integrated operation optimization rule, The first expression of the dataflow graph pattern matching language includes an expression for identifying a set of nodes in the initial dataflow graph that represent a series of data processing operations of at least a threshold number, which can be integrated and represented by a single node using the integrated operation optimization rule, Applying the first optimization rule includes applying the integrated operation optimization rule to the first portion of the initial data flow graph, A method for applying the integrated operation optimization rule, comprising replacing the set of nodes with a single node representing the integration of the operations represented by the set of nodes.
2. The method according to claim 1, wherein the second plurality of nodes has fewer nodes than the first plurality of nodes, and the second plurality of links has fewer links than the first plurality of links.
3. The second set of nodes includes a node representing a first data processing operation and another node representing a second data processing operation. The method described above is The step of executing the updated data flow graph by, at least partially, performing the first data processing operation using a first computer system process and performing the second data processing operation using a second computer system process different from the first computer system process. The method according to claim 1, further comprising:
4. The step further includes assigning a processing layout to each of one or more nodes of the updated data flow graph, The method according to claim 3, wherein the execution of the updated data flow graph is performed according to one or more assigned processing layouts.
5. The first expression of the dataflow graph pattern matching language includes an expression for identifying nodes in the initial dataflow graph that represent mutually commutative data processing operations, The method according to claim 1, comprising identifying a first node of the first plurality of nodes that represent a first data processing operation, wherein identifying the first portion of the initial data flow graph is interchangeable with a second data processing operation represented by a second node of the first plurality of nodes, and the second node is connected to the first node.
6. Applying the first optimization rule to the first portion of the initial data flow graph is Changing the order of the nodes identified in the initial data flow graph, which represent mutually commutative data processing operations, Applying the first optimization rule to the identified first portion of the initial dataflow graph having the nodes whose order has been changed The method according to claim 5, including the method described in claim 5.
7. The method according to claim 1, wherein applying the first optimization rule includes applying an optimization selected from the group consisting of eliminating redundant data processing operations, intensity reduction optimization, integrated operation optimization, width reduction optimization, and deduplication optimization.
8. The method according to claim 1, wherein generating an updated dataflow graph includes identifying a first node representing redundant operation and removing the first node from the initial dataflow graph.
9. Generating an updated dataflow graph includes identifying two nodes in the initial dataflow graph that represent each sort data processing operation, and replacing the two nodes representing each sort data processing operation with a single node representing the sort data processing operation. The method according to claim 1, wherein the two nodes representing the respective sort data processing operations are not adjacent to each other in the initial data flow graph.
10. The method according to claim 1, wherein generating an updated dataflow graph includes replacing a first node representing a first data processing operation with a second node representing a second data processing operation that requires fewer computations than the first data processing operation.
11. The method according to claim 1, wherein generating an updated dataflow graph includes identifying two nodes in the initial dataflow graph that represent each of the combined data processing operations, and replacing the two nodes representing each of the combined data processing operations with a single node representing the combined data processing operation.
12. The initial data flow graph mentioned above is generated by To obtain a Structured Query Language (SQL) query, To generate a query plan for the aforementioned SQL query, Using the aforementioned query plan, the initial data flow graph is generated. The method according to claim 1, including the method described in claim 1.
13. A data processing system, The system comprises at least one computer hardware processor and at least one non-temporary computer-readable recording medium, wherein the at least one non-temporary computer-readable recording medium is A processor-executable instruction, when executed by the at least one computer hardware processor, that causes the at least one computer hardware processor to perform the method according to any one of claims 1 to 12. A data processing system that records data.
14. At least one non-temporary computer-readable recording medium for recording processor-executable instructions causing the at least one computer hardware processor to perform the method according to any one of claims 1 to 12 when executed by the at least one computer hardware processor.
Citation Information
Patent Citations
Generating device for parallel processing program
JP1994161773A
Inquiry processing optimization device in relational data base system
JP2001222452A
Query optimizer system and method
JP2005018767A
Database system and program
JP2005309684A
Managing Data Queries
JP2014519080A
