Scheduling of query pipeline execution

Abstract

To provide systems that reduces memory consumption during query processing.SOLUTION: A system is configured to: receive a query execution plan associated with a plurality of query execution pipelines, estimated execution costs and estimated intermediate result cardinalities; determine one or more precedence relationships of the plurality of query execution pipelines; determine an execution order of the plurality of query execution pipelines based on the estimated execution costs, the estimated intermediate result cardinalities, and the one or more precedence relationships; and provide the execution order of the plurality of query execution pipelines and the query execution plan to a query execution engine.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 263,703, filed November 8, 2021, the contents of which are incorporated herein by reference for all purposes. [Background technology]

[0002] In a conventional database system, a query optimizer receives a database query and generates a query execution plan based thereon. More specifically, the query optimizer determines several candidate execution plans based on the received query, estimates the cost of each execution plan, and selects the plan with the lowest execution cost. An execution engine executes the selected query execution plan to the database and returns the corresponding result set.

[0003] A query execution plan may conform to a pipeline-based query execution model. According to such a model, a query execution plan consists of multiple execution pipelines. An execution pipeline takes as input a base table or result set of a previously executed execution pipeline and performs a mixture of unary operators or lookups (e.g., hash joins) on the base table / result set to construct another result set. Thus, the execution pipelines are interdependent in that execution of a given execution pipeline may require the prior execution of one or more other execution pipelines.

[0004] Due to the increase in data size, query workload, and query complexity, optimizing memory consumption during query processing is important, especially in cloud scenarios. For example, modern cloud-based systems face a large number of complex online analytical processing (OLAP) queries that reference huge tables. These queries are typically executed using execution pipelines as described above. Furthermore, optimal use of existing hardware resources in main memory database systems can lead to low operational costs and high throughput. Summary of the Invention [Means for solving the problem]

[0005] A system that reduces memory consumption during query processing is desired, especially for pipeline-based query execution plans. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram of a system for determining a pipeline execution order for a query execution plan according to some embodiments. [Figure 2] FIG. 10 illustrates an example query execution plan according to some embodiments. [Figure 3] 10 illustrates a flow diagram for determining a pipeline execution order for a query execution plan according to some embodiments. [Figure 4] FIG. 10 is a diagram of a pipeline specialization of a query execution plan according to some embodiments. [Figure 5] 10A-10C comprise a flow diagram of some embodiments of determining a pipeline execution order for a query execution plan based on the memory cost of each of the candidate pipeline execution orders. [Figure 6] 1 is a graph illustrating precedence relationships and metadata associated with an execution pipeline of a query execution plan according to some embodiments. [Figure 7] FIG. 1 is a generalized diagram of a query execution plan subtree illustrating logical entities for pipeline memory cost determination according to some embodiments. [Figure 8] 10A-10C comprise a flow diagram of determining a pipeline execution order for a query execution plan based on execution pipeline depth according to some embodiments. [Figure 9] FIG. 10 is a diagram of a pipeline specialization of a query execution plan according to some embodiments. [Figure 10] 1 is a graph illustrating precedence relationships and metadata associated with an execution pipeline of a query execution plan according to some embodiments. [Figure 11] FIG. 1 is a block diagram of a database node according to some embodiments. [Figure 12] FIG. 1 is a diagram of a cloud-based architecture according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0007] The following description is provided to enable any person skilled in the art to make and use the described embodiments and sets forth the best modes contemplated for carrying out some embodiments, although various modifications will be readily apparent to those skilled in the art.

[0008] As described above, a query execution plan output by a query optimizer may include several execution pipelines. The pipelines may be executed in various orders while conforming to the query execution plan, but each execution order produces the same result set and is associated with the same execution cost. The inventors have discovered that different pipeline execution orders may consume memory differently (i.e., exhibit different memory consumption profiles). Therefore, for the purpose of reducing memory consumption, it may be desirable to execute the execution pipelines of a query execution plan in a specific execution order that conforms to the query execution plan rather than in a different execution order that also conforms to the query execution plan.

[0009] The inventors have also discovered a system and method for selecting a pipeline execution order from several pipeline execution orders that conform to a given query execution plan based on their relative memory consumption profiles. The system and method can be efficiently incorporated into a database management system for implementation after query optimization and before plan execution. In this regard, some embodiments leverage various data structures output by a query optimizer, including the query execution plan and various estimated costs and cardinality.

[0010] 1 is a block diagram of system 100 according to some embodiments. Each illustrated element of system 100 may be implemented using any suitable combination of computing hardware and / or software that is or becomes known. System 100 may comprise components of an independent or distributed (i.e., multi-node) database system. In some embodiments, two or more elements of system 100 are implemented by a single computing device. One or more elements of system 100 may be implemented as a cloud service (e.g., software as a service, platform as a service).

[0011] Query optimizer 110 receives a query from a client and determines a query execution plan for executing the query on tables 124 in database storage 120. Embodiments are not limited to any particular client or to any particular query language. Query optimizer 110 may comprise any suitable query processor known or to become known therefor. As known in the art, query optimizer 110 may generate several alternative query execution plans and calculate the execution cost of each based on metadata 122 in database storage 120. Metadata 122 may define the structure and relationships of tables 124 (e.g., database schema) as well as statistics describing the data in tables 124. These statistics may be periodically refreshed by a statistics server (not shown) in system 100.

[0012] The query optimizer 110 selects one of the query execution plans and provides the selected query execution plan to the pipeline ordering component 130. The query execution plan identifies the execution pipeline, pipeline breakers, input tables and cardinality estimates of intermediate results, and execution costs for subtrees of the query execution plan. The query execution plan may also include other data in some embodiments.

[0013] 2 illustrates a representation 200 of elements of a query execution plan received from a query optimizer according to some embodiments. The query execution plan includes input tables R, S, T, U, and V, and physical operators 202-208 represented as hash joins. Embodiments are not limited to hash joins.

[0014] Each hash join includes a build side and a probe side, as defined by the query execution plan. According to representation 200, the build side is the right side of the hash join, while the probe side is depicted on the left side of the hash join. A hash join is defined herein as a build-side pipeline breaker, and therefore terminates a pipeline that enters the right side of the hash join. For example, hash join 202 is the pipeline breaker for a pipeline starting from table T, hash join 204 is the pipeline breaker for a pipeline starting from table V, hash join 206 is the pipeline breaker for a pipeline starting from table S, and hash join 208 is the pipeline breaker for a pipeline starting from table U.

[0015] An execution pipeline may therefore depend on intermediate results produced by the pipeline breakers of other execution pipelines. For example, the intermediate results produced by hash joins 202 and 204 must remain in memory until they are consumed by hash join 208. Once a pipeline produces a new intermediate result (or final result), all intermediate results referenced by it, such as the hash table searched by the current pipeline, can be released from memory.

[0016] The total memory usage during execution of a query execution plan varies over time because intermediate results have different sizes (e.g., number of records * record width) and the number of intermediate results maintained in memory changes during execution. The inventors have noted that the order of execution of individual pipelines can significantly affect memory usage because it dictates the order and time in which intermediate results are generated, stored, and consumed by later-executed pipelines.

[0017] Thus, the pipeline ordering component 130 determines the order of execution of the individual pipelines (i.e., the pipeline execution order) that results in the most desirable memory usage. According to some embodiments, the pipeline ordering component 130 determines all possible pipeline execution orders that conform to the query execution plan provided by the query optimizer 110. Using the data provided with the query execution plan, for each pipeline execution order, the ordering component 130 estimates a memory usage, or memory profile, over time based on the memory size and lifetime of intermediate results generated by pipeline breakers during execution according to the execution order. The pipeline execution order associated with the least consuming memory profile may then be selected as the pipeline execution order for executing the query execution plan.

[0018] Pipeline ordering component 130 then provides the query execution plan and the selected pipeline execution order to execution engine 140. Execution engine 140 executes the query execution plan as known in the art and in accordance with the selected pipeline execution order to generate result set 150 based on the data in table 124.

[0019] 3 comprises a flow diagram of process 300 according to some embodiments. In some embodiments, various hardware elements of system 100 execute program code to perform process 300. Process 300, and all other processes mentioned herein, may be embodied in processor-executable program code read from one or more non-transitory computer-readable storage media, such as a hard disk drive, volatile or non-volatile random access memory, DVD-ROM, flash drive, and magnetic tape, and may be executed by one or more processing units, including, but not limited to, hardware processors, processor cores, and processor threads. In some embodiments, hardwired circuitry may be used in place of or in combination with program code for implementation of processes according to some embodiments. Embodiments are therefore not limited to any specific combination of hardware and software.

[0020] Prior to process 300, a query is received based on which a query execution plan is generated. The query may be received from any client application by any form of query processor. According to some embodiments, the query is received by a query optimizer of a Structured Query Language (SQL) server. The query may therefore include an SQL query, although embodiments are not limited thereto.

[0021] A query optimizer, such as query optimizer 110, generates a query execution plan based on the received query as is known in the art. Generating the query execution plan may include estimating an execution cost associated with the query execution plan. The cost may be estimated based on metadata describing the database tables to be queried and the structure and contents of the tables.

[0022] A query execution plan is received at S310 of process 300. The query execution plan includes the execution costs of the tree of the execution pipeline, the tree and subtrees of the tree, and the estimated cardinality of intermediate results produced by pipeline breakers of the execution pipeline. The query execution plan may be received by pipeline ordering component 130 of system 100, although embodiments are not limited thereto. For example, the functions attributed herein to pipeline ordering component 130 may be performed in whole or in part by query optimizer 110.

[0023] The term "tree" is used herein to indicate that an execution pipeline can be viewed as branching out and down through various pipeline breakers from the pipeline breaker that produces the final query result, as illustrated by representation 200. A query execution plan describes the inputs to each pipeline breaker and identifies which inputs comprise the build side and which inputs comprise the probe side. The pipeline breakers, their inputs, and build / probe side information of the query execution plan are sufficient to generate a representation of the query execution plan as shown in FIG. 2.

[0024] Figure 4 is a diagram of pipeline specializations for the query execution plan illustrated in Figure 2. Each pipeline is labeled in Figure 4 by a circled letter that corresponds to the name of the pipeline's input table. Diagram 400 identifies the estimated cardinality (i.e., number of records) of the intermediate results produced by each pipeline breaker.

[0025] As defined herein, each pipeline breaker terminates its build-side pipeline. For example, pipeline T terminates with a hash join with probe-side pipeline S, resulting in a hash table containing an estimated 20 records. Pipeline S terminates with a hash join with probe-side pipeline R, resulting in a hash table containing an estimated 50 records.

[0026] Returning to process 300, the precedence relationships of the execution pipelines are determined in S320. The precedence relationships define which pipelines must be executed before which other pipelines according to the query execution plan. The precedence relationships may be derived from the query execution plan tree. Referring to FIG. 4, pipeline T must be executed before pipeline S, and pipeline S must be executed before pipeline R. Similarly, pipeline V must be executed before pipeline U, and pipeline U must be executed before pipeline R. However, there is no precedence relationship between pipelines U, V and pipelines S, T. The precedence relationships may be provided directly by the query optimizer along with the query execution plan according to some embodiments.

[0027] Next, in S330, an execution order of the execution pipelines is determined. The pipeline execution order is determined based on precedence relationships, execution costs, and intermediate result cardinality. For example, all possible pipeline execution orders that comply with the query execution plan are first determined. In this example, the possible pipeline execution orders are (V, U, T, S, R), (V, T, S, U, R), (V, T, U, S, R), (T, S, V, U, R), (T, V, S, U, R), and (T, V, U, S, R).

[0028] For each pipeline execution order, a memory usage indicator is determined based on the execution cost and intermediate result cardinality received in S310. This determination may include determining the memory size and lifetime of intermediate results generated by the pipeline breaker during execution according to each execution order. A pipeline execution order is determined based on the indicator. For example, the determined pipeline execution order may be the one associated with the best memory usage. Further details of S330 according to some embodiments will be described below.

[0029] The query execution plan and the determined pipeline execution order are provided to an execution engine at S340. The execution engine may then execute the query execution plan according to the pipeline execution order to generate a result set, as known in the art. The memory usage of such an execution is presumed to be more desirable than that which would have resulted from executing the query execution plan according to a different (e.g., randomly selected) pipeline execution order.

[0030] 5 comprises a flow diagram for determining a pipeline execution order for a query execution plan based on the memory cost of each of the candidate pipeline execution orders according to some embodiments. Thus, process 500 may include an implementation of S330 of process 300.

[0031] In S510, all pipeline execution orders that satisfy the precedence relationships of the query execution plan are determined. FIG. 6 is a graph illustrating the precedence relationships of the execution pipelines described above. As described above, the pipeline execution orders that satisfy the graph of FIG. 6 are (V, U, T, S, R), (V, T, S, U, R), (V, T, U, S, R), (T, S, V, U, R), (T, V, S, U, R), and (T, V, U, S, R). Execution of the query execution plan according to one of these pipeline execution orders may result in the creation and consumption of intermediate results of the query execution plan at different times than execution according to a different pipeline execution order, thereby resulting in different memory consumption profiles.

[0032] 6 also shows, for each pipeline, the cardinality f of its associated pipeline breaker. This value may be received from and / or determined based on the query optimizer. As described above, the query optimizer provides an estimated cardinality of each intermediate result in the query execution plan. Because each intermediate result corresponds to a pipeline breaker, the cardinality of the pipeline breaker is determined to be the cardinality of its corresponding intermediate result.

[0033] At S520, for each of the pipeline execution orders determined at S510, the lifetime of each pipeline breaker is determined. The lifetime of a pipeline breaker may be determined based on the execution cost of its corresponding pipeline.

[0034] 7 illustrates the logical entities referenced in the following discussion of execution cost, breaker lifetime, and memory size. According to some embodiments, a query execution plan is i each with an estimated execution cost c i The subtrees and estimated execution costs are provided by a query optimizer as known in the art. i The intermediate result is the base f i and the row (i.e., record) size w i and the pipeline p i The corresponding pipeline breaker b defines the end of i is written to.

[0035] The entire subtree i The execution cost of c i In contrast to the execution cost e i is the pipeline p i The cost of executing only the subtree s provided by the query optimizer i Estimated execution cost of c i But, pipeline breaker b i To subtrees i This does not include the cost of writing the result of i This is expressed in Figure 7 as: i is a pipeline breaker b i represents the survival time of

[0036] According to some embodiments, a pipeline breaker b i The survival time of l i is the corresponding pipeline p i Next, the execution time of the pipeline p iThe execution time of the pipeline is i It can be roughly calculated by the pipeline p i The execution cost of e i is the pipeline p i The subtree s corresponding to i Estimated execution cost of c i From, s i All subtrees of s j Estimated execution cost of c j may be derived by subtracting s j is a pipeline breaker b j is the pipeline p i Contains only the subtree consumed by the pipeline breaker b i To subtrees i and pipeline p i To take into account the cost of writing the results of i are added together. Therefore, the result is as shown below.

[0037]

number

[0038] Some embodiments are out We utilize a cost function, which is the sum of the output cardinality of all operators, including the base table scan. C out Since the cost function does not distinguish between build and probe costs in the cost of a hash join, for example, the cost of writing a hash table to a pipeline breaker, σ i is C out is set to 0 because it already contains the base of the result.

[0039] Pipeline breaker b i The lifetime of each pipeline p i The execution time of e i In addition, pipeline breaker b iFor the example pipeline execution order (V, T, S, U, R) given above, the pipeline breaker lifetime of pipeline V is the sum of the execution times of pipelines V, T, S, and U. The pipeline execution order O=(p1,…,p i ,…,p j ,…,p k ,…,p n ), and the corresponding pipeline execution cost vector

number

[0040]

number

[0041] Therefore, the pipeline execution order O=(p1,…,p i ,…,p n ) pipeline survival vector

number

number

[0042] The memory cost of each pipeline execution order is determined in S530. The memory cost of the pipeline execution order may be determined based on the determined lifetime and intermediate result cardinality of each execution pipeline in the pipeline execution order. First, the pipeline breaker b i Memory size m i is the corresponding base f i and row size wi As a product of m i =w i f i Referring to the hash join example, this approximation is based on the assumption that the pipeline breaker consists of a dense hash table, regardless of the hash table implementation. Given a pipeline execution order O=(p0,...,p i ,…,p n ) vector of pipeline breaker memory sizes

number

number

[0043] According to some embodiments, the memory cost determined in S530 is a memory accumulation metric, which is a vector of pipeline breaker memory costs:

number

number

[0044]

number

[0045] Therefore, the memory amount MI is the sum of all pipeline breakers b i their survival time l over i Their memory size m is multiplied by i The memory consumption of a pipeline breaker is 0 before it is created, and increases as soon as each pipeline starts running. i Until it is destroyed i and then 0.

[0046] At S540, a pipeline execution order with the lowest memory cost is determined, which, together with the corresponding query execution plans, may then be provided to an execution engine for execution of the execution pipelines of the query execution plans in the determined pipeline execution order.

[0047] According to some embodiments, the determination of all pipeline execution orders that satisfy precedence relations in S510 ignores pipelines whose pipeline breakers are associated with low memory costs. Such pipeline breakers are associated with short lifetimes, small memory sizes, or both. By ignoring such pipelines, the number of pipeline execution orders determined in S510 may be significantly reduced, leading to faster overall execution of process 500.

[0048] For example, assume that pipeline V is associated with a small memory cost and is therefore selected to be ignored in S510. The pipeline execution orders that satisfy the dependencies in the graph of FIG. 6 are therefore determined in S510 as (U, T, S, R), (T, S, U, R), and (T, U, S, R), which are half the pipeline execution orders determined in the above example of S510. Once the pipeline execution order associated with the lowest memory cost of these three pipeline execution orders is determined in S540, pipeline V is added to the determined pipeline execution order at a position in the order that complies with the dependencies in the query execution plan.

[0049] 8 is a flow diagram of a process 800 according to some embodiments. Process 800 may be performed to determine a pipeline execution order from multiple pipeline execution orders that conform to a given query execution plan. Some embodiments of process 800 do not utilize the cardinality or execution cost of intermediate results provided by a query optimizer to determine the pipeline execution order.

[0050] Rather, process 800 is a heuristic algorithm intended to determine a pipeline execution order that has a low, but not necessarily the lowest memory cost, of all pipeline execution orders that comply with a target query execution plan. At the cost of a suboptimal selection of the pipeline execution order, process 800 may execute much more quickly than process 500 of FIG. 5. The heuristic of process 800 is based on the observation that executing the deepest of interdependent pipelines first will yield a near-optimal pipeline execution order. In other words, execution of the subtree in length order from the start node to the result node is the "best guess" pipeline execution order.

[0051] Figure 9 is a diagram of a pipeline specialization of a query execution plan for purposes of illustrating some embodiments of process 800. The query execution plan of Figure 9 includes nine base tables and eight intermediate hash tables. Figure 10 represents the pipeline of the query execution plan as nodes in a dependency graph.

[0052] At S810 of process 800, the length of each subtree in the execution pipeline is determined. For the example of Figure 10, the subtrees and corresponding lengths determined at S810 are as follows: (E, D, B, A, R; 5), (C, B, A, R; 4), (U, T, S, R; 4).

[0053] Next, at S820, an execution priority is determined for each execution pipeline based on the longest subtree to which the execution pipeline belongs. Continuing with the present example, execution pipelines E, D, B, A, and R are assigned a priority of 5, execution pipeline C is assigned a priority of 4, and execution pipelines U, T, and S are assigned a priority of 4. Because execution pipeline C is assigned a priority of 4 and execution pipelines U, T, and S are also assigned a priority of 4, potential ordering conflicts may be addressed by incrementing the priority of either execution pipeline C or execution pipelines U, T, and S, and similarly incrementing the priorities of all higher priority pipelines. In this example, as shown by the dotted box in FIG. 10 , the priority of execution pipeline C is incremented from 4 to 5 due to its relationship to higher priority execution pipelines B, A, and R. Correspondingly, the priorities of higher priority execution pipelines E, D, B, A, and R are also incremented from 5 to 6.

[0054] A pipeline execution order is determined in S830 based on the assigned priorities. The determination of the pipeline execution order is primarily based on dependencies in the query execution plan. However, as described above, at a certain execution point, two or more pipelines may be selected as the next pipeline to be executed while complying with the dependencies. In some embodiments of S830, the next pipeline selected is the pipeline assigned the highest priority among the two or more pipelines.

[0055] Referring to this example, any of execution pipelines E, C, and U could be the first pipeline in the pipeline execution order. However, execution pipeline E is selected because it has been assigned the highest priority of the three pipelines. The next execution pipeline could be D, C, or U based on the dependency graph. Execution pipeline D is selected as the next execution pipeline because it has been assigned the highest priority of the three pipelines. At this execution point, only pipelines C or U can execute (e.g., pipeline B cannot execute because it requires the intermediate results produced by pipeline C). Pipeline C is selected for next execution due to its higher assigned priority. The same logic continues as above for the remaining execution pipelines, leading to a determined pipeline execution order determined as (E, D, C, B, A, U, T, S, R).

[0056] According to some embodiments, process 800 is performed to relatively quickly determine a pipeline execution order, and the memory cost of this determined pipeline execution order is determined as described above. Process 500 is then performed as described above (with or without the pipeline-ignoring variant). However, if during the determination of the memory cost of a particular pipeline execution order, a sub-result determined in S530 exceeds the memory cost of the pipeline execution order determined via process 800, then the determination of the memory cost of that particular pipeline execution order pipeline is aborted.

[0057] 11 is a block diagram of a database architecture that may determine a pipeline execution order for a query execution plan according to some embodiments, although embodiments are not limited to the architecture of FIG.

[0058] A server node 1110 may receive a query from one of client applications 1130 and 1140 and return results based on data stored within the server node 1110. The node 1110 executes program code to provide an application server 1115 and a query processor 1120. The application server 1115 provides services for running server applications. For example, a web application running on the application server 1115 may receive a Hypertext Transfer Protocol (HTTP) request from a client application 1140 as shown in FIG. 11 .

[0059] The query processor 1120 may include an engine for processing stored data and data. The query processor 1120 may also be responsible for processing Structured Query Language (SQL) and Multidimensional Expressions (MDX) statements, and may receive such statements directly from a client application 1130.

[0060] Query processor 1120 includes a query optimizer 1122 for use in determining a query execution plan, a pipeline ordering component 1123 for determining a pipeline execution order for the query execution plan as described herein, and an execution engine 1124 for executing the query execution plan against tables 1126 in storage 1125 using the determined pipeline execution order. Query processor 1120 may also include a statistics server (not shown) in some embodiments for determining statistics used to estimate query execution plan costs.

[0061] In some embodiments, the data in storage 1125 may include one or more of traditional tabular data, row-stored data, column-stored data, and object-based data. Moreover, the data may be indexed and / or selectively replicated to enable fast search and retrieval thereof. Server node 1110 may support multi-tenancy to separately support multiple unrelated clients by providing multiple logical database systems that are programmatically decoupled from each other.

[0062] Metadata 1128 includes data describing the database schema identified by tables 1126. Metadata 1128 may therefore describe the columns and properties of tables 1126, the properties of each column in each table 1126, the interrelationships between columns, and any other suitable information. In one example, metadata 1128 may include information for identifying one or more columns of table 1126 as dictionary-compressed and for locating the column dictionary and dictionary index associated with each dictionary-compressed column.

[0063] Server node 1110 may implement storage 1125 as an "in-memory" database, where the entire database is stored in volatile (e.g., non-disk-based) memory (e.g., random access memory). The entire database may be persisted and / or backed up to a fixed disk (not shown). Embodiments are not limited to in-memory implementations. For example, data may be stored in random access memory (e.g., cache memory for storing recently used data) and one or more fixed disks (e.g., persistent memory for storing their respective portions of the entire database).

[0064] 12 illustrates a cloud-based database deployment according to some embodiments. The illustrated components may reside on one or more public clouds, providing self-service and instant provisioning, auto-scaling, security, compliance, and identity management capabilities.

[0065] User device 1210 may interact with applications running on application server 1220, for example, via a web browser running on user device 1210, to create, read, update, and delete data managed by database system 1230 and persisted in distributed file storage 1235. Database system 1230 may store data and may execute processes as described herein to determine a pipeline execution order for executing query execution plans on the data. Application server 1220 and / or database system 1230 may comprise cloud-based computing resources, such as virtual machines, allocated by a public cloud provider. As such, application server 1220 and database system 1230 may exhibit demand-based elasticity.

[0066] The above diagrams depict logical architectures for describing processes according to some embodiments; actual implementations may include more or different components arranged in other manners. Other topologies may be used in conjunction with other embodiments. Moreover, each component or device described herein may be implemented by any number of devices in communication via any number of other public and / or private networks. Two or more of such computing devices may be located remotely from each other and may communicate with each other via any known manner of network and / or dedicated connection. Each component or device may include any number of hardware and / or software elements appropriate to provide the functionality described herein as well as any other functionality. For example, any computing device used in the implementations described herein may include a programmable processor that executes program code to cause the computing device to operate as described herein.

[0067] All systems and processes described herein may be embodied in program code stored on one or more non-transitory computer-readable storage media. Such storage media may include, for example, DVD-ROMs, flash drives, magnetic tapes, and solid-state random access memory (RAM) or read-only memory (ROM) storage units. Embodiments are therefore not limited to any specific combination of hardware and software.

[0068] Elements described herein as communicating with each other may communicate directly or indirectly through any number of different systems for transferring data, including, but not limited to, shared memory communications, local area networks, wide area networks, telephone networks, cellular networks, fiber optic networks, satellite networks, infrared networks, radio frequency networks, and any other type of network that may be used to transmit information between devices. Moreover, communication between systems may proceed through any one or more transmission protocols that are or become known, such as Asynchronous Transfer Mode (ATM), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), and Wireless Application Protocol (WAP).

[0069] The embodiments described herein are for purposes of example only. Those skilled in the art will recognize that other embodiments may be implemented with changes and modifications to those described above. [Explanation of symbols]

[0070] 100 systems 110 Query Optimizer 120 Database Storage 122 Metadata 124 tables 130 Pipeline Sequencing Components 140 Execution Engine 150 result sets 202,204,206,208 Hash Join 1110 Server Node 1115 Application Server 1120 Query Processor 1122 Query Optimizer 1123 Pipeline Ordering Component 1124 Execution Engine 1125 Storage 1126 Table 1128 Metadata 1130,1140 Client Applications 1210 User Device 1220 Application Server 1230 Database System 1235 Distributed File Storage

Claims

1. 1. A system comprising: a memory storing processor-executable program code; Processing unit and Equipped with The processing unit executes the processor-executable program code to provide the system with: receiving a query execution plan associated with a plurality of query execution pipelines, an estimated execution cost, and an estimated intermediate result cardinality; determining one or more precedence relationships of the plurality of query execution pipelines; determining an execution order of the plurality of query execution pipelines based on the estimated execution cost, the estimated intermediate result cardinality, and the one or more precedence relations; providing the execution order of the plurality of query execution pipelines and the query execution plan to a query execution engine; A system that allows the following to be performed.

2. The determination of the execution order is determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations; determining a memory cost for each of the plurality of execution orders based on the estimated execution cost and the estimated intermediate result cardinality; determining the one execution order from the plurality of execution orders based on the memory cost of each of the plurality of execution orders; The system of claim 1 , comprising:

3. Determining the one execution order from the plurality of execution orders The system of claim 2 further comprising determining an execution order of the plurality of execution orders having a lowest memory cost.

4. determining the execution orders of the query execution pipelines based on the one or more precedence relations, determining an execution order for the plurality of query execution pipelines having memory costs below a threshold; The system of claim 2 , wherein the determined execution orders do not include the execution orders of the query execution pipelines having memory costs below a threshold.

5. The determination of the execution order is determining, for each of the plurality of query execution pipelines, a longest pipeline path to which the query execution pipeline belongs based on the one or more precedence relations; determining the execution order based on the longest pipeline path to which each of the plurality of query execution pipelines belongs; The system of claim 1 , comprising:

6. The determination of the execution order is determining a first execution order based on the longest pipeline path to which each of the plurality of query execution pipelines belongs; determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations; determining a memory cost for the first execution order based on the estimated execution cost and the estimated intermediate result cardinality; determining a memory cost for each of the plurality of execution orders based on the estimated execution cost and the estimated intermediate result cardinality, wherein determining the memory cost for one execution order of the plurality of execution orders terminates when the memory cost of the one execution order of the plurality of execution orders exceeds the memory cost of the first execution order; determining the one execution order from the plurality of execution orders based on the memory cost of each of the plurality of execution orders; The system of claim 5, comprising:

7. determining the execution orders of the query execution pipelines based on the one or more precedence relations, determining an execution order for the plurality of query execution pipelines having memory costs below a threshold; The system of claim 6 , wherein the determined execution orders do not include the execution orders of the query execution pipelines having memory costs below a threshold.

8. 1. A computer-implemented method comprising: receiving a query execution plan associated with a plurality of query execution pipelines, an estimated execution cost, and an estimated intermediate result cardinality; determining one or more precedence relationships of the plurality of query execution pipelines; determining an execution order of the plurality of query execution pipelines based on the estimated execution cost, the estimated intermediate result cardinality, and the one or more precedence relations; providing the execution order of the plurality of query execution pipelines and the query execution plan to a query execution engine; A method comprising:

9. the step of determining an execution order comprises: determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations; determining a memory cost for each of the plurality of execution orders based on the estimated execution cost and the estimated intermediate result cardinality; determining the one execution order from the plurality of execution orders based on the memory cost of each of the plurality of execution orders; 9. The method of claim 8, comprising:

10. the step of determining the one execution order from the plurality of execution orders comprises: The method of claim 9 , further comprising determining an execution order of the plurality of execution orders that has a lowest memory cost.

11. determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations, determining an execution order for the plurality of query execution pipelines having memory costs below a threshold; The method of claim 9 , wherein the determined execution orders do not include the execution orders of the query execution pipelines having memory costs below a threshold.

12. the step of determining an execution order comprises: determining, for each of the plurality of query execution pipelines, a longest pipeline path to which the query execution pipeline belongs based on the one or more precedence relations; determining the execution order based on the longest pipeline path to which each of the plurality of query execution pipelines belongs; 9. The method of claim 8, comprising:

13. the step of determining an execution order comprises: determining a first execution order based on the longest pipeline path to which each of the plurality of query execution pipelines belongs; determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations; determining a memory cost for the first execution order based on the estimated execution cost and the estimated intermediate result cardinality; determining a memory cost for each of the plurality of execution orders based on the estimated execution cost and the estimated intermediate result cardinality, wherein determining the memory cost for one execution order of the plurality of execution orders terminates when the memory cost of the one execution order of the plurality of execution orders exceeds the memory cost of the first execution order; determining the one execution order from the plurality of execution orders based on the memory cost of each of the plurality of execution orders; 13. The method of claim 12, comprising:

14. determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations, determining an execution order for the plurality of query execution pipelines having memory costs below a threshold; The method of claim 13 , wherein the determined execution orders do not include the execution orders of the query execution pipelines having memory costs below a threshold.

15. A non-transitory storage medium having processor-executable program code stored thereon, The processor-executable program code is executable by a computing system, the computing system comprising: receiving a query execution plan associated with a plurality of query execution pipelines, an estimated execution cost, and an estimated intermediate result cardinality; determining one or more precedence relationships of the plurality of query execution pipelines; determining an execution order of the plurality of query execution pipelines based on the estimated execution cost, the estimated intermediate result cardinality, and the one or more precedence relations; providing the execution order of the plurality of query execution pipelines and the query execution plan to a query execution engine; A non-transitory storage medium that allows

16. The determination of the execution order is determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations; determining a memory cost for each of the plurality of execution orders based on the estimated execution cost and the estimated intermediate result cardinality; determining the one execution order from the plurality of execution orders based on the memory cost of each of the plurality of execution orders; 16. The storage medium of claim 15, comprising:

17. determining the execution orders of the query execution pipelines based on the one or more precedence relations, determining an execution order for the plurality of query execution pipelines having memory costs below a threshold; 17. The storage medium of claim 16, wherein the determined execution orders do not include the execution orders of the query execution pipelines having memory costs below a threshold.

18. The determination of the execution order is determining, for each of the plurality of query execution pipelines, a longest pipeline path to which the query execution pipeline belongs based on the one or more precedence relations; determining the execution order based on the longest pipeline path to which each of the plurality of query execution pipelines belongs; 16. The storage medium of claim 15, comprising:

19. The determination of the execution order is determining a first execution order based on the longest pipeline path to which each of the plurality of query execution pipelines belongs; determining a plurality of execution orders of the plurality of query execution pipelines based on the one or more precedence relations; determining a memory cost for the first execution order based on the estimated execution cost and the estimated intermediate result cardinality; determining a memory cost for each of the plurality of execution orders based on the estimated execution cost and the estimated intermediate result cardinality, wherein determining the memory cost for one execution order of the plurality of execution orders terminates when the memory cost of the one execution order of the plurality of execution orders exceeds the memory cost of the first execution order; determining the one execution order from the plurality of execution orders based on the memory cost of each of the plurality of execution orders; 20. The storage medium of claim 18, comprising:

20. determining the execution orders of the query execution pipelines based on the one or more precedence relations, determining an execution order for the plurality of query execution pipelines having memory costs below a threshold; 20. The storage medium of claim 19, wherein the determined execution orders do not include the execution orders of the query execution pipelines having memory costs below a threshold.

Images