Query pipeline execution scheduling

Abstract

To provide systems that reduces memory consumption during query processing.SOLUTION: A system is configured to: receive a query execution plan associated with a plurality of query execution pipelines, estimated execution costs and estimated intermediate result cardinalities; determine one or more precedence relationships of the plurality of query execution pipelines; determine an execution order of the plurality of query execution pipelines based on the estimated execution costs, the estimated intermediate result cardinalities, and the one or more precedence relationships; and provide the execution order of the plurality of query execution pipelines and the query execution plan to a query execution engine.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Application No. 63 / 263,703, filed 8 November 2021, the contents of which are incorporated herein by reference for all purposes. [Background technology]

[0002] In traditional database systems, a query optimizer receives a database query and generates a query execution plan based on it. More specifically, the query optimizer determines several candidate execution plans based on the received query, estimates the cost of each execution plan, and selects the plan with the lowest execution cost. The execution engine then executes the selected query execution plan on the database and returns the corresponding result set.

[0003] A query execution plan may conform to a pipeline-based query execution model. According to such a model, a query execution plan consists of multiple execution pipelines. Each execution pipeline takes a base table or result set from a previously executed execution pipeline as input and performs a mixture of unary operators or searches (e.g., hash joins) on the base table / result set to construct another result set. Thus, execution pipelines are interdependent in that the execution of a given execution pipeline may require the preceding execution of one or more other execution pipelines.

[0004] Due to the increase in data size, query workload, and query complexity, especially in cloud scenarios, optimizing memory consumption during query processing is important. For example, modern cloud-based systems face numerous complex online analytical processing (OLAP) queries that reference huge tables. Those queries are typically executed using an execution pipeline as described above. Moreover, optimal use of existing hardware resources in main memory database systems can lead to low operating costs as well as high throughput. Summary of the Invention Means for Solving the Problems

[0005] Particularly in the case of a pipeline-based query execution plan, a system for reducing memory consumption during query processing is desired. Brief Description of the Drawings

[0006] [Figure 1] It is a block diagram of a system for determining a pipeline execution order for a query execution plan according to some embodiments. [Figure 2] It is a diagram illustrating a query execution plan according to some embodiments. [Figure 3] It consists of a flowchart for determining a pipeline execution order for a query execution plan according to some embodiments. [Figure 4] It is a diagram of pipeline specialization of a query execution plan according to some embodiments. [Figure 5] It consists of a flowchart for determining a pipeline execution order for a query execution plan based on the respective memory costs of candidate pipeline execution orders according to some embodiments. [Figure 6] It is a graph illustrating precedence relationships and metadata associated with an execution pipeline of a query execution plan according to some embodiments. [Figure 7] It is a general diagram of a query execution plan subtree illustrating logical entities for determining pipeline memory costs according to some embodiments. [Figure 8] This consists of a flowchart that determines the pipeline execution order for a query execution plan based on the execution pipeline depth according to some embodiments. [Figure 9] This is a pipeline-specific diagram of a query execution plan according to one embodiment. [Figure 10] This graph illustrates precedents and metadata associated with the execution pipeline of a query execution plan according to certain embodiments. [Figure 11] This is a block diagram of a database node according to one embodiment. [Figure 12] This is a diagram of a cloud-based architecture according to one embodiment. [Modes for carrying out the invention]

[0007] The following description is provided to enable those skilled in the art to manufacture and use the embodiments described and describes the best modes intended for carrying out some of the embodiments. However, various modifications will be immediately apparent to those skilled in the art.

[0008] As described above, the query execution plan output by the query optimizer may include several execution pipelines. These pipelines may be executed in various orders while adhering to the query execution plan, but each execution order produces the same result set and is associated with the same execution cost. We have discovered that different pipeline execution orders may consume memory differently (i.e., exhibit different memory consumption profiles). Therefore, to reduce memory consumption, it may be desirable to execute the execution pipelines of the query execution plan in a specific execution order that conforms to the query execution plan, rather than in another execution order that also conforms to the query execution plan.

[0009] The inventors have also discovered a system and method for selecting a pipeline execution sequence from several pipeline execution sequences conforming to a given query execution plan, based on their relative memory consumption profiles. This system and method can be efficiently integrated into a database management system for implementation after query optimization and before planned execution. In this regard, some embodiments leverage various data structures output by the query optimizer, including the query execution plan and various estimated costs and cardinality.

[0010] Figure 1 is a block diagram of system 100 according to some embodiments. Each exemplary element of system 100 may be implemented using any suitable combination of known or to become known computing hardware and / or software. System 100 may comprise components of an isolated or distributed (i.e., multi-node) database system. In some embodiments, two or more elements of system 100 are implemented by a single computing device. One or more elements of system 100 may be implemented as a cloud service (e.g., software as a service, platform as a service).

[0011] The query optimizer 110 receives a query from a client and determines a query execution plan for executing the query on table 124 in database storage 120. The embodiments are not limited to any particular client or any particular query language. The query optimizer 110 may have any suitable query processor that is or may become publicly known. As is publicly known in the art, the query optimizer 110 may generate several alternative query execution plans and calculate the execution cost of each based on metadata 122 of database storage 120. The metadata 122 may define the structure and relationships of table 124 (e.g., database schema) and statistics representing the data of table 124. These statistics may be periodically refreshed by a statistics server (not shown) of system 100.

[0012] The query optimizer 110 selects one of the query execution plans and provides the selected query execution plan to the pipeline ordering component 130. The query execution plan identifies the execution pipeline, pipeline breakers, cardinality estimates of the input tables and intermediate results, and the execution cost for the subtree of the query execution plan. In some embodiments, the query execution plan may also include other data.

[0013] Figure 2 illustrates a representation 200 of elements of a query execution plan received from a query optimizer according to one embodiment. The query execution plan includes input tables R, S, T, U, and V, as well as physical operators 202-208, which are represented as hash joins. Embodiments are not limited to hash joins.

[0014] Each hash join includes a build side and a probe side, as defined by the query execution plan. According to representation 200, the build side is depicted to the right of the hash join, while the probe side is depicted to the left of the hash join. A hash join is defined herein as a pipeline breaker on the build side, and thus terminates pipelines that enter to the right of the hash join. For example, hash join 202 is a pipeline breaker for a pipeline starting from table T, hash join 204 is a pipeline breaker for a pipeline starting from table V, hash join 206 is a pipeline breaker for a pipeline starting from table S, and hash join 208 is a pipeline breaker for a pipeline starting from table U.

[0015] An execution pipeline may therefore depend on intermediate results generated by pipeline breakers of other execution pipelines. For example, intermediate results generated by hash joins 202 and 204 must remain in memory until they are consumed by hash join 208. Once a pipeline produces a new intermediate result (or final result), all intermediate results referenced by it, such as the hash table explored by the current pipeline, can be released from memory.

[0016] Because intermediate results have different sizes (e.g., number of records * record width) and the number of intermediate results maintained in memory changes during execution, the total memory usage during the execution of a query execution plan fluctuates over time. The inventors noted that the execution order of individual pipelines can significantly affect memory usage because it determines the order and time in which intermediate results are generated, stored, and consumed by subsequent pipelines.

[0017] Therefore, the pipeline ordering component 130 determines the execution order of individual pipelines that results in the most desirable memory usage (i.e., the pipeline execution order). According to some embodiments, the pipeline ordering component 130 determines all possible pipeline execution orders that conform to the query execution plan provided by the query optimizer 110. Using the data provided with the query execution plan, for each pipeline execution order, the ordering component 130 estimates the memory usage over time, or memory profile, based on the memory size and the lifetime of intermediate results generated by pipeline breakers during execution according to the execution order. The pipeline execution order associated with the least memory consumption profile may then be selected as the pipeline execution order for executing the query execution plan.

[0018] The pipeline ordering component 130 then provides the execution engine 140 with a query execution plan and the selected pipeline execution order. The execution engine 140 executes the query execution plan according to the selected pipeline execution order, as is known in the art, to generate a result set 150 based on the data in table 124.

[0019] Figure 3 consists of a flowchart of process 300 according to some embodiments. In some embodiments, various hardware elements of system 100 execute program code to perform process 300. Process 300 and all other processes referred to herein may be embodied in processor-executable program code read from one or more non-temporary computer-readable storage media, such as hard disk drives, volatile or non-volatile random access memory, DVD-ROMs, flash drives, and magnetic tapes, and may be executed by one or more processing units, including but not limited to hardware processors, processor cores, and processor threads. In some embodiments, hardwired networks may be used instead of, or in combination with, program code for implementing processes according to some embodiments. Embodiments are therefore not limited to any specific combination of hardware and software.

[0020] Prior to process 300, a query is received, and a query execution plan is generated based on it. The query may be received from any client application by any form of query processor. According to some embodiments, the query is received by the query optimizer of a Structured Query Language (SQL) server. The query may therefore include, but is not limited to, an SQL query.

[0021] A query optimizer, such as query optimizer 110, generates a query execution plan based on the received query, as is known in the art. The generation of the query execution plan may include estimating the execution cost associated with the query execution plan. The cost may be estimated based on the database table to be queried and metadata describing the structure and contents of the table.

[0022] The query execution plan is received in S310 of process 300. The query execution plan includes the execution pipeline tree, the execution costs of the tree and its subtrees, and the estimated cardinality of intermediate results generated by the pipeline breakers of the execution pipeline. The query execution plan may be received by the pipeline ordering component 130 of system 100, but the embodiment is not limited thereto. For example, the functions attributed to the pipeline ordering component 130 herein may be performed entirely or partially by the query optimizer 110.

[0023] The term “tree” is used herein to indicate that the execution pipeline can be viewed as branching outward and downward through various pipeline breakers from a pipeline breaker that produces the final query result, as illustrated by representation 200. The query execution plan describes the inputs to each pipeline breaker and identifies which inputs are build-side and which are probe-side. The pipeline breakers of the query execution plan, their inputs, and build / probe-side information are sufficient to generate a representation of the query execution plan, as illustrated in Figure 2.

[0024] Figure 4 is a pipeline-specific diagram of the query execution plan illustrated in Figure 2. Each pipeline is labeled with a circled character corresponding to the name of the pipeline's input table, in the case of Figure 4. Figure 400 identifies the estimated cardinality (i.e., number of records) of the intermediate results generated by each pipeline breaker.

[0025] As defined herein, each pipeline breaker terminates its build-side pipeline. For example, pipeline T terminates with a hash join with probe-side pipeline S, ending up in a hash table containing an estimated 20 records. Pipeline S terminates with a hash join with probe-side pipeline R, ending up in a hash table containing an estimated 50 records.

[0026] Returning to process 300, the precedence relationships of the execution pipelines are determined in S320. Precedence relationships define which pipelines must be executed before which other pipelines according to the query execution plan. Precedence relationships may be derived from the query execution plan tree. Referring to Figure 4, pipeline T must be executed before pipeline S, and pipeline S must be executed before pipeline R. Similarly, pipeline V must be executed before pipeline U, and pipeline U must be executed before pipeline R. However, there are no precedence relationships between pipelines U, V and pipelines S, T. Precedence relationships may be provided directly by the query optimizer along with the query execution plan according to some embodiments.

[0027] Next, in S330, the execution order of the execution pipeline is determined. The pipeline execution order is determined based on precedence relationships, execution costs, and intermediate result cardinality. For example, all possible pipeline execution orders that conform to the query execution plan are determined first. In this example, the possible pipeline execution orders are (V, U, T, S, R), (V, T, S, U, R), (V, T, U, S, R), (T, S, V, U, R), (T, V, S, U, R), and (T, V, U, S, R).

[0028] For each pipeline execution sequence, a memory usage metric is determined based on the execution cost and intermediate result radicto received in S310. This determination may include determining the memory size and the lifetime of the intermediate results generated by the pipeline breaker during execution according to each execution sequence. Based on the metric, the pipeline execution sequence is determined. For example, the determined pipeline execution sequence may be associated with the best memory usage. Further details of S330 according to some embodiments are described below.

[0029] The query execution plan and the determined pipeline execution order are provided to the execution engine in S340. The execution engine may then execute the query execution plan according to the pipeline execution order to produce a result set, as is known in the art. The memory usage of such an execution is presumably more desirable than that which would result from executing the query execution plan according to a different (e.g., randomly selected) pipeline execution order.

[0030] Figure 5 consists of a flowchart that determines the pipeline execution order for a query execution plan based on the memory cost of each candidate pipeline execution order according to one embodiment. Thus, process 500 may include an implementation of S330 of process 300.

[0031] In S510, all pipeline execution sequences that satisfy the query execution plan's precedence relationships are determined. Figure 6 is a graph illustrating the execution pipeline precedence relationships described above. As previously mentioned, the pipeline execution sequences that satisfy the graph in Figure 6 are (V, U, T, S, R), (V, T, S, U, R), (V, T, U, S, R), (T, S, V, U, R), (T, V, S, U, R), and (T, V, U, S, R). Execution of a query execution plan following one of these pipeline execution sequences may result in the creation and consumption of intermediate results of the query execution plan at different points in time than execution following a different pipeline execution sequence, thereby resulting in different memory consumption profiles.

[0032] FIG. 6 also shows, for each pipeline, the cardinality f of its associated pipeline breaker. This value may be received from and / or determined based on the query optimizer. As described above, the query optimizer provides an estimated cardinality for each intermediate result of the query execution plan. Since each intermediate result corresponds to a pipeline breaker, the cardinality of the pipeline breaker is determined to be the cardinality of its corresponding intermediate result.

[0033] At S520, for each of the pipeline execution orders determined at S510, the lifetime of each pipeline breaker is determined. The lifetime of the pipeline breaker may be determined based on the execution cost of its corresponding pipeline.

[0034] FIG. 7 illustrates the logical entities referred to in the following description of execution cost, breaker lifetime, and memory size. According to some embodiments, the query execution plan includes subtrees s i , i each with an estimated execution cost c i . The subtrees and estimated execution costs are provided by the query optimizer as is known in the art. The intermediate result of subtree s i exhibits a cardinality f i and a row (i.e., record) size w i and is written to the corresponding pipeline breaker b i that defines the end of pipeline p i .

[0035] The execution cost e i for the entire subtree s i is contrasted with the execution cost c i which is the cost of executing only pipeline p i . Since the estimated execution cost c i of subtree s i provided by the query optimizer does not include the cost of writing the result of subtree s i to pipeline breaker b i , this write cost is the execution cost σi This is represented in Figure 7. i is pipeline breaker b i Express the survival time.

[0036] According to some embodiments, pipeline breaker b i survival time l i This corresponds to the pipeline p i The execution time is estimated in S520 by then estimating the execution time of pipeline p. i The execution time is the pipeline execution cost e i This can be estimated by the pipeline p i Execution cost e i is pipeline p i corresponding subtrees i Estimated execution cost c i From, s i all subtrees j Estimated execution cost c j This can be derived by subtracting, where s j is pipeline breaker b j is pipeline p i Includes only the subtrees consumed by pipeline breaker b. i subtrees i and pipeline p i To account for the cost of writing the results, the difference in results is calculated using the pipeline execution time σ. i This is added. Therefore, it will be as shown below.

[0037]

number

[0038] Some embodiments are C out A cost function is used, which is the sum of the output cardins of all operators, including base table scans. out Since the cost function does not distinguish between build costs and probe costs in the cost of hash joins, for example, the cost σ of writing the hash table to the pipeline breakeri C out Since it already contains the cardinal direction of the result, it is set to 0.

[0039] Pipeline breaker b i The survival time of each pipeline p i Execution time e i In addition, pipeline breaker b i It consists of the execution time of other pipelines that must be executed in order for it to be consumed. For the example pipeline execution order (V, T, S, U, R) described above, the lifetime of the pipeline breaker in pipeline V is the sum of the execution times of pipelines V, T, S, and U. Pipeline execution order O=(p1,…,p i ,…,p j ,…,p k ,…,p n ), corresponding pipeline execution cost vector

number

[0040]

number

[0041] Therefore, the pipeline execution order O=(p1,…,p i ,…,p n Pipeline survival time vector for )

number

number

[0042] The memory cost of each pipeline execution sequence is determined in S530. The memory cost of a pipeline execution sequence may be determined based on the determined lifetime and intermediate result radix of each execution pipeline in the pipeline execution sequence. Firstly, pipeline breaker b i Memory size m i However, the corresponding base f i and row size w i m as the product of i =w i ·f i It is estimated to be as follows. Referring to the hash join example, this estimate is based on the assumption that the pipeline breaker consists of a high-density hash table regardless of the hash table implementation. Given the pipeline execution order O=(p0,…,p i ,…,p n Vector of pipeline breaker memory size for )

number

number

[0043] According to some embodiments, the memory cost determined in S530 is the memory integration metric. The memory integration metric is the pipeline breaker memory cost vector.

number

number

[0044]

number

[0045] Therefore, the memory integration amount MI is calculated for all pipeline breakers b i Their survival time over the course of i The memory sizes m that are multiplied by these m i This is the sum of the two. The memory consumption of a pipeline breaker is 0 before it is created, and as soon as each pipeline starts operating, it becomes b i Until it is destroyed i It is 0, and then it is 0.

[0046] In S540, the pipeline execution order with the lowest memory cost is determined. This determined pipeline execution order, along with the corresponding query execution plan, may then be provided to the execution engine for the execution of the query execution plan's execution pipeline in the determined pipeline execution order.

[0047] According to some embodiments, the determination of the pipeline execution order for all pipelines that satisfy the precedence relationship in S510 ignores pipelines where the pipeline breaker is associated with low memory cost. Such pipeline breakers are associated with short lifetime, small memory size, or both. By ignoring such pipelines, the number of pipeline execution orders determined in S510 can be significantly reduced, leading to the overall fast execution of process 500.

[0048] For example, suppose pipeline V is associated with a low memory cost and is therefore selected to be ignored in S510. The pipeline execution order that satisfies the dependencies in the graph of Figure 6 is therefore determined in S510 as (U, T, S, R), (T, S, U, R), and (T, U, S, R), which is half the pipeline execution order determined in the above example in S510. Once the pipeline execution order associated with the lowest memory cost among these three pipeline execution orders is determined in S540, pipeline V is added to the determined pipeline execution order in a position within the order that conforms to the query execution plan dependencies.

[0049] Figure 8 is a flowchart of process 800 according to one embodiment. Process 800 may be executed to determine a single pipeline execution sequence from a plurality of pipeline execution sequences conforming to a given query execution plan. Some embodiments of process 800 do not utilize the cardinality or execution cost of intermediate results provided by the query optimizer to determine the pipeline execution sequence.

[0050] Rather, process 800 is a heuristic algorithm intended to determine a pipeline execution order with a memory cost that, while low, is not necessarily the lowest memory cost of all pipeline execution orders conforming to the target query execution plan. In exchange for a near-optimal selection of the pipeline execution order, process 800 can run much faster than process 500 in Figure 5. The heuristic approach of process 800 is based on the observation that the deepest first execution of interdependent pipelines will give the near-optimal pipeline execution order. In other words, the execution of subtrees in length order from the start node to the result node is the "best guess" pipeline execution order.

[0051] Figure 9 is a pipeline-specific diagram of a query execution plan intended to illustrate a specific embodiment of process 800. The query execution plan in Figure 9 includes nine base tables and eight intermediate hash tables. Figure 10 represents the query execution plan pipeline as nodes in a dependency graph.

[0052] In process 800, at S810, the length of each subtree in the execution pipeline is determined. For the example in Figure 10, the subtrees and their corresponding lengths determined at S810 are as follows: (E, D, B, A, R; 5), (C, B, A, R; 4), (U, T, S, R; 4).

[0053] Next, in S820, the execution priority is determined for each execution pipeline based on the longest subtree to which the execution pipeline belongs. Continuing in this example, execution pipelines E, D, B, A, and R are assigned a priority of 5, execution pipeline C is assigned a priority of 4, and execution pipelines U, T, and S are assigned a priority of 4. Since execution pipeline C is assigned a priority of 4, and execution pipelines U, T, and S are also assigned a priority of 4, potential ordering conflicts can be addressed by incrementing the priority of either execution pipeline C or execution pipelines U, T, and S, and similarly incrementing the priorities of all higher-priority pipelines. In this example, as shown by the dotted box in Figure 10, the priority of execution pipeline C is incremented from 4 to 5 due to its relationship to higher-priority execution pipelines B, A, and R. Accordingly, the priorities of higher-priority execution pipelines E, D, B, A, and R are also incremented from 5 to 6.

[0054] The pipeline execution order is determined in S830 based on the assigned priority. The determination of the pipeline execution order is primarily based on the dependencies of the query execution plan. However, as described above, at a given execution point, two or more pipelines may be selected as the next pipeline to be executed, while still adhering to the dependencies. In some embodiments of S830, the next pipeline selected is the one with the highest priority among the two or more pipelines.

[0055] Referring to this example, execution pipelines E, C, and U could be the first pipeline in the pipeline execution order. However, since execution pipeline E is assigned the highest priority among the three pipelines, execution pipeline E is selected. The next execution pipeline can be D, C, or U, based on the dependency graph. Since execution pipeline D is assigned the highest priority among the three pipelines, execution pipeline D is selected as the next execution pipeline. At this execution point, only pipelines C or U can be executed (for example, pipeline B cannot be executed because it requires an intermediate result generated by pipeline C). Pipeline C is selected for the next execution due to its higher assigned priority. The same logic continues for the remaining execution pipelines as described above, leading to a determined pipeline execution order (E, D, C, B, A, U, T, S, R).

[0056] According to some embodiments, process 800 is executed to determine the pipeline execution order relatively quickly, and the memory cost of this determined pipeline execution order is determined as described above. Next, process 500 is executed as described above (with or without pipeline ignoring modifications). However, if the lower result determined in S530 during the determination of the memory cost of a particular pipeline execution order exceeds the memory cost of the pipeline execution order determined via process 800, the determination of the memory cost of that particular pipeline execution order is terminated.

[0057] Figure 11 is a block diagram of a database architecture that can determine the pipeline execution order for a query execution plan according to some embodiments. Embodiments are not limited to the architecture of Figure 11.

[0058] Server node 1110 may receive a query from one of client applications 1130 and 1140 and return a result based on data stored within server node 1110. Node 1110 executes program code to provide application server 1115 and query processor 1120. Application server 1115 provides services for running server applications. For example, a web application running on application server 1115 may receive a Hypertext Transfer Protocol (HTTP) request from client application 1140, as shown in Figure 11.

[0059] The query processor 1120 may include stored data and an engine for processing the data. The query processor 1120 may also be responsible for processing structured query language (SQL) and multidimensional expression (MDX) statements, and may receive such statements directly from the client application 1130.

[0060] The query processor 1120 includes a query optimizer 1122 for use in determining a query execution plan, a pipeline ordering component 1123 for determining the pipeline execution order for the query execution plan as described herein, and an execution engine 1124 for executing the query execution plan against a table 1126 in storage 1125 using the determined pipeline execution order. In some embodiments, the query processor 1120 may also include a statistics server (not shown) for determining statistics used to estimate the query execution plan cost.

[0061] In some embodiments, the data in storage 1125 may include one or more conventional tabular data, row-storage data, column-storage data, and object-based data. Furthermore, the data may be indexed and / or selectively copied to the index to enable its high-speed retrieval and reading. Server node 1110 may support multi-tenancy to support multiple unrelated clients separately by providing multiple logical database systems that are programmatically isolated from one another.

[0062] Metadata 1128 contains data describing the database schema that table 1126 examines. Metadata 1128 may therefore describe the columns and properties of table 1126, the properties of each column in each table 1126, the relationships between columns, and any other relevant information. For example, metadata 1128 may identify one or more columns of table 1126 as dictionary-compressed and include information for finding the column dictionary and the dictionary index associated with each dictionary-compressed column.

[0063] Server node 1110 may implement storage 1125 as an "in-memory" database where the entire database is stored in volatile (e.g., non-disk-based) memory (e.g., random-access memory). The entire database may be persistently and / or backed up to a fixed disk (not shown). Embodiments are not limited to an in-memory implementation. For example, data may be stored in random-access memory (e.g., cache memory for storing recently used data) and one or more fixed disks (e.g., persistent memory for storing those respective parts of the entire database).

[0064] Figure 12 illustrates a cloud-based database deployment according to one embodiment. The illustrated components may reside on one or more public clouds that provide self-service and immediate provisioning, auto-scaling, security, compliance, and identity management capabilities.

[0065] The user device 1210 may interact with applications running on the application server 1220, for example, via a web browser running on the user device 1210, to create, read, update, and delete data managed by the database system 1230 and persistent in the distributed file storage 1235. The database system 1230 may store data and may perform processes as described herein to determine the pipeline execution order for executing query execution plans on the data. The application server 1220 and / or the database system 1230 may have cloud-based computing resources, such as virtual machines, allocated by a public cloud provider. Thus, the application server 1220 and the database system 1230 may exhibit demand-based resilience.

[0066] The figures above represent a logical architecture for illustrating a process according to some embodiments, and actual implementations may include more or different components arranged in other ways. Other topologies may be used in conjunction with other embodiments. Furthermore, each component or device described herein may be implemented by any number of devices communicating over any number of other public and / or private networks. Two or more such computing devices may be located apart from each other and may communicate with each other over any known means of network and / or dedicated connections. Each component or device may comprise any number of hardware and / or software elements appropriate to provide the functions described herein and any other functions. For example, any computing device used in an implementation described herein may include a programmable processor that executes program code to operate the computing device as described herein.

[0067] All systems and processes described herein may be embodied in program code stored on one or more non-temporary computer-readable storage media. Such storage media may include, for example, DVD-ROMs, flash drives, magnetic tapes, and solid-state random-access memory (RAM) or read-only memory (ROM) storage units. Embodiments are therefore not limited to any particular combination of hardware and software.

[0068] Elements described herein as communicating with one another can communicate directly or indirectly through any number of different systems for transferring data, including but not limited to shared memory communications, local area networks, wide area networks, telephone lines, cellular networks, fiber optic networks, satellite networks, infrared networks, radio frequency networks, and any other types of networks that may be used to transmit information between devices. Furthermore, communication between systems may proceed through any one or more known or to become known transmission protocols, such as Asynchronous Transfer Mode (ATM), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), and Wireless Application Protocol (WAP).

[0069] The embodiments described herein are for illustrative purposes only. Those skilled in the art will recognize that other embodiments can be carried out by modifications and alterations to those described above. [Explanation of Symbols]

[0070] 100 Systems 110 Query Optimizer 120 Database Storage 122 Metadata 124 Tables 130 Pipeline Ordering Components 140 Execution Engines 150 result sets 202,204,206,208 hash join 1110 Server Nodes 1115 Application Server 1120 Query Processors 1122 Query Optimizer 1123 Pipeline Sequencing Component 1124 Execution Engine 1125 Storage 1126 Table 1128 Metadata 1130,1140 Client Applications 1210 User Device 1220 Application Server 1230 Database System 1235 Distributed File Storage

Claims

1. It is a system, Memory containing processor-executable program code, Processing unit and Equipped with, The processing unit executes the processor-executable program code and puts the system into action. Receiving a query execution plan associated with the query execution pipeline, estimated execution cost, and estimated intermediate result cardinality, The aforementioned query execution pipeline tree is divided into multiple subtrees by a pipeline breaker. The aforementioned estimated execution cost is the estimated execution cost of each subtree. The aforementioned estimated cardinal of intermediate results is the estimated cardinal of the intermediate results obtained by executing each subtree, Determining one or more precedence relationships of the multiple subtrees of the query execution pipeline, Determining the execution order of the multiple subtrees based on the one or more precedence relationships, For each of the above-mentioned multiple execution sequences, the memory size of the intermediate results of each subtree and the lifetime of the intermediate results of each subtree are determined based on the estimated execution cost and the estimated intermediate result cardinality. Based on the memory size and lifespan of the intermediate results of each subtree determined for each of the plurality of execution orders, one execution order of the plurality of subtrees is determined from the plurality of execution orders. To provide the query execution engine with the execution sequence of the plurality of subtrees and the query execution plan. A system that enables this to happen.

2. The determination of the execution order mentioned above is Determination of the memory cost of each of the multiple execution sequences based on the memory size and lifespan of the intermediate results of each subtree determined for each of the multiple execution sequences, The determination of one execution order from the plurality of execution orders based on the memory cost determined for each of the plurality of execution orders, The system according to claim 1, including the following:

3. The determination of one execution order from the aforementioned plurality of execution orders is The system according to claim 2, comprising determining the execution order having the lowest memory cost among the plurality of execution orders.

4. The determination of the execution order of the multiple subtrees based on the one or more precedence relationships is This includes determining the execution order of the plurality of subtrees having a memory cost below a threshold, The system according to claim 2, wherein the determined plurality of execution orders does not include the execution order of the plurality of subtrees having a memory cost below a threshold.

5. The determination of the execution order mentioned above is Determination of the longest tree path of the query execution pipeline based on the one or more precedence relationships, Determination of the first execution order based on the longest tree path, Determination of the memory cost of the first execution order based on the memory size and lifespan of the intermediate results of each subtree determined for the first execution order, A determination of the memory cost of each of the plurality of execution sequences based on the memory size and lifetime of the intermediate results of each subtree determined for each of the plurality of execution sequences, wherein the determination of the memory cost of one of the plurality of execution sequences terminates when the memory cost of the one of the plurality of execution sequences exceeds the memory cost of the first execution sequence, The determination of one execution order from the plurality of execution orders based on the memory cost determined for each of the plurality of execution orders, The system according to claim 1, including the following:

6. The determination of the execution order of the multiple subtrees based on the one or more precedence relationships is This includes determining the execution order of the plurality of subtrees having a memory cost below a threshold, The system according to claim 5, wherein the determined plurality of execution orders does not include the execution order of the plurality of subtrees having a memory cost below a threshold.

7. A method performed by a computer, A step of receiving a query execution plan associated with a query execution pipeline, estimated execution cost, and estimated intermediate result cardinality, The aforementioned query execution pipeline tree is divided into multiple subtrees by a pipeline breaker. The aforementioned estimated execution cost is the estimated execution cost of each subtree. The estimated cardinality of the intermediate results is the estimated cardinality of the intermediate results obtained by executing each subtree, step by step. The steps include determining one or more precedence relationships of the plurality of subtrees in the query execution pipeline, The steps include determining a plurality of execution orders of the plurality of subtrees based on the one or more precedence relationships, For each of the plurality of execution sequences, the steps include determining the memory size of the intermediate results of each subtree and the lifetime of the intermediate results of each subtree based on the estimated execution cost and the estimated intermediate result cardinality, A step of determining one execution order of the plurality of subtrees from the plurality of execution orders based on the memory size and lifespan of the intermediate results of each subtree determined for each of the plurality of execution orders, The steps include providing the query execution engine with the execution sequence of the plurality of subtrees and the query execution plan. A method that includes this.

8. The step of determining the execution order is, A step of determining the memory cost of each of the plurality of execution sequences based on the memory size and lifespan of the intermediate results of each subtree determined for each of the plurality of execution sequences, A step of determining one execution order from the plurality of execution orders based on the memory cost determined for each of the plurality of execution orders. The method according to claim 7, including the method described in claim 7.

9. The step of determining one execution order from the plurality of execution orders is, The method according to claim 8, further comprising the step of determining the execution order having the lowest memory cost among the plurality of execution orders.

10. The step of determining a plurality of execution orders of the plurality of subtrees based on the one or more precedence relationships, The step includes determining the execution order of the plurality of subtrees having a memory cost below a threshold, The method according to claim 8, wherein the determined plurality of execution orders does not include the execution order of the plurality of subtrees having a memory cost below a threshold.

11. The step of determining the execution order is The steps include determining the longest tree path of the query execution pipeline based on the one or more precedence relationships, The steps include determining a first execution order based on the longest tree path, A step of determining the memory cost of the first execution sequence based on the memory size and lifespan of the intermediate results of each subtree determined for the first execution sequence, A step of determining the memory cost of each of the plurality of execution sequences based on the memory size and lifetime of the intermediate results of each subtree determined for each of the plurality of execution sequences, wherein the determination of the memory cost of one of the plurality of execution sequences is terminated when the memory cost of the one of the plurality of execution sequences exceeds the memory cost of the first execution sequence. A step of determining one execution order from the plurality of execution orders based on the memory cost determined for each of the plurality of execution orders. The method according to claim 7, including the method described in claim 7.

12. The step of determining a plurality of execution orders of the plurality of subtrees based on the one or more precedence relationships, The step includes determining the execution order of the plurality of subtrees having a memory cost below a threshold, The method according to claim 11, wherein the determined plurality of execution orders does not include the execution order of the plurality of subtrees having a memory cost below a threshold.

13. A non-temporary storage medium that stores processor-executable program code, The processor-executable program code is executable by the computing system, and the computing system Receiving a query execution plan associated with the query execution pipeline, estimated execution cost, and estimated intermediate result cardinality, The aforementioned query execution pipeline tree is divided into multiple subtrees by a pipeline breaker. The aforementioned estimated execution cost is the estimated execution cost of each subtree. The aforementioned estimated cardinal of intermediate results is the estimated cardinal of the intermediate results obtained by executing each subtree, Determining one or more precedence relationships of the multiple subtrees of the query execution pipeline, Determining the execution order of the multiple subtrees based on the one or more precedence relationships, For each of the above-mentioned multiple execution sequences, the memory size of the intermediate results of each subtree and the lifetime of the intermediate results of each subtree are determined based on the estimated execution cost and the estimated intermediate result cardinality. Based on the memory size and lifespan of the intermediate results of each subtree determined for each of the plurality of execution orders, one execution order of the plurality of subtrees is determined from the plurality of execution orders. To provide the query execution engine with the execution sequence of the plurality of subtrees and the query execution plan. A non-temporary storage medium that enables the operation of [a certain function].

14. The determination of the execution order mentioned above is Determination of the memory cost of each of the multiple execution sequences based on the memory size and lifespan of the intermediate results of each subtree determined for each of the multiple execution sequences, The determination of one execution order from the plurality of execution orders based on the memory cost determined for each of the plurality of execution orders, A storage medium according to claim 13, including the storage medium described in claim 13.

15. The determination of the execution order of the multiple subtrees based on the one or more precedence relationships is This includes determining the execution order of the plurality of subtrees having a memory cost below a threshold, The storage medium according to claim 14, wherein the determined plurality of execution orders does not include the execution order of the plurality of subtrees having a memory cost below a threshold.

16. The determination of the execution order mentioned above is Determination of the longest tree path of the query execution pipeline based on the one or more precedence relationships, Determination of the first execution order based on the longest tree path, Determination of the memory cost of the first execution order based on the memory size and lifespan of the intermediate results of each subtree determined for the first execution order, A determination of the memory cost of each of the plurality of execution sequences based on the memory size and lifetime of the intermediate results of each subtree determined for each of the plurality of execution sequences, wherein the determination of the memory cost of one of the plurality of execution sequences terminates when the memory cost of the one of the plurality of execution sequences exceeds the memory cost of the first execution sequence, The determination of one execution order from the plurality of execution orders based on the memory cost determined for each of the plurality of execution orders, A storage medium according to claim 13, including the storage medium described in claim 13.

17. The determination of the execution order of the multiple subtrees based on the one or more precedence relationships is This includes determining the execution order of the plurality of subtrees having a memory cost below a threshold, The storage medium according to claim 16, wherein the determined plurality of execution orders does not include the execution order of the plurality of subtrees having a memory cost less than a threshold.

Images