Batch data query method and device

Abstract

The embodiment of the invention provides a batch data query method and a device. The method comprises following steps: obtaining multiple logic inquiry plan trees corresponding to multiple query statements; diving multiple query groups by all operators of each logical query plan tree, wherein logical query plan trees of each query group are in the relationships as follows: inter-query write-read dependency relationships; overlapping relationships between operators or operand overlapping relationships; optimizing inter-query logical query plan trees of each query group and generating multiple corresponding physical query plan trees based on multiple optimized query plan trees; and finally, executing physical obtaining query results of multiple sentences based on multiple physical query plan trees. The method helps to increase optimization opportunities between logical plan trees in query groups by grouping the multiple logical query plan trees. By performing inter-query optimization on the logical query trees of query groups, efficiency of batch data query is increased.

Description

technical field [0001] The embodiment of the present invention relates to data warehouse query optimization technology, in particular to a batch data query method and device. Background technique [0002] At present, representative big data query systems (such as: Hive, Shark, Impala) all use a single query statement as the basic unit of parsing and optimization. The basic query processing flow is as follows: First, parse the read single query statement into a tree The logical query plan tree of the structure, and then, by selecting and implementing the algorithm for each operator of the logical query plan tree, and determining the execution order of these operators, the logical query plan tree is converted into a physical query plan, and finally, the query execution engine Execute the physical query plan and output the query result. [0003] A key performance of a big data query system is query efficiency. At present, the commonly used method to improve the query efficienc...

AI Technical Summary

Problems solved by technology

[0004] In sharp contrast to the lack of opportunities for intra-query optimization, there are abundant inter-query optimization opportunities in the data warehouse batch query application scenario. Inter-query optimization opportunities refer to the optimization opportunities between multiple query statements. In the batch query application scenario Therefore, there are a lot of opportunities for query optimization. However, in the prior art, query optimization is only performed for a single query statement, which makes the query efficiency of big data query low.

Images