OLAP query engine dynamic cost evaluation method and device

Abstract

The invention discloses an OLAP query engine dynamic cost evaluation method and device, and the method comprises the steps: collecting the data size and field cardinal number of a data table in a data warehouse; meanwhile, historical query sql and corresponding query time are recorded; performing discrete processing on the execution time of the sql to generate a label; meanwhile, table information, a filtering field, an analysis field, a time partition, a table data size and an analysis field cardinal number of the data table are used for generating respective feature codes; performing classification algorithm training on an existing query sample according to the extracted features of the data table by using an existing machine learning model; and converting the execution plan of the OLAP query engine into a feature vector, and inputting the feature vector into the machine learning model for cardinality evaluation, thereby obtaining the estimated query time of the data table. According to the method and the device, the OLAP query engine can quickly and accurately perform cost evaluation on the input query sql to generate the optimal execution plan.

Description

technical field [0001] The invention relates to the field of OLAP query engines, in particular to an OLAP query engine dynamic cost evaluation method and device. Background technique [0002] Currently, the mainstream open source OLAP (On-Line Analytical Processing) query engines on the market, including Hive, Spark SQL, Presto, Kylin, Impala, Druid, Clickhouse, Greplum, etc., use a rule-based approach to cost assessment, such as figure 1 As shown, the following problems currently exist: [0003] 1. Low accuracy: information such as table data volume and field cardinality has a great impact on query performance. Currently, the rule-based cost evaluation engine cannot use this information to perform optimal cost evaluation for SQL. [0004] 2. No dynamic improvement ability: no self-learning ability, unable to dynamically learn query records, improve and improve the accuracy of cost evaluation. Contents of the invention [0005] In order to solve the above-mentioned prob...

AI Technical Summary

Problems solved by technology

[0003] 1. Low accuracy: information such as table data volume and field cardinality has a great impact on query performance. Currently, the rule-based cost evaluation engine cannot use this information to perform optimal cost evaluation for SQL.

[0004] 2. No dynamic improvement ability: no self-learning ability, unable to dynamically learn query records, improve and improve the accuracy of cost evaluation

Images