Cross-platform unified big data SQL query method

Abstract

The invention discloses a cross-platform unified big data SQL (Structured Query Language) query method, which comprises the following steps of: expanding part of SQL semantics, and providing a unifiedcross-platform SQL query language for a user; enabling a unified SQL parser to parse the query statement submitted by the user into a logic query plan, and verifying the legality of the query statement according to the meta information stored in the unified meta database; enabling a cross-platform optimizer to optimize the structure and the connection sequence of the logic query plan and convertthe logic query plan into an optimal physical execution plan composed of a plurality of sub-queries bound with an execution platform; enabling a cross-platform scheduler to convert the optimal physical execution plan into a task flow graph, automatically schedule the task flow graph according to the dependency relationship among the tasks and execute all the tasks; designing a unified platform layer interface meeting cross-platform SQL query requirements, and shielding operation differences between different execution platforms. The cross-platform unified big data SQL query method solves the problems that an existing cross-platform query method is poor in usability, low in performance, large in data migration cost and the like.

Description

Technical field [0001] The invention relates to the field of big data SQL query and automatic scheduling, in particular to a cross-platform unified big data SQL query method. Background technique [0002] SQL is currently the most widely used data query and analysis language. More and more big data systems provide support for SQL. Typical representatives are Apache Hive, Apache Spark SQL, and Apache Impala. In fact, big data SQL query analysis is still one of the most widely used technologies in the industry's big data analysis applications. However, in order to meet the different needs of various industries for big data analysis, many database query systems focusing on different fields have emerged. These systems often have great differences in query languages, data formats, computing models, system architectures, and underlying storage technologies, which greatly increase the learning and use costs of data analysts, and raise the threshold for big data query and analysis. Li...

AI Technical Summary

Problems solved by technology

[0006] Purpose of the invention: Aiming at the current lack of an easy-to-use, unified and efficient cross-platform query method, the purpose of the present invention is to provide an efficient cross-platform unified big data SQL query method, provide users with a unified query language and shield the bottom layer from different The heterogeneity of the platform divides the cross-platform query into multiple sub-queries, and automatically completes the scheduling execution and data migration of sub-queries between multiple execution platforms, so that users only need to pay attention to the query statement itself, thereby greatly improving user query. The efficiency of analysis solves the problems of poor usability, low performance, and huge data migration overhead of existing cross-platform query methods

Images