A unified SQL query system supporting multi-source heterogeneous data

Abstract

The present invention provides a unified SQL query system supporting multi-source heterogeneous data, including a metadata management unit, a task management unit, an SQL execution unit and a cache unit; Perform renaming mapping; the task management unit provides an editing page for SQL statements, performs task scheduling and progress monitoring and feedback, and asynchronously transmits SQL statements and execution results to the cache unit; the SQL execution unit reads The data of the software package of the data source is converted into a specific data structure and stored in the resource space corresponding to the resource handle; then each data source data executes the subtasks in the SQL execution plan in turn and returns the execution result to the task management unit; The cache unit stores the sql statement and the execution result into a key-value structure, performs query and returns the query result. Therefore, the query of multi-source heterogeneous data can be completed in one SQL statement, and the programming and operation efficiency is high.

Description

technical field [0001] The invention relates to a unified query system for big data. Background technique [0002] In the big data environment, enterprises are faced with various data structures: relational databases, file data, No-SQL types, streaming data, inverted indexes; data sources are also rich: an enterprise usually has Oracle, SQL server, MySQL Such as multiple relational databases; CSV, Parquet, Json type files of different departments will be stored in different places. Different businesses need to use various types of No-SQL databases such as HBase and MongDB. The existence of multi-source heterogeneous data brings the following difficulties to unified data analysis: [0003] 1. High cost of use and learning: non-relational databases provide non-SQL syntax query primitives, and file-type data needs to be queried by programming, and users have a long learning curve; [0004] 2. Different database engines cannot access each other, and relational databases canno...

AI Technical Summary

Problems solved by technology

[0003] 1. High learning costs: non-relational databases provide query primitives with non-SQL syntax, and file-type data requires programming to implement queries, and the user has a long learning curve;

[0004] 2. Different database engines cannot access each other, and relational databases cannot access each other (Oracle cannot query the table data of SQL server and Mysql); No-SQL databases cannot access each other; relational databases and No-SQL databases They cannot communicate with each other, and users need to frequently switch between query workbenches, which is extremely inconvenient to operate

[0005] 3. Association analysis is difficult. If the data of a SQL query comes from relational databases, No-SQL databases and files, many SQL query engines are powerless at present, and users can only solve it by programming

[0006] 4. Lack of a unified query task management interface to schedule tasks, allocate resources and access management

But it still has the following defects: the data that can be processed is limited to relational databases and MongDB, and it cannot handle file inverted index files; the data structure is an Xml document (on the disk) that stores the mapping between documents and relational tables; the processing engine is Mycat, Mycat It is an open source middleware with great limitations in distributed processing; it does not have a cache system and task management system; in metadata management, both fields and rows must be mapped: document replaces row in relational database; key mapping is Column columm in relational database

Images