Multi-source heterogeneous data interaction analysis engine and method based on SQL (Structured Query Language)

Abstract

The invention discloses a multi-source heterogeneous data interaction analysis engine and method based on SQL, the engine comprises an interaction console, a coordinator, an actuator and a source driver which are connected in sequence, and the method comprises the steps that S1, the interaction console obtains SQL data and feeds back an SQL execution result; s2, the coordinator analyzes the SQL data to generate a logic plan tree, the logic plan tree comprises a series of tasks with dependency relationships, the logic plan tree is split into sub-plans capable of being executed concurrently, each sub-plan comprises a group of tasks, and the sub-plans are scheduled to an executor; s3, the executor obtains data of the data source through the source driver according to the sub-plans, executes tasks in the sub-plans, and feeds SQL execution results back to the interactive console. The purpose of interactive association fusion analysis of multi-source heterogeneous data is achieved, cost is reduced, data accuracy and timeliness are improved, and expansibility and maintainability are improved.

Description

technical field [0001] The invention relates to the technical field of interactive analysis of big data, in particular to a SQL-based multi-source heterogeneous data interactive analysis engine and method. Background technique [0002] In the era of big data, with the explosion of data volume and the diversification of data structure, two core problems that need to be solved urgently for data analysis are brought about: real-time and correlation integration. For application systems, data is often scattered in multiple heterogeneous data storage structures, such as databases, search engines, cache systems, message queues, local files, and so on. Analysis scenarios with multiple sources and different architectures bring difficulties to data association analysis. A business analysis often requires multiple intermediate steps to be completed collaboratively. The actual operation cost is high, and the accuracy and timeliness of the data cannot meet the requirements. In recent ye...

AI Technical Summary

Problems solved by technology

Analysis scenarios with multiple sources and different architectures bring difficulties to data association analysis. A business analysis often requires multiple intermediate steps to be completed in coordination. The actual operation costs are high, and the accuracy and timeliness of data cannot meet the requirements.

In recent years, although the data lake has been proposed to integrate various data sources in a unified manner, its storage-centric characteristics also have many inconveniences in the access and management of massive and diverse data.

Images