Data exchange system for supporting interaction between streaming data and batch data in Gaia system

Abstract

The invention discloses a data exchange system for supporting interaction between streaming data and batch data in a Gaia system, which realizes connection between the streaming data and the batch data at a low latitude operator level of Gaia and can realize directional support for a specified database. In order to achieve the purpose, the system comprises an operator module, a core operator module, a batch data definition module, a cache module and an external data interface module. In the system, a batch data definition module, a cache module and a core operator module can respectively obtain required tool classes from an auxiliary tool module; the core operator module obtains connection related information by calling the batch data definition module, calls the external data interface module to obtain batch data from different data sources, and calls the cache module to cache the data. The operator module carries out external packaging on the core operator module and the batch data definition module, so that the Gaia system can be called in a unified manner.

Description

technical field [0001] The invention relates to the technical field of distributed big data processing, in particular to a data exchange system in a Gaia system that supports the interaction between stream data and batch data. Background technique [0002] The Gaia system is a new generation of big data analysis system with high timeliness and scalability for the hybrid coexistence of multiple computing models. Solve a series of key technical problems at several core levels of big data analysis systems such as adaptive and scalable big data storage, batch-flow fusion big data computing, high-dimensional large-scale machine learning, and high-time-effective big data intelligent interactive guides. Build an independent, controllable, time-effective and scalable new-generation big data analysis system, and master the core technology of the world's leading big data analysis system. [0003] The Gaia system has full-cycle multi-scale optimization and a unified computing engine f...

AI Technical Summary

Problems solved by technology

[0006] The existing dimension table connection technology is based on SQL query, which lowers the threshold for users to use, but also greatly limits the use methods

In the face of complex scenarios that require customized development, SQL query is already a top-level abstraction, and it is impossible to perform business abstraction. Simple SQL query will not be enough to guarantee 100% business logic

At the same time, SQL query only supports a few databases provided by the system itself, and cannot provide targeted support for specified databases

Images