A data exchange system that supports interaction between stream data and batch data in the gaia system

Abstract

The invention discloses a data exchange system supporting the interaction of stream data and batch data in the Gaia system, which realizes the connection of stream data and batch data at the low-latitude operator level of Gaia, and can realize directional support for a specified database. To achieve the above purpose, the system includes operator module, core operator module, batch data definition module, cache module and external data interface module. In this system, the batch data definition module, the cache module, and the core operator module can respectively obtain the tools they need from the auxiliary tool module; the core operator module obtains connection-related information by calling the batch data definition module, and calls the external data interface The module obtains batch data from different data sources, and calls the cache module to cache the data. The operator module externally encapsulates the core operator module and batch data definition module, so that the Gaia system can be called uniformly.

Description

technical field [0001] The invention relates to the technical field of distributed big data processing, in particular to a data exchange system in a Gaia system that supports the interaction between flow data and batch data. Background technique [0002] The Gaia system is a new generation of big data analysis system with high timeliness and scalability for the hybrid coexistence of multiple computing models. Solve a series of key technical problems at several core levels of big data analysis systems such as adaptive and scalable big data storage, batch-flow fusion big data computing, high-dimensional large-scale machine learning, and high-time-effective big data intelligent interactive guides. Build an independent, controllable, time-effective and scalable new-generation big data analysis system, and master the core technology of the world's leading big data analysis system. [0003] The Gaia system has full-cycle multi-scale optimization and a unified computing engine for...

AI Technical Summary

Problems solved by technology

[0006] The existing dimension table connection technology is based on SQL query, which lowers the threshold for users to use, but also greatly limits the use methods

In the face of complex scenarios that require customized development, SQL query is already a top-level abstraction, and it is impossible to perform business abstraction. Simple SQL query will not be enough to guarantee 100% business logic

At the same time, SQL query only supports a few databases provided by the system itself, and cannot provide targeted support for specified databases

Images