A configurable heterogeneous data real-time synchronization and visualization system and method

Abstract

The invention discloses a configurable heterogeneous data real-time synchronization and visualization system and method, and the system comprises a meta-bin subsystem module which stores historical and real-time incremental meta-data information; a historical data batch processing synchronization subsystem module which is used for acquiring and processing historical metadata information and storing the historical metadata information in a preset data warehouse in a preset format; a real-time synchronization subsystem module which is used for acquiring real-time incremental metadata information, processing the real-time incremental metadata information to complete data type conversion and storing the real-time incremental metadata information in a preset data warehouse; a visual large screen subsystem module which is used for carrying out configuration management on data and user-defined statistical indexes in a preset data warehouse and the visual large screen; And a visualization module which is used for displaying the data on a large screen for a user to check and manage. The system can combine real-time data flow in a big data warehouse Hive with a statistical analysis task in aself-defined statistical index form, is bound with a real-time big screen, and provides a solution of data from an online business system to big data statistical analysis.

Description

technical field [0001] The invention relates to the field of big data technology, in particular to a system and method for real-time synchronization and visualization of configurable heterogeneous data. Background technique [0002] The data synchronization solution between relational database and big data warehouse Hive is reflected in open source products on the one hand: 1. Sqoop is widely used to import offline data from relational database to Hadoop big data platform; 2. Kettle can pass The timestamp field sorts the original full amount of data, and combines the intermediate table to record the timestamp of each update to achieve real-time incremental synchronization of data. On the other hand, it is reflected in the research of peers: 1. Use message middleware to shield the differences between the two heterogeneous databases. For example, a real-time synchronization method for heterogeneous databases based on message middleware has been disclosed. The method includes:...

AI Technical Summary

Problems solved by technology

[0003] Although open source synchronization tools provide powerful functions for data synchronization, they generally need to be deployed separately, and it is difficult to integrate big data statistical analysis platforms together

In addition, Kettle can sort the original full amount of data through the timestamp field, and combine the intermediate table to record the timestamp of each update to achieve real-time incremental synchronization of data. The obvious disadvantages of this method are: 1. For relational data The source performs IO operations, which affects online business performance; 2. Periodic data synchronization, poor real-time performance

Sqooop is mainly used to synchronize offline data. Its disadvantage is that it uses the command line to perform synchronization operations. The use threshold is relatively high and the usability is poor.

The use of message systems in peer research can shield the heterogeneity of heterogeneous databases, but it does not reflect the convenience of use and the convenience of integration with specific statistical analysis tasks

Images