A configurable heterogeneous data real-time synchronization and visualization system and method

A technology of real-time synchronization and heterogeneous data, applied in the field of big data, it can solve the problems of difficult integration of big data statistical analysis platforms, poor real-time performance, and high threshold for use, and achieve the effect of convenient use.

Active Publication Date: 2019-05-31
BEIJING UNIV OF POSTS & TELECOMM
View PDF5 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Although open source synchronization tools provide powerful functions for data synchronization, they generally need to be deployed separately, and it is difficult to integrate big data statistical analysis platforms together
In addition, Kettle can sort the original full amount of data through the timestamp field, and combine the intermediate table to record the timestamp of each update to achieve real-time incremental synchronization of data. The obvious disadvantages of this method are: 1. For relational data The source performs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A configurable heterogeneous data real-time synchronization and visualization system and method
  • A configurable heterogeneous data real-time synchronization and visualization system and method
  • A configurable heterogeneous data real-time synchronization and visualization system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0033] The system and method for real-time synchronization and visualization of configurable heterogeneous data according to the embodiments of the present invention will be described below with reference to the accompanying drawings. First, the real-time synchronization and visualization of configurable heterogeneous data according to the embodiments of the present invention will be described with reference to the accompanying drawings system.

[0034] figure 1 It is a schematic structural diagram of a system for real-time sync...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a configurable heterogeneous data real-time synchronization and visualization system and method, and the system comprises a meta-bin subsystem module which stores historical and real-time incremental meta-data information; a historical data batch processing synchronization subsystem module which is used for acquiring and processing historical metadata information and storing the historical metadata information in a preset data warehouse in a preset format; a real-time synchronization subsystem module which is used for acquiring real-time incremental metadata information, processing the real-time incremental metadata information to complete data type conversion and storing the real-time incremental metadata information in a preset data warehouse; a visual large screen subsystem module which is used for carrying out configuration management on data and user-defined statistical indexes in a preset data warehouse and the visual large screen; And a visualization module which is used for displaying the data on a large screen for a user to check and manage. The system can combine real-time data flow in a big data warehouse Hive with a statistical analysis task in aself-defined statistical index form, is bound with a real-time big screen, and provides a solution of data from an online business system to big data statistical analysis.

Description

technical field [0001] The invention relates to the field of big data technology, in particular to a system and method for real-time synchronization and visualization of configurable heterogeneous data. Background technique [0002] The data synchronization solution between relational database and big data warehouse Hive is reflected in open source products on the one hand: 1. Sqoop is widely used to import offline data from relational database to Hadoop big data platform; 2. Kettle can pass The timestamp field sorts the original full amount of data, and combines the intermediate table to record the timestamp of each update to achieve real-time incremental synchronization of data. On the other hand, it is reflected in the research of peers: 1. Use message middleware to shield the differences between the two heterogeneous databases. For example, a real-time synchronization method for heterogeneous databases based on message middleware has been disclosed. The method includes:...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/26G06F16/25G06F16/27G06F16/28
Inventor 鄂海红宋美娜刘行行
Owner BEIJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap