Multi-source database real-time fusion system and fusion method thereof

Abstract

The invention discloses a multi-source database real-time fusion system and a fusion method thereof, and belongs to the technical field of database management. The system comprises a source database,a rule definition module, a rule execution module, an operation monitoring module and a big data center; the source database adopts Oracle, and performs memory log mining by utilizing LogMiner in theOracle; the rule definition module comprises source database configuration information, target database configuration information, a rule conversion plug-in and a rule definition result; the rule execution module comprises a rule analysis component, a message queue and a rule execution component; the operation monitoring module is used for counting the number of messages in a message queue by adopting a graph; and the big data center uses an HDFS to store historical data. The problem of low timelines and customizability of database fusion is solved; and by means of the scheme, under the condition that the business operation is not affected, production data can be customized in real time according to the requirements of the data center, so that the timelines and the customizability are achieved.

Description

technical field [0001] The invention relates to a multi-source database real-time fusion system and a fusion method thereof, belonging to the technical field of database management. Background technique [0002] At present, in the process of informatization construction, many enterprise departments have built enterprise big data centers according to their own business and functional needs. Most of the business databases used in the prior art are oracle, and data synchronization mostly adopts two methods: one is manual periodic copying, which uses manual methods to regularly back up data from the business database, desensitizes it by itself, and performs formatting when importing into the data center. Conversion, this method is inefficient, data synchronization is not timely, and the manual workload of format processing is large, which is prone to errors; the second is to use OGG method to synchronize data. As a commercial product mainly used for data replication, OGG can cap...

AI Technical Summary

Problems solved by technology

Most of the business databases used in the prior art are oracle, and data synchronization mostly adopts two methods: one is manual periodic copying, which uses manual methods to regularly back up data from the business database, desensitizes it by itself, and performs formatting when importing into the data center. Conversion, this method is inefficient, data synchronization is not timely, and the manual workload of format processing is large, which is prone to errors; the second is to use OGG method to synchronize data. As a commercial product mainly used for data replication, OGG can capture Oracle's Redo log changes information, and generates ogg messages and sends them out through the transmission process

However, since most of the traditional databases are relational databases, with the rapid increase in the amount of data, the limitations of relational databases are reflected, that is, slow query speed, inflexible structure, redundant, waste of storage space, etc., so in large The construction of the data center requires the use of distributed NOSQL databases, and OGG lacks applicability in data fusion with various NOSQL databases. At the same time, traditional data synchronization tools are mainly aimed at non-computer professional users who can implement operations, so they lack functions. The flexibility of complex data conversion leads to ogg's lack of flexibility in the customization of rules, and it is unable to refer to external dynamic requirements for customized development. Therefore, a corresponding data fusion tool is urgently needed

Images