Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Multi-source heterogeneous incremental data synchronization method and system

An incremental data, multi-source heterogeneous technology, applied in the field of data processing, can solve the problems of high pressure for full synchronization, unsatisfactory cycle scheduling, and high cost

Active Publication Date: 2020-09-29
STATE GRID ZHEJIANG ELECTRIC POWER +1
View PDF5 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the current situation where there are many types of business systems and multi-source and heterogeneous underlying data storage, offline batch data synchronization has a large network and disk IO cost, full synchronization pressure is high and time-consuming when the source database has no incremental identification, and offline data synchronization adopts Periodic scheduling cannot meet pain points such as real-time analysis on the business side

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-source heterogeneous incremental data synchronization method and system
  • Multi-source heterogeneous incremental data synchronization method and system
  • Multi-source heterogeneous incremental data synchronization method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] like figure 1 As shown, the embodiment of the present invention provides a method for synchronizing multi-source heterogeneous incremental data, and the method may include the following steps.

[0058] S1. Obtain incremental streaming data of at least one source.

[0059] In practical applications, this step may also be to obtain incremental data at the source. The method in this embodiment implements a method for synchronizing incremental data.

[0060] This step may include the following sub-steps:

[0061] For each source, check whether the data format of the source is JSON semi-structured data format;

[0062] If so, use the data replication tool corresponding to the source data to analyze the data in the database at the source to obtain incremental streaming data at the source;

[0063] Otherwise, use the data replication tool corresponding to the source data to analyze the data in the source database, obtain the incremental streaming data of the source, and ca...

Embodiment 2

[0085] like figure 2 As shown, the embodiment of the present invention provides a flow chart of a method for synchronizing multi-source heterogeneous incremental data. The method of this embodiment may include the following steps:

[0086] Step (1): Configure the data replication middleware for acquiring data changes at the source end and configure the incremental data information to be synchronized in the data transmission service unit.

[0087] First of all, for different relational storage, select the corresponding data replication middleware, obtain the incremental data in the source database in real time, such as incremental change data, and send it downstream.

[0088] In this embodiment, a corresponding data replication tool is selected according to different data storage types at the source. like figure 2 Oracle's source database in Oracle will choose Oracle Golden Gate (ogg for short) data replication middleware to realize real-time synchronization of Oracle datab...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a multi-source heterogeneous incremental data synchronization method and system. The method comprises the steps of obtaining incremental streaming data of at least one sourceend; synchronizing the obtained incremental streaming data of each source end in a distributed message queue Kafka for caching; conducting logic processing on the incremental streaming data in the distributed message queue Kafka according to a storage strategy, storing the processed incremental streaming data in a target data source, wherein the storage strategy is information which is configuredin a data transmission service unit in advance and used for storing the incremental streaming data. According to the method, the incremental streaming data in the relational database can be synchronized to the distributed storage in real time, and high expansibility, high reliability and durability are achieved.

Description

technical field [0001] The invention relates to data processing technology, in particular to a multi-source heterogeneous incremental data synchronization method and system. Background technique [0002] With the development of informatization construction in the power grid industry, many informatization systems have accumulated a large amount of massive data such as marketing business, electricity consumption information, and customer service. Traditional relational databases have prominent limitations in storage, calculation, and innovative applications. In order to improve the basic support capability of power big data analysis and application, it is necessary to build a distributed storage and computing platform, so it is necessary to synchronize the data of the business information system to the distributed storage. [0003] In the current situation where there are many types of business systems and multi-source and heterogeneous underlying data storage, offline batch d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/23G06F16/25G06F16/28G06F9/54
CPCG06F9/546G06F2209/548G06F16/23G06F16/258G06F16/27G06F16/284
Inventor 郑斌胡若云李国良柴成亮孙钢王锦志张爽景伟强陈欢军陆春光吕诗宁
Owner STATE GRID ZHEJIANG ELECTRIC POWER
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com