A universal multi-source heterogeneous large-scale data synchronization system

A large-scale data, multi-source heterogeneous technology, applied in structured data retrieval, electronic digital data processing, database distribution/replication, etc., can solve problems such as limited data throughput

Active Publication Date: 2021-01-05
GUANGDONG CONSTR INFORMATION CENT
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to overcome the defect of limited data throughput of existing synchronization methods for massive heterogeneous data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A universal multi-source heterogeneous large-scale data synchronization system
  • A universal multi-source heterogeneous large-scale data synchronization system
  • A universal multi-source heterogeneous large-scale data synchronization system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0055] This embodiment provides a universal multi-source heterogeneous large-scale data synchronization system, such as figure 1 As shown, it includes: a synchronization network planning management unit 11 , an installation configuration unit 12 , a block pipeline processing unit 13 , a one-way synchronization unit 14 , a two-way synchronization unit 15 and a synchronization correctness guarantee unit 16 .

[0056] Synchronous network planning management unit 11, used to construct a synchronous topology of a tree structure, the synchronous topology includes a plurality of synchronous pairs formed by a pair of adjacent nodes, and the synchronous pairs include clients (relative lower-level nodes) Compared with the server (relative superior node), the intermediate nodes of the synchronization tree (nodes except leaf and root nodes) can be equipped with both roles of client and server, but they belong to different synchronization pairs. The client includes an intermediate layer so...

Embodiment 2

[0074] This embodiment provides a heterogeneous data synchronization method, which is applicable to the universal multi-source heterogeneous large-scale data synchronization system described in Embodiment 1. The following uses the client as an example to introduce the heterogeneous data synchronization method, as shown in image 3 shown, including the following steps:

[0075] S11: Synchronous configuration initialization stage. The client obtains the metadata information of the source data terminal participating in the synchronization data table and transmits it to the server. Then, the client creates several (preset value) data synchronization threads, and according to the heterogeneous data mapping rules and change capture rule templates obtained from the server, creates "delete", "insert" and "change" three threads on the source data side. Change capture triggers (collectively "Triggers").

[0076] S12: Change event trigger stage. When a change event is triggered, the t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a universal multi-source heterogeneous large-scale data system, including a synchronization network planning management unit, an installation configuration unit, a block pipeline processing unit, a one-way synchronization unit, a two-way synchronization unit and a synchronization correctness guarantee unit. The system works at an intermediate layer above the node database layer and below the application logic layer. During the data synchronization process, the client transmits the captured local change information to the server according to the synchronization task plan; the server receives the change information and stores it in the corresponding message queue by the asynchronous parallel message processing mechanism; the server polls the local message The queue reads the change information to be processed, and then implements subsequent data changes according to the heterogeneous data mapping rules to maintain the consistency of the synchronized data between the source data copy and the target data copy. The system of the invention runs independently in parallel with the local application of the synchronization node, and provides a loose transaction guarantee mechanism for Internet distributed multi-source heterogeneous data synchronization through loosely coupled collaboration.

Description

technical field [0001] The present invention relates to the technical field of Internet data processing, in particular to a loose computing environment oriented to a wide-area network connected through an Internet communication protocol, applicable to a variety of mainstream data system types, and capable of covering a large number of autonomously managed heterogeneous data source nodes Scale data synchronization middle layer system. Background technique [0002] Heterogeneous data refers to data with different structures. The heterogeneity of data is mainly reflected in: the heterogeneity of computer architecture, the physical storage of data comes from computers with different architectures; the heterogeneity of operating systems, the storage of data comes from different The operating system; the data format is heterogeneous, and the data storage management mechanism is different. It can be a relational database system, such as Oracle, SQL Server, DB2, etc., or a file-type...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/27
CPCG06F16/275G06F16/273G06F8/61G06F9/546G06F2209/541G06F9/4881G06F9/5061G06F2209/5018G06F2209/5011G06F16/2246
Inventor 杨海涛徐飞阮镇江
Owner GUANGDONG CONSTR INFORMATION CENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap