Distributed data synchronization method and system

Abstract

The invention relates to a distributed data synchronization method and system. The method comprises the following steps that: a scheduler inquires whether a synchronization task needs to be carried out from a source database, and elastically adjusts the query interval time of the scheduler according to a query result; the scheduler selects one synchronization component in a synchronization component cluster to execute the task; the synchronization component performs task segmentation and adopts a plurality of concurrent threads for processing; the synchronization component reads a synchronization data set from the source database and carries out data processing; and the synchronization component generates temporary unique identifiers to replace original unique identifiers of the synchronous data set, then writes the synchronous data set into the target database in batches, deletes updated old data records in the target database, and finally restores the original unique identifiers of the synchronous data set in batches. According to the distributed data synchronization method and system, the whole data synchronization performance is greatly improved.

Description

technical field [0001] The invention relates to data synchronization technology, in particular to a distributed data synchronization method and system. Background technique [0002] Data synchronization is a means to maintain data consistency among multiple data sources, and is widely used in various computer systems. Data synchronization has the following common application scenarios: 1. The computer system realizes disaster recovery and backup through data synchronization to avoid single point of failure; 2. The computer system realizes system load balancing through data synchronization and improves system performance; Data sharing is realized through data synchronization. There are generally two implementation methods for data synchronization. One is to read the database BinLog file, obtain the modification operation statement in it and send it to the database to be synchronized for execution, so as to realize data synchronization. The second way is to query the records...

AI Technical Summary

Problems solved by technology

While satisfying the low synchronization delay, frequent queries bring a large performance loss to the source database and synchronization middleware. On the contrary, if the query density is reduced, the synchronization delay will be increased, and it is difficult to meet both requirements at the same time.

[0004] (2) Traditional data extraction has no way to take advantage of the multi-core advantages of modern computers, and the extraction efficiency is relatively low

[0005] (3) Traditional data extraction middleware cannot handle a large number of synchronization tasks at the same time. When there are too many data tables to be synchronized, the synchronization delay will increase and even the system will crash.

[0006] (4) Traditional data extraction middleware has no way to roll back the transmission content of this synchronization task when an exception occurs when writing to the target data table, resulting in dirty data in the target data table

Images