Massive data synchronization system and method

Abstract

The invention relates to a massive data synchronization system and method. The synchronization system comprises an incremental data input interface, a full data input interface, a storage module, a filter module and a data output interface, wherein the incremental data input interface is used for receiving the incremental data of the changed data of a resource system from an incremental data interface of the resource system, and the full data input interface is used for calling a full data interface of a target system and receiving a full data identification code from the target system; the storage module is used for storing the received incremental data and full data identification code, and the filter module is used for filtering the incremental data based on the full data identification code, and using the incremental data corresponding to the full data identification code as filtered data; and the data output interface is used for submitting the filtered data as synchronous data to a synchronous data interface of the target system, and the filtered data is stored in the target system. Through the system and the method, the massive data can be synchronized efficiently, reliably and fast.

Description

technical field [0001] The present application relates to a data synchronization technology, in particular to a self-repairing mass data real-time synchronization system and method. Background technique [0002] In various applications of distributed databases and distributed information systems with certain business relationships, it is often necessary to synchronize some key data in different distributed databases and / or information application systems to maintain data between them consistency. When the amount of data that needs to be synchronized, the independence of the information application system, real-time performance and fault tolerance are not high, usually using methods such as database synchronization, ftp, http download or simple interface call can meet the requirements. [0003] However, the database synchronization method will undoubtedly lead to a considerable part of the database table structure of the two associated information application systems being e...

AI Technical Summary

Problems solved by technology

[0015] The database query method often needs to execute a series of operations such as SQL statement query request, compilation of SQL statement, SQL statement optimization, index search, data collection, and result return. The cost of database query is very expensive

The real-time performance is relatively poor, and it is not suitable for the synchronization of massive data

[0016] The file indexing method is similar to the database query method, except that there is less network transmission, but it needs to maintain the creation of index files for new data, merge with the original index, and other operations, which are also costly

[0017] The memory index method will consume a large amount of memory as the amount of data increases.

[0018] Although the redo log monitoring method can sense data changes by monitoring the database (redo log) of system A in real time or quasi-real time, when only a small part of the changed data volume needs to be synchronized, the existing technology is still This small part of data cannot be effectively filtered out, so there are still problems such as a large number of repeated operations and excessive memory usage

[0019] In addition, database query, file index, memory index, and redo log monitoring methods cannot solve the problem of abnormal fault tolerance (that is, the ability to make errors or exceed service capabilities without crashing) and self-healing problems

Therefore, none of the above solutions can provide multiple requirements such as efficient synchronization of massive data, high degree of decoupling, flow control and self-healing required in practical applications.

Images