Incremental data pushing system and method

Abstract

The invention provides an incremental data pushing system and method. The incremental data pushing method comprises the following treatments carried out under task allocation and scheduling of a distributed scheduling center: capturing treatment; increment analysis treatment; logic theme merge treatment; intermediate storage treatment; data field filtration treatment; export treatment.

Description

technical field [0001] The present invention relates to a data push system and method, more specifically, to an incremental data push system and method based on incremental synchronization based distributed task decomposition and intermediate data generation and merging logic. Background technique [0002] Due to the development of cloud computing technology, more and more third-party developers use cloud platform technology. A large part of OLAP (On-Line Analytical Processing: Online Analytical Processing) applications require some shared data information based on relational databases, and it is a common requirement to accurately provide data from shared databases to analysis systems at regular intervals. [0003] In the existing technology, the data providing system regularly backs up the full amount of the database, saves the data snapshot at a certain moment in the form of a file, and then directly imports the target data platform or processes the full amount of data thr...

AI Technical Summary

Problems solved by technology

[0004] There are the following disadvantages in the above-mentioned prior art: 1). High data repetition: each data synchronization is a snapshot of the full amount of data captured at a certain moment, and a large part of the data is repeated, which affects calculation and throughput. Performance is a relatively large waste; 2). Poor timeliness: Since it takes a long time to capture the full amount of data, it is impossible to synchronize data more frequently, and there is a big difference between frequently updated data and real data; 3). It has an impact on database performance: every time the data is synchronized, a snapshot of the database needs to be captured, which increases the operating load of the database with high operating pressure. Although it can be solved by increasing the slave library, the slave synchronization also has stability and delay, etc. There are many problems, and it is also unfavorable for permission control; 4). Data synchronization is a single-point operation: due to the principle of full snapshot, the operation must be performed by a single thread. If there is an exception, the operation must start from the beginning

However, if multi-threaded decomposition is used, it will lead to data consistency problems

Images