Increment data capturing method and system

Abstract

The invention provides an incremental data acquisition method and an incremental data acquisition system; wherein, the method comprises that: a relational database snapshot list corresponding to a target list is created according to the structure information of the target list; extraction of snapshots in the target list with time points appointed is completed with the created snapshot list; and snapshot intercomparison is performed according to the snapshots with different time points so as to capture incremental data. The invention has the advantages that the snapshot list is adopted to store snapshots, and maintenance of metadata information of the incremental data is not needed, thus versatility is enhanced; furthermore, intercomparison time of snapshot record is shortened, computer system computation is decreased, and computer system resources are saved through creating the cluster index of the snapshot list.

Description

technical field [0001] The invention relates to the technical field of databases, in particular to an incremental data capture method and system in the technical field of database data integration. Background technique [0002] Incremental data capture (Change Data Capture, CDC) refers to the process of capturing the incremental data generated in the data source and providing it to other enterprise application software. CDC is one of the main problems to be solved in the process of ETL (Extract Transform Load) in data integration. CDC is used to capture data insertion, deletion, and update operations in business databases, and provides incremental data capture services for the synchronization of enterprise application database data such as OLAP databases, report databases, data warehouses, and business intelligence databases. [0003] The snapshot (Snapshot) comparison method is one of the main methods of incremental data capture in the prior art, and the incremental data i...

AI Technical Summary

Problems solved by technology

The algorithm also includes a large number of IO operations, which makes the time-consuming more serious; and if the number of IO operations is reduced to reduce the time-consuming, it will take up a lot of memory space

In the case of limited memory and a large amount of snapshot data, the execution efficiency of the algorithm will be greatly affected

[0006] 2. It is necessary to maintain database metadata information, which limits the versatility of the algorithm to a certain extent

Therefore, the incremental data capture using files as snapshots must additionally maintain this part of the lost metadata, which increases the complexity of snapshot comparison incremental extraction

Moreover, there are deviations in the definition of metadata between different types of databases, and to provide support for different types of databases requires an extended algorithm, which limits the versatility of the algorithm to a certain extent.

[0008] 3. Does not provide the recovery function after the incremental update of the target database

Images