Increment data capturing method and system

Abstract

The present invention provides a method for capturing incremental data, comprising: creating a relational database snapshot table corresponding to the target table according to the structural information of the target table; using the created snapshot table to complete snapshot extraction of the target table at a specified time point; according to the Snapshots at different time points mentioned above are compared to capture incremental data. The snapshot table is used to store snapshots, and metadata information of incremental data does not need to be maintained, which enhances versatility. Further, the present invention reduces the time for comparison of snapshot records, reduces the computing load of the computer system, saves computer system resources, and improves the efficiency of capturing incremental data by creating a clustered index of the snapshot table. The invention also discloses an incremental data capture system.

Description

technical field [0001] The invention relates to the technical field of databases, in particular to an incremental data capture method and system in the technical field of database data integration. Background technique [0002] Incremental data capture (Change Data Capture, CDC) refers to the process of capturing the incremental data generated in the data source and providing it to other enterprise application software. CDC is one of the main problems to be solved in the process of ETL (Extract Transform Load) in data integration. CDC is used to capture data insertion, deletion, and update operations in business databases, and provides incremental data capture services for the synchronization of enterprise application database data such as OLAP databases, report databases, data warehouses, and business intelligence databases. [0003] The snapshot (Snapshot) comparison method is one of the main methods of incremental data capture in the prior art, and the incremental data i...

AI Technical Summary

Problems solved by technology

The algorithm also includes a large number of IO operations, which makes the time-consuming more serious; and if the number of IO operations is reduced to reduce the time-consuming, it will take up a lot of memory space

In the case of limited memory and a large amount of snapshot data, the execution efficiency of the algorithm will be greatly affected

[0006] 2. It is necessary to maintain database metadata information, which limits the versatility of the algorithm to a certain extent

Therefore, the incremental data capture using files as snapshots must additionally maintain this part of the lost metadata, which increases the complexity of snapshot comparison incremental extraction

Moreover, there are deviations in the definition of metadata between different types of databases, and to provide support for different types of databases requires an extended algorithm, which limits the versatility of the algorithm to a certain extent.

[0008] 3. Does not provide the recovery function after the incremental update of the target database

Images