System and method for extraction, transformation and distribution of data increments

Abstract

The present invention discloses a system and a method for extraction, transformation and distribution of data increments. The system comprises an extraction transformation designer, an extraction transformation actuator and an an extraction transformation monitoring unit, wherein the extraction transformation designer is used for defining a data source of a source database, designing a transformation rule, then defining a data target, and packaging the designed transformation rule into a KTR file; the extraction transformation actuator is used for capturing incremental data of the source database in real time, analyzing the source of the incremental data, transmitting the incremental data to a server which analyzes the obtained information and selectively calls the corresponding transformation rule to perform transformation processing, the KTR file is executed in batches; and the extraction transformation monitoring unit can monitor the running condition of the rule and can control the start and the stop of data synchronization. A technology combining a log analysis technology and ETL is adopted, so that the functions of capturing the incremental data of the source database in real time, performing transformation processing on the incremental data, and finally loading the incremental data into a target database in a quasi-realtime manner are realized.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a system and method for incremental data extraction, conversion and distribution. Background technique [0002] With the in-depth application of big data in human resources and social security services, business systems are becoming larger and larger in terms of data flow, data processing volume, and data storage volume. Whether the business system is stable or not, and whether the system protection and data protection are sound or not have become key issues affecting the normal development of human resources and social security business systems. The security of existing system data cannot be guaranteed, and the extraction speed is low, which cannot meet the needs of big data analysis business. [0003] In actual business, there are big differences between the production database as the source database and the synchronization database as the target database in terms of databas...

AI Technical Summary

Problems solved by technology

The security of existing system data cannot be guaranteed, and the extraction speed is low, which cannot meet the needs of big data analysis business

[0003] In actual business, there are large differences between the production database as the source database and the synchronization database as the target database in terms of database type, database structure, and field structure of the data table. A single synchronous replication cannot satisfy this type of complex synchronization. need

Images