System and method for extraction, transformation and distribution of data increments

A technology of incremental extraction and distribution system, which is applied in the field of data incremental extraction conversion and distribution system, which can solve problems such as low extraction speed, inability to meet the needs of big data analysis business, synchronous replication and inability to meet synchronization requirements, etc.

Active Publication Date: 2015-11-18
DAREWAY SOFTWARE
View PDF3 Cites 49 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The security of existing system data cannot be guaranteed, and the extraction speed is low, which cannot meet the needs of big data analysis business
[0003] In actual business, there are large differences between the production database as the sou

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for extraction, transformation and distribution of data increments

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] Incremental data extraction, conversion and distribution systems, such as figure 1 As shown, it includes three parts: decimate transformation designer, decimate transformation executor and decimate transformation monitor unit.

[0048] The extraction transformation designer is implemented based on ETL, and adopts the streamlined design pattern of "Define Data Source >> Design Transformation Rules >> Define Data Target", and through the flow formed by connecting nodes, the data information or fields are transferred from the previous Module is transferred to the next module. In defining the data source, define the connection information of the data source; in the design of the conversion rule, use the powerful conversion design function of kettle to design a variety of conversion rules; in defining the data target, define the connection informatio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention discloses a system and a method for extraction, transformation and distribution of data increments. The system comprises an extraction transformation designer, an extraction transformation actuator and an an extraction transformation monitoring unit, wherein the extraction transformation designer is used for defining a data source of a source database, designing a transformation rule, then defining a data target, and packaging the designed transformation rule into a KTR file; the extraction transformation actuator is used for capturing incremental data of the source database in real time, analyzing the source of the incremental data, transmitting the incremental data to a server which analyzes the obtained information and selectively calls the corresponding transformation rule to perform transformation processing, the KTR file is executed in batches; and the extraction transformation monitoring unit can monitor the running condition of the rule and can control the start and the stop of data synchronization. A technology combining a log analysis technology and ETL is adopted, so that the functions of capturing the incremental data of the source database in real time, performing transformation processing on the incremental data, and finally loading the incremental data into a target database in a quasi-realtime manner are realized.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a system and method for incremental data extraction, conversion and distribution. Background technique [0002] With the in-depth application of big data in human resources and social security services, business systems are becoming larger and larger in terms of data flow, data processing volume, and data storage volume. Whether the business system is stable or not, and whether the system protection and data protection are sound or not have become key issues affecting the normal development of human resources and social security business systems. The security of existing system data cannot be guaranteed, and the extraction speed is low, which cannot meet the needs of big data analysis business. [0003] In actual business, there are big differences between the production database as the source database and the synchronization database as the target database in terms of databas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 徐铭孔兰菊肖宗水王振坤
Owner DAREWAY SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap