Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Massive multi-source heterogeneous data ETL method and system supporting interface adaptation

A multi-source heterogeneous data and interface adaptation technology, applied in the field of ETL management, can solve problems such as high business dependence, large manpower and time, and inability to meet the needs of ETL operations

Inactive Publication Date: 2018-11-20
DAREWAY SOFTWARE
View PDF5 Cites 109 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the big data environment, the data presents the characteristics of large capacity, multiple styles, and frequent interactions. With the continuous increase of collected data, the data processing logic is gradually complicated, and it is faced with the transmission efficiency of massive multi-source heterogeneous data between different databases. question
[0003] Traditional ETL tools are expensive, highly dependent on specific business, and have a centralized architecture, that is, design, operation and management are all concentrated on one server, and the requirements for hardware are very high
In the traditional ETL management mode, generally according to the attributes of the source database and the target database, manually determine the ETL tool, and set the ETL task process, set parameters, and start the task. This manual ETL management mode has a complicated process and consumes a lot of manpower and time. And it cannot meet the ETL operation requirements of massive multi-source heterogeneous data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Massive multi-source heterogeneous data ETL method and system supporting interface adaptation
  • Massive multi-source heterogeneous data ETL method and system supporting interface adaptation
  • Massive multi-source heterogeneous data ETL method and system supporting interface adaptation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0088] It should be pointed out that the following detailed description is exemplary and intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0089]It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and / or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and / or combinations thereof.

[0090] refer to figure 1 , is a flow chart of the massive multi-source heterogeneous data ETL method of the pres...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a massive multi-source heterogeneous data ETL method and system supporting interface adaptation. The method comprises a data extraction step of setting basic information of data sources and a target database, adaptively matching corresponding ETL tools for different data sources and performing parameter setting on the ETL tools, a data conversion step of finishing ETL operation control execution and scheduling management, performing buffer storage and management on extracted data and finishing processing of data cleaning and conversion and the like, a data loading stepof carrying out quality inspection on converted data objects and updating and loading the data inspected to be correct into the target database according to table structure output defined by a data model, and a data monitoring step of performing monitoring management on an ETL operation execution process, an operation resource usage condition and a system operation condition. The proper ETL tool is adaptively matched; the extraction and conversion of massive data are achieved; and efficient execution and orderly management of ETL operation are realized.

Description

technical field [0001] The invention relates to the field of ETL management, in particular to an ETL method and system for massive multi-source heterogeneous data supporting interface adaptation. Background technique [0002] At present, the industry has accumulated a large amount of data, and the volume, type and change of data are increasing rapidly. However, big data has not been fully utilized, and the huge value contained in it has yet to be tapped. Big data often has multi-source heterogeneous characteristics and comes from different and dispersed business systems. There are structured data, semi-structured data, unstructured data and other types, which are difficult to extract and convert into required data. In the big data environment, the data presents the characteristics of large capacity, multiple styles, and frequent interactions. With the continuous increase of collected data, the data processing logic is gradually complicated, and it is faced with the transmiss...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 史玉良王新军张晖管永明吕梁刘智勇
Owner DAREWAY SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com