Check patentability & draft patents in minutes with Patsnap Eureka AI!

A kind of etl data processing method and device

A data and source data technology, applied in the field of data integration, can solve problems such as low data processing efficiency and increased time-consuming data processing

Active Publication Date: 2021-08-20
HANGZHOU HIKVISION DIGITAL TECH
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Based on the above situation, in an ETL task, when storing a piece of data to the destination data warehouse, it is necessary to perform two ETL data processing processes, which increases the time consumption of the data processing process and reduces the data processing efficiency.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A kind of etl data processing method and device
  • A kind of etl data processing method and device
  • A kind of etl data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0063] The present invention will be described in detail below through specific examples.

[0064] refer to figure 1 , figure 1 A schematic flow chart of an ETL data processing method provided by an embodiment of the present invention, the method includes:

[0065] S101: Obtain configuration parameters of different types of source data;

[0066] Among them, the configuration parameters include: extraction parameters, loading parameters and ETL processing mode pa...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the present invention discloses an ETL data processing method and device. The method includes: acquiring configuration parameters of different types of source data; the configuration parameters include: extraction parameters, loading parameters and ETL processing mode parameters of each type of source data ; The extraction parameters at least include: source data warehouse information; the loading parameters include at least: destination data warehouse information; parse all configuration parameters, create an ETL task that includes multiple ETL processing operations; each of the ETL processing operations is used for For one type of source data, the data to be processed is extracted from the source data warehouse according to the extraction parameters, and the data to be processed is loaded to the destination data warehouse corresponding to the loading parameters according to the loading parameters and the ETL processing mode parameters; each step in the ETL task is executed An ETL processing operation. By applying the embodiment of the present invention, the data processing efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of data integration, in particular to an ETL data processing method and device. Background technique [0002] Information is an important resource of modern enterprises and the basis for enterprises to use scientific management and decision-making analysis. Enterprises use ETL (Extract-Transform-Load, extract, transform, load) technical means to extract heterogeneous source data from different data sources, clean, transform, and load the extracted source data, and store the processed data into the primary destination data warehouse. [0003] At present, in order to put data into the first-purpose data warehouse, generally, various source data of different types are converted into CSV (Comma Separated Value, comma-separated value) format files, and ETL data processing is uniformly performed on the CSV format files. Here, converting different types of source data into CSV format files is actually an ETL proc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25
CPCG06F16/254
Inventor 刘金柱
Owner HANGZHOU HIKVISION DIGITAL TECH
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com