Data processing method, node and ETL (extract transform and load) system

A technology for processing nodes and data processing. It is applied in the information field and can solve problems such as I/O performance bottlenecks, inability to process massive data or processing massive data.

Active Publication Date: 2015-03-11
HUAWEI TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the prior art, the front node is used to read and forward the data to each processing node, and then each processing node performs ETL processing on the data read by the front node. However, in this method, the front node is used as the entrance Nodes have I / O performance bottlenecks, cannot process massive data or have difficulties in processing massive data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, node and ETL (extract transform and load) system
  • Data processing method, node and ETL (extract transform and load) system
  • Data processing method, node and ETL (extract transform and load) system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0034] figure 1 A flowchart of an embodiment of the data processing method provided by the present invention, such as figure 1 As shown, the method includes:

[0035] S101. The front node obtains the file names and storage paths of multiple files corresponding to the data to be processed from the source data source;

[0036] S102. The fron...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a data processing method, a node and an ETL (extract transform and load) system. The method comprises the steps of: a preposed node obtains file names and storage paths of multiple files corresponding to data to be processed from a source data source; the advanced node evenly distributes the file names and the storage paths of multiple files to each ETL processing node in a plurality of ETL processing nodes; and therefore each ETL processing node obtains data to be processed in a file corresponding to the received filename from the source data source according to the received storage path. According to the data processing method, the node and the ETL (extract transform and load) system, the capability of the ETL system on processing mass data is improved.

Description

technical field [0001] The invention relates to information technology, in particular to a data processing method, a node and an extraction, conversion and loading ETL system. Background technique [0002] In information and communication systems, it is often necessary to dump data into the file system, text transfer protocol (File Transfer Protocol, FTP) or network file system (Network File System, NFS), and each node in the network performs data extraction, loading and processing. Conversion (Extract Load Transform, ETL) processing. [0003] In the prior art, the front node is used to read and forward the data to each processing node, and then each processing node performs ETL processing on the data read by the front node. However, in this method, the front node is used as the entrance Nodes have I / O performance bottlenecks, cannot process massive data or have difficulties in processing massive data. Contents of the invention [0004] The embodiment of the present inve...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 邱恒威蒙安库
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap