Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing method, node and ETL (extract transform and load) system

A processing node and data processing technology, applied in the information field, can solve problems such as inability to process massive data or processing massive data, I/O performance bottlenecks, etc.

Active Publication Date: 2012-09-26
HUAWEI TECH CO LTD
View PDF4 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the prior art, the front node is used to read and forward the data to each processing node, and then each processing node performs ETL processing on the data read by the front node. However, in this method, the front node is used as the entrance Nodes have I / O performance bottlenecks, cannot process massive data or have difficulties in processing massive data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, node and ETL (extract transform and load) system
  • Data processing method, node and ETL (extract transform and load) system
  • Data processing method, node and ETL (extract transform and load) system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0034] figure 1 A flowchart of an embodiment of the data processing method provided by the present invention, such as figure 1 As shown, the method includes:

[0035] S101. The front-end node obtains the file names and storage paths of multiple files corresponding to the data to be processed from the source data source;

[0036] S102. Th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data processing method, a node and an ETL (extract transform and load) system. The method comprises the steps of: a preposed node obtains file names and storage paths of multiple files corresponding to data to be processed from a source data source; the advanced node evenly distributes the file names and the storage paths of multiple files to each ETL processing node in a plurality of ETL processing nodes; and therefore each ETL processing node obtains data to be processed in a file corresponding to the received filename from the source data source according to the received storage path. According to the data processing method, the node and the ETL (extract transform and load) system, the capability of the ETL system on processing mass data is improved.

Description

Technical field [0001] The present invention relates to information technology, in particular to a data processing method, node, and extraction, conversion and loading ETL system. Background technique [0002] In information and communication systems, it is often necessary to pour data into file systems, text transfer protocols (File Transfer Protocol, FTP) or network file systems (Network File System, NFS), and each node in the network performs data extraction, loading and Transformation (Extract Load Transform, ETL) processing. [0003] In the prior art, the pre-node is used to read and forward the data to each processing node, and each processing node performs ETL processing on the data read by the pre-node. However, in this method, the pre-node is used as the entrance Nodes have I / O performance bottlenecks, unable to process massive amounts of data or have difficulties in processing massive amounts of data. Summary of the invention [0004] The embodiment of the present invent...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 邱恒威蒙安库
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com