Component big data processing method, system, and computer readable storage medium

A big data processing and componentization technology, applied in database management systems, structured data retrieval, etc., to solve efficiency bottlenecks and reduce technical difficulties

Pending Publication Date: 2019-02-26
ZTE CORP
View PDF3 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The main purpose of the present invention is to provide a componentized big data processing method, system and compute...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Component big data processing method, system, and computer readable storage medium
  • Component big data processing method, system, and computer readable storage medium
  • Component big data processing method, system, and computer readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0033] Such as figure 1 as shown, figure 1 It is a schematic diagram of the system structure of the hardware operating environment involved in the solution of the embodiment of the present invention.

[0034] The system of the embodiment of the present invention includes a terminal and a big data cluster. The terminal can be a PC, or a terminal device such as a smart phone, a tablet computer, or a portable computer; the big data cluster can be a big data spark cluster or a Hadoop cluster.

[0035] Such as figure 1 As shown, the system may include: a processor 1001 , such as a CPU, a terminal 1004 , a big data cluster 1003 , a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The memory 1005 can be a high-speed ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a component-based large data processing method, a system and a computer-readable storage medium. The method comprises the following steps: receiving an orchestration command ofa user and generating an ETL job according to the orchestration command; Submitting the ETL job to a large data cluster; An execution command is sent to the large data cluster for running the ETL jobin a distributed engine of the large data cluster. Through the invention, the user can complete the arrangement of ETL operations through simple operation, greatly reducing technical difficulty, solving the problem that the prior non-component big data ETL technology requires the user to hard code to realize calculation logic, and the problem that the calculation logic cannot be reused.

Description

technical field [0001] The present invention relates to the technical field of ETL based on big data, in particular to a componentized big data processing method, system and computer-readable storage medium. Background technique [0002] With the rapid development of computer technology and the Internet, computer applications are becoming more and more popular, and their scale, scope and depth are also expanding. Especially in recent years, with the continuous maturity of big data technology, the development of big data analysis technology And widely used, the collection, cleaning and storage of massive data become the basis of its analysis. Big data ETL technology provides technical support to meet the current massive data collection, cleaning, and storage requirements. The current existing ETL technical solutions mainly have the following shortcomings: [0003] (1) Non-componentized, all ETL calculation logic needs to be customized by users, which is inconvenient to use....

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25
Inventor 胡德莲
Owner ZTE CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap