Lightweight ETL processing platform

A processing platform, lightweight technology, applied in the field of big data processing, can solve the problems of high professional requirements and slow processing speed of big data processing

Pending Publication Date: 2020-05-08
合肥科拉斯特网络科技有限公司
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Therefore, the embodiment of the present invention provides a lightweight ETL processing platform

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Lightweight ETL processing platform
  • Lightweight ETL processing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0022] This embodiment discloses a lightweight ETL processing platform, including a front-end task sending and receiving module and a back-end data processing module, the front-end task sending and receiving module performs input of operations and output display of processing results, and the back-end data processing module processes data , parameter configuration is separated from data calculation, making the ETL platform more lightweight.

[0023] The front-end task transceiver module includes a browser, an H5 operating unit and a MySQL database management system, the browser is used to configure operating parameters, the H5 operating unit visually displays the operation, and different support functions are placed in the MySQL database management system A function library or an operation instruction operated by Spark, the back-end data processing module includes a Spark cluster, and the data is stored in a distributed manner in the Spark cluster, and the large-scale operation...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a lightweight ETL processing platform. The lightweight ETL processing platform comprises a front-end task transceiving module and a rear-end data processing module. The front-end task transceiving module comprises a browser, an H5 operation unit and a MySQL database management system; the browser is used for configuring operation parameters. The H5 operation unit is used for visually displaying the operation; different function libraries or operation instructions supporting Spark operation are placed in the MySQL database management system. And the rear-end data processing module comprises a Spark cluster, distributed storage is performed on data in the Spark cluster, big data calculation is performed according to a function library or an operationinstruction of Spark operation placed in the MySQL database management system, and a calculation result is output to the MySQL database management system. The big data processing method and the device solve the problems that existing big data processing is high in professional degree requirement and low in processing speed.

Description

technical field [0001] The embodiment of the present invention relates to the technical field of big data processing, and in particular to a lightweight ETL processing platform. Background technique [0002] Cloud computing systems can provide massive storage and reliable services, so they are increasingly valued. In the cloud infrastructure, thousands of interconnected computers form a "cloud" that provides services. A large number of users can share this "cloud" at the same time, and tailor the required resources according to their actual needs. . As an important part of cloud data processing, most of the current cloud storage systems use distributed hash tables to build indexes, and data is organized into key-value pairs. Big data refers to a collection of data that is so large that it exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management, and analysis. It has massive data scale, fast data flow, diverse data types, ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/182G06F16/25G06F16/28
CPCG06F16/254G06F16/289G06F16/182
Inventor 李甫
Owner 合肥科拉斯特网络科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap