Visualized big data workflow configuration method and system based on WEB

A workflow and big data technology, applied in the field of big data processing, can solve problems such as long startup time, achieve the effects of reducing personnel costs, improving system performance, and improving scheduling

Inactive Publication Date: 2018-05-15
北京博宇通达科技有限公司
View PDF3 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

All configuration data and program files of the open source workflow engine are stored on HDFS. When the task

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Visualized big data workflow configuration method and system based on WEB
  • Visualized big data workflow configuration method and system based on WEB
  • Visualized big data workflow configuration method and system based on WEB

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0059] Example one

[0060] figure 2 Shows the flow chart of the WEB-based visualization big data workflow configuration method of this embodiment, refer to figure 1 , The method may include:

[0061] Step 11. The interface layer sets a visual process design component, and receives the big data processing business process designed by the process design component and the corresponding data processing node.

[0062] In this embodiment, the interface layer mainly provides a visual interface that is convenient for users to use, and provides corresponding design components. These design components can be freely pushed and combined according to user needs and operations. The big data processing business processes are formed according to the user's combination and design, and these business processes are corresponding to the corresponding data processing nodes. That is, the user achieves the purpose of controlling and combining data processing nodes by operating the design component, and ...

Example Embodiment

[0079] Example two

[0080] Such as image 3 As shown, a web-based visual big data workflow configuration system is disclosed, in which,

[0081] The interface layer 21 is used to set a visual process design component, receive the big data processing business process designed by the design component and the corresponding data processing node; display the execution result of the big data processing business process;

[0082] The engine layer 22 is used to schedule the data processing nodes in sequence according to the big data processing business process; package the data processing nodes into a distributed program; receive and organize the execution results of the distributed program; The execution result is returned to the interface layer 21;

[0083] The execution layer 23 is used to execute the distributed program in the distributed program running environment and return the execution result to the engine layer 22.

[0084] Further, the engine layer 22 includes a workflow engine an...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a visualized big data workflow configuration method and system based on the WEB. Workflow configuration is achieved through a browser by the drag method and an wizard interface, the development efficiency is improved, and the personnel cost is reduced. Meanwhile, complete MapReduce/Hive/Spark support is provided, the current mainstream big-data processing programs refer toHiveSQL, MapReduce, Spark and SparkSQL, and the method attempts to provide thorough support to the big-data processing programs. Moreover, according to the scheme, the running speed can be improved, an lightweight distributed workflow engine is achieved, the scheduling speed and running speed of the distributed programs are improved, and performance of the system is improved.

Description

technical field [0001] The present disclosure relates to the technical field of big data processing, in particular to a WEB-based visualized big data workflow configuration method and system. Background technique [0002] In the big data environment, the data sources are very rich and the data types are diverse. The amount of data stored and analyzed and mined is huge, and the requirements for data display are high, and the efficiency and availability of data processing are very important. [0003] Traditional data collection has a single source, and the amount of data stored, managed and analyzed is relatively small, and most of them can be processed by relational databases and parallel data warehouses. In terms of relying on parallel computing to improve data processing speed, traditional parallel database technology pursues high consistency and fault tolerance. According to the CAP theory, it is difficult to guarantee its availability and scalability. [0004] The tradit...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F8/34G06F9/445G06Q10/10
CPCG06F8/34G06F9/44505G06Q10/103
Inventor 张祺君黄超曹正凤
Owner 北京博宇通达科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap