A hybrid big data task asynchronous submission method and system

A big data and hybrid technology, applied in the direction of electronic digital data processing, program synchronization, transaction processing, etc., can solve problems such as troublesome, inconsistent commands, and inability to monitor different types of tasks in real time, so as to improve efficiency Effect

Active Publication Date: 2019-04-26
杭州玳数科技有限公司
View PDF7 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] At present, there are many ways to submit tasks for big data frameworks, but tasks submitted for big data frameworks such as spark, flink, hadoopMR, and storm are all submitted manually through the command line, and the commands submitted by different frameworks are various and inconsistent This is more troublesome, and does not support batch submission. It is inefficient to submit multiple tasks, and it is impossible to achieve real-time monitoring and unified management of different types of submitted tasks.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A hybrid big data task asynchronous submission method and system
  • A hybrid big data task asynchronous submission method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] The present invention involves two parts of the main body, namely DTEngineMonitor component (management and control node) and DTEngineWork component (work node), wherein, DTEngineMonitor component (management and control node) can be at least one, used to start HttpServer to receive the task that comes from the outside, to DTEngineWork (work node) achieve real-time monitoring, remove unavailable nodes, and distribute received...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a hybrid big data task asynchronous submission method and system, and the method comprises the steps that at least one working node sends heartbeat information data to a Zookeeper according to a preset time interval, obtains the load information of the working node, and sends the load information to the Zookeeper; The master control node in the at least one control node obtains heartbeat information data of each working node from the Zookeeper, and available working nodes are determined; Any one management and control node in the at least one management and control nodereceives an externally submitted task; obtaining load information of all available working nodes from the Zookeeper, obtaining working node addresses selected from all the available working nodes according to the load information and a preset scheduling algorithm, and sending a task to the selected working nodes; And the selected working node reads the tasks in the task priority queue, generates different cluster clients according to task attributes, and submits the tasks by using the different cluster clients.

Description

technical field [0001] The invention relates to the field of big data technology data processing, in particular to a method and system for asynchronously submitting hybrid big data tasks. Background technique [0002] At present, there are many ways to submit tasks for big data frameworks, but tasks submitted for big data frameworks such as spark, flink, hadoopMR, and storm are all submitted manually through the command line, and the commands submitted by different frameworks are various and inconsistent This is cumbersome, and it does not support batch submission. It is inefficient to submit multiple tasks, and it cannot achieve real-time monitoring and unified management of different types of submitted tasks. Contents of the invention [0003] The present invention aims to provide a method and system for asynchronously submitting hybrid big data tasks that overcome one of the above problems or at least partially solve any of the above problems. [0004] In order to achi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/46G06F9/52
CPCG06F9/466G06F9/526
Inventor 杨思枢
Owner 杭州玳数科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap