Supercharge Your Innovation With Domain-Expert AI Agents!

A method and system for distributing Hadoop cluster management tasks

A hadoop cluster and task distribution technology, which is applied in the distribution field of Hadoop cluster management tasks, can solve the problems of high mutual exclusion rate between stages, small number of stages, and low concurrent throughput of tasks, and achieve the effect of improving throughput and efficiency

Active Publication Date: 2018-05-25
BEIJING SOHU NEW MEDIA INFORMATION TECH
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, when using this criterion to filter, the mutual exclusion rate between stages is high, resulting in a small number of concurrent stages in each scheduling cycle, which in turn leads to low task concurrency and throughput

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for distributing Hadoop cluster management tasks
  • A method and system for distributing Hadoop cluster management tasks
  • A method and system for distributing Hadoop cluster management tasks

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0052] In order to make the object, technical solution and technical effect of the present invention more clear and complete, the specific implementation manner of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0053] Before introducing the embodiments of the present invention, technical terms that need to be used when describing the embodiments of the present invention are firstly introduced.

[0054] Because mainly relate to Hadoop cluster in the embodiment of the present invention, so the cluster management that the present invention mentions is exactly Hadoop cluster management, so-called Hadoop cluster management is to realize the management such as installation, start, stop, check and update of service and component in Hadoop cluster operate.

[0055] A cluster management task is a command issued to a relevant cluster node, that is, a computer, to complete a specific cluster management operation. For example, whe...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a device for distributing Hadoop cluster management tasks. The method first carries out stage planning to the management tasks according to the dependencies of the Hadoop components, then processes the management tasks in each stage in turn, and plans the management tasks assigned to the same component node in the same stage as a sub-stage; then when entering a After a scheduling cycle, scan all sub-phases currently to be scheduled and sort them. Finally, according to the preset filter conditions, it is judged whether the current sub-phase is suitable for task distribution in the current scheduling period according to the order of the sorted sub-phases from front to back. In this distribution method, a sub-phase is the smallest scheduling unit, and sub-phases within the same sub-phase and between sub-phases within the same parent phase can be executed in parallel. The invention can realize the parallel distribution of tasks on a finer granularity. Moreover, the method provided by the invention can improve the throughput of task distribution, and further improve the efficiency of Hadoop cluster management.

Description

technical field [0001] The invention relates to the technical field of computer clusters, in particular to a method and system for distributing Hadoop cluster management tasks. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Hadoop mainly includes core services such as HDFS, MapReduce2, YARN, and Hbase. Each service includes multiple service components. For example, HBase services include components such as HBaseMaster and Region Server. [0003] A Hadoop cluster refers to a group of computers deployed with Hadoop-related service components, and these computers provide external services through mutual cooperation between components. [0004] The Hadoop cluster includes core services such as HDFS, MapReduce2, YARN, and Hbase, and each service includes multiple service components. For example, the HBase service includes components such as HBaseMaster and Region Server. These components are discretely distributed...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50
Inventor 彭毅
Owner BEIJING SOHU NEW MEDIA INFORMATION TECH
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More