Unlock instant, AI-driven research and patent intelligence for your innovation.

A resource scheduling method and device for hadoop clusters

A Hadoop cluster and resource scheduling technology, applied in the direction of resource allocation, multiprogramming device, program control design, etc., can solve the problem that the resource scheduling method of Hadoop cluster cannot be scheduled.

Active Publication Date: 2020-06-30
BEIJING SOHU NEW MEDIA INFORMATION TECH
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In order to solve the problem that the resource scheduling method of the traditional Hadoop cluster cannot be scheduled according to the characteristics of the nodes in the cluster, the application provides a resource scheduling method and device for the Hadoop cluster, and its technical solution is as follows:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A resource scheduling method and device for hadoop clusters
  • A resource scheduling method and device for hadoop clusters
  • A resource scheduling method and device for hadoop clusters

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0063] In the Hadoop cluster provided by the embodiment of the present application, the node type is set for the node according to the physical configuration of each node in the cluster, and the task queue is classified. Nodes are classified according to the hardware configuration of the nodes in different dimensions, for example, whether a g...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a resource scheduling method of a Hadoop cluster. According to the hardware configuration difference situation of a node in the Hadoop cluster, a node type is preset for the node in the Hadoop cluster, and a queue type is set for a task queue in the cluster. When a to-be-scheduled task sent by a client side, obtaining appointed execution information from the to-be-scheduled task, wherein the appointed execution information comprises the node type and at least one item in the queue type; and then, from a current callable node in the Hadoop cluster, obtaining a target node of which the node type is matched with the appointed execution information, and distributing the to-be-scheduled task to the target node to be executed. By use of the method, the target node matched with the appointed execution information is called from the current callable node to execute the to-be-scheduled task, the node is reasonably scheduled according to the difference situation of the physical configuration of the node, a node resource use ratio is improved, and the task scheduling performance of the Hadoop cluster is optimized.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a resource scheduling method and device for a Hadoop cluster. Background technique [0002] Hadoop cluster is an open source distributed storage and big data processing framework, mainly including Common, Distributed File System (Hadoop Distributed File System, HDFS), Hadoop Resource Manager (Yet AnotherResource Negotiator, YARN) and MapReduce MapReduce (for MapReduce parallel computing on large datasets). Heterogeneous Hadoop clusters usually refer to differences in the hardware (CPU, memory, disk) configurations of the nodes that make up the cluster. [0003] The YARN resource scheduler is responsible for the unified management and allocation of all resources in the Hadoop cluster. It receives resource report information from each node and allocates the information to each task according to a certain strategy. The current YARN resource scheduler thinks that the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50
CPCG06F9/4843G06F9/5083
Inventor 李昀晖王蕾
Owner BEIJING SOHU NEW MEDIA INFORMATION TECH