Hadoop scheduling method and system and management node

A technology for managing nodes and scheduling methods, applied in the field of cloud computing, can solve problems such as inability to do it, insufficient single-machine resources, and inability to make full use of cluster resources, and achieve the effect of improving single-machine concurrency and resource utilization.

Inactive Publication Date: 2013-08-14
BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF3 Cites 30 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] (1) The number of slots accommodated by each machine is fixed, and the resources corresponding to each slot are also fixed. Hadoop defaults that each slot corresponds to 800MB of memory, and a Task that only requires 100MB of memory during actual operation, in From the perspective of JobTracker and TaskTracker, it still occupies a slot and still consumes 800MB of memory;
[0006] (2) The number of slots occupied by a specific task is completely converted according to the configuration of the su

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop scheduling method and system and management node
  • Hadoop scheduling method and system and management node
  • Hadoop scheduling method and system and management node

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0040] In describing the present invention, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", The orientation or positional relationship indicated by "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than Nothing indicating or implying that a referenced device or elem...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Hadoop scheduling method. The method comprises that a management node obtains resource consumption information of completed tasks in a plurality of computational nodes; the management node generates resource scheduling values according to the resource consumption information of completed tasks in the plurality of computational nodes; the management node receives an assignment request of new tasks and assigns resources for new tasks according to resource scheduling values. According to the Hadoop scheduling method, the stand-alone concurrency of Hadoop computational nodes (TaskTracker) can be improved, so that the resource utilization ratio of the whole cluster (the plurality of computational nodes) can be improved. The invention also discloses a Hadoop scheduling system and the management node.

Description

technical field [0001] The invention relates to the technical field of cloud computing, in particular to a Hadoop scheduling method, system and management node. Background technique [0002] Apache Hadoop is a software platform capable of distributed processing of large amounts of data. There are more and more massive data services, and the use of Hadoop is becoming more and more extensive. With the increasing scale of a single cluster (the first-generation Hadoop cluster can support about 4,000 machines), how to improve the utilization of cluster resources has gradually become a topic of concern. The key to improving cluster resource utilization is cluster scheduling. [0003] At present, Hadoop supports a variety of schedulers. Basically, the TaskTracker is allocated a fixed number of slots (slots) according to the machine configuration information, such as 16, which means that a single TaskTracker machine can execute up to 16 Tasks at the same time. The number of bits i...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
Inventor 孙垚光黎樵
Owner BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap