Dynamic label matching scheduling method under Hadoop Platform

A scheduling method and labeling technology, applied in resource allocation, program startup/switching, program control design, etc., can solve problems affecting job execution time, etc.

Active Publication Date: 2017-08-11
BEIJING UNIV OF TECH
View PDF2 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Imagine that if a machine learning task with a large amount of calculation is allocated to a machine node with poor CPU computing power, it will obviously affect the overall execution time of the job

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Dynamic label matching scheduling method under Hadoop Platform
  • Dynamic label matching scheduling method under Hadoop Platform
  • Dynamic label matching scheduling method under Hadoop Platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] In order to make the object, technical solution and features of the present invention clearer, the present invention will be further described in detail below in conjunction with specific implementation examples and with reference to the accompanying drawings. YARN scheduling framework such as figure 1 shown.

[0032] The individual steps are explained below:

[0033] (1) The user submits the application program to YARN, including the user program and starts the ApplicationMaster command.

[0034] (2) ResourceManager allocates the first Container for the application, and communicates with the corresponding NodeManager, asking it to start the ApplicationMaster of the application.

[0035] (3) ApplicationMaster registers with ResourceManager, applies for resources for each task, and monitors their running status until the end of the operation

[0036] (4) NodeManager performs self-test before sending heartbeat to generate dynamic node labels, and reports resources to R...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention discloses a dynamic label matching scheduling method under a Hadoop platform, and belongs to the field of computer software. Targeting at problems of large performance difference of Hadoop cluster nodes, randomness of resource allocation and too long execution time, the present invention provides a scheduler for dynamically matching a node performance label (hereinafter referred to as a node label) and a job category label (hereinafter referred to as a job label). A node performs initial classification and assigns a label to an original node; the node detects a performance indicator of the node to generate a dynamic node label; a job performs classification according to partial operation information to generate a job label; and a resource scheduler allocates a node resource to a job corresponding to the label. As an experimental result shows, the scheduler provided by the present invention shortens job execution time compared with the scheduler carried by YARN.

Description

technical field [0001] The invention belongs to the field of computer software, and relates to the design and realization of a dynamic label matching DLMS scheduling method based on Hadoop platform. Background technique [0002] Early Hadoop versions integrated resource scheduling management and the MapReduce framework into one module, resulting in poor code decoupling, poor scalability, and no support for multiple frameworks. The Hadoop open source community has designed and implemented a new-generation Hadoop system with a new architecture. The system is Hadoop2.0, which extracts resource scheduling and builds a new resource scheduling framework, that is, the new-generation Hadoop system YARN. It is well known that an appropriate scheduling algorithm in a certain environment can effectively improve the overall performance of the Hadoop operating platform and the resource utilization of the system while satisfying user job requests. There are three default schedulers in YA...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50H04L29/08
CPCG06F9/4881G06F9/5083H04L67/10
Inventor 毛韦竹翠
Owner BEIJING UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap