A load-adaptive task scheduling method based on mapreduce

A task scheduling and self-adaptive technology, applied in multi-programming devices, resource allocation and other directions, can solve the problems of poor system performance and low cluster resource awareness, and achieve the effect of enhancing applicability

Active Publication Date: 2018-06-12
HUAZHONG UNIV OF SCI & TECH
View PDF5 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] Aiming at the defects of existing MapReduce task scheduling technology, the purpose of the present invention is to provide a task scheduling scheme based on cluster node computing capability evaluation system and load self-adaption, aiming to solve the problem of low awareness of cluster resources caused by existing task strategies, The problem of poor system performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A load-adaptive task scheduling method based on mapreduce
  • A load-adaptive task scheduling method based on mapreduce
  • A load-adaptive task scheduling method based on mapreduce

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0054] The above description is only a generalization of the technical solution of the present invention. In order to understand the technical means of the present invention more clearly and implement it according to the contents of the specification, the above solution will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are used to illustrate the present invention and not to limit the scope of the present invention.

[0055] In order to clearly understand the present invention, the terms used in the present invention are explained below:

[0056] Heterogeneous cluster: In the cluster, due to the different hardware and software operating environments of the nodes, there are performance differences between nodes.

[0057] MapReduce: It is a software architecture proposed by Google for parallel computing of large-scale data sets (greater than 1TB). Reliability is achieved by distribut...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a load adaptive task scheduling method based on MapReduce, comprising: (1) dynamically monitoring the load status of the Hadoop cluster, (2) dynamically monitoring the software information generated by each execution node of the cluster during task execution, (3) ) Dynamically monitor the hardware information of each execution node of the cluster during the execution of the task, (4) summarize the load monitoring information, software monitoring information and Hardware monitoring information Three-party monitoring information, modeling and computing the computing power of each execution node of the cluster, (5) execute the cluster load warning function, and perform intelligent task scheduling according to the computing power of each execution node of the cluster. The invention solves the problems that the existing Hadoop scheduler has low awareness of cluster resources and unreasonable task allocation, and provides a load-adaptive, more scientific and effective task scheduling scheme.

Description

technical field [0001] The invention belongs to the field of distributed parallel computing, in particular to a load adaptive task scheduling method based on MapReduce. Background technique [0002] With the advent of the era of big data and the Internet, the geometric explosion of data has brought great challenges to traditional distributed storage and computing systems. A more simplified distributed parallel computing model—Hadoop MapReduce has emerged as the times require. MapReduce is a distributed parallel programming system for processing massive data sets. Its framework is composed of a master control node and multiple execution nodes. The master control node usually divides the input data set into several independent Data blocks, which divide jobs into subtasks of fixed granularity, are assigned to multiple execution nodes for concurrent execution to improve cluster throughput. Therefore, the task scheduling strategy of MapReduce directly affects the resource utiliz...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50
Inventor 王芳冯丹杨静怡吴雪瑞
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap