MapReduce based load self-adaptive task scheduling method

A task scheduling and self-adaptive technology, applied in multi-programming devices, resource allocation, etc., can solve problems such as poor system performance and low awareness of cluster resources, and achieve the effect of enhancing applicability

Active Publication Date: 2015-01-28
HUAZHONG UNIV OF SCI & TECH
View PDF5 Cites 124 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] Aiming at the defects of existing MapReduce task scheduling technology, the purpose of the present invention is to provide a task scheduling scheme based on cluster node computing capability evaluation system and load self-adaption, aiming to solve the problem of low awareness of cluster resources caused by existing task strategies, The problem of poor system performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • MapReduce based load self-adaptive task scheduling method
  • MapReduce based load self-adaptive task scheduling method
  • MapReduce based load self-adaptive task scheduling method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0054] The above description is only a summary of the technical solutions of the present invention. In order to have a clearer understanding of the technical means of the present invention and implement them in accordance with the content of the description, the above solutions are further described below with reference to the drawings and specific embodiments. It should be understood that these embodiments are used to illustrate the present invention and not to limit the scope of the present invention.

[0055] In order to clearly understand the present invention, the terms used in the present invention are explained below:

[0056] Heterogeneous cluster: In a cluster, there are performance differences between nodes due to different node hardware and software operating environments.

[0057] MapReduce: It is a software architecture proposed by Google for parallel computing of large-scale data sets (greater than 1TB). Reliability is achieved by distributing large-scale operations of ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a MapReduce based load self-adaptive task scheduling method, which comprises the following steps: (1) dynamically monitoring a Hadoop cluster load condition; (2) dynamically monitoring software information generated in the task execution process of each execution node in the cluster; (3) dynamically monitoring hardware information generated in the task execution process of each execution node in the cluster; (4) summarizing load monitoring information, software monitoring information and hardware monitoring information of each execution node in the cluster collected in the steps (1), (2) and (3), and modeling and computing the computing power of each execution node in the cluster; (5) executing a cluster load pre-warning function, and carrying out the intelligent task scheduling according to the computing power of each execution node in the cluster. The MapReduce based load self-adaptive task scheduling method provided by the invention solves the problems that the existing Hadoop scheduler has low perceptibility to cluster resources and unreasonable task allocation, and provides a load self-adaptive and more scientific and effective task scheduling scheme.

Description

Technical field [0001] The invention belongs to the field of distributed parallel computing, and specifically is a load adaptive task scheduling method based on MapReduce. Background technique [0002] With the advent of big data and the Internet era, data has exploded at geometric levels, which has brought great challenges to traditional distributed storage and computing systems. A more simplified distributed parallel computing model—Hadoop MapReduce has emerged. MapReduce is a distributed parallel programming system for processing massive data sets. Its framework is composed of a master node and multiple execution nodes. The master node usually divides the input data set into several independent Data block, that is to say, the job is divided into subtasks of fixed granularity, which are allocated to multiple execution nodes for concurrent execution to improve cluster throughput. Therefore, the task scheduling strategy of MapReduce directly affects the resource utilization of t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
Inventor 王芳冯丹杨静怡吴雪瑞
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap