Hadoop-oriented dynamic scheduling method

A dynamic scheduling and job technology, applied in the field of electronic technology, can solve the problems of large resources occupied by large jobs, unresponsive high real-time jobs, difficult to control job execution speed, etc., to achieve the effect of ensuring normal operation

Active Publication Date: 2015-01-21
NANTONG UNIVERSITY
View PDF6 Cites 25 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] For the above-mentioned problems in the existing scheduling algorithm, the purpose of the present invention is to provide a dynamic scheduling method for Hadoop, to solve the problem that the high real-time job cannot be responded to in the prior art, the job execution spee

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop-oriented dynamic scheduling method
  • Hadoop-oriented dynamic scheduling method
  • Hadoop-oriented dynamic scheduling method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] The present invention will be described in further detail below in conjunction with the accompanying drawings.

[0049]A Hadoop-oriented dynamic scheduling method in the present invention is described in detail by taking optimization based on the existing Fair Scheduler (fair scheduling) algorithm as an example, but is not limited thereto. The idea of ​​the whole method is: for the real-time job submitted by the user, build and read the historical execution data, generate a cost model to estimate the job execution time, and calculate the actual number of jobs by analyzing the expected execution time of the job added when the user submits the job, so that Enables jobs to be computed to completion at a user-defined desired execution time. The present invention improves the widely used high-performance platform Hadoop scheduling algorithm, aims to solve the problem that the existing scheduling algorithm cannot respond in time to the interactive jobs submitted by users in r...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a hadoop-oriented dynamic scheduling method. The hadoop-oriented dynamic scheduling method includes the following steps that (1) computing power of all nodes is consistent; execution time of each operation is linearly reduced with increasing of execution times; (2) whether the operations are similar operations or not is determined, and execution conditions of the similar operations are obtained and statistically calculated; (3) expected execution time which is designed during submit of each operation is analyzed; (4) if the operations are determined to be the similar operations, a cost model is established to obtain operation weight value; (5) if the operations are not the similar operations, a minimum weight value is assigned; (6) operation actual resource quantity is adjusted according to the operation weight value, and responsivity of real-time operations is improved. By means of the hadoop-oriented dynamic scheduling method, defects of real-time operation scheduling by existing scheduling algorithms can be effectively overcome, resource control can be performed on the real-time operations, the real-time operation efficiency is increased, and thereby, a user can finely control the operation execution speed.

Description

technical field [0001] The invention relates to the application field of electronic technology, in particular to a Hadoop-oriented dynamic scheduling method which improves the Fair Scheduler algorithm of the Hadoop platform. Background technique [0002] With the rapid development of network information technology, billions of requests per day have resulted in PB-level database storage. As an open source large-scale data processing platform, Hadoop has become the best choice for cloud computing. Hadoop is a software framework capable of distributed processing of large amounts of data, because it can be deployed in heterogeneous cluster infrastructure and common hardware, making cloud computing easy to expand. Hadoop is currently the mainstream cloud computing solution, which solves the analysis problem of large data sets by implementing two simple functions. [0003] In the Hadoop framework, job scheduling is the key to job execution efficiency and user response. Existing ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/46
Inventor 施佺肖瑶施振佺马松王徐露刘德靖李冬冬
Owner NANTONG UNIVERSITY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap