Hadoop-oriented dynamic scheduling method

Abstract

The invention provides a hadoop-oriented dynamic scheduling method. The hadoop-oriented dynamic scheduling method includes the following steps that (1) computing power of all nodes is consistent; execution time of each operation is linearly reduced with increasing of execution times; (2) whether the operations are similar operations or not is determined, and execution conditions of the similar operations are obtained and statistically calculated; (3) expected execution time which is designed during submit of each operation is analyzed; (4) if the operations are determined to be the similar operations, a cost model is established to obtain operation weight value; (5) if the operations are not the similar operations, a minimum weight value is assigned; (6) operation actual resource quantity is adjusted according to the operation weight value, and responsivity of real-time operations is improved. By means of the hadoop-oriented dynamic scheduling method, defects of real-time operation scheduling by existing scheduling algorithms can be effectively overcome, resource control can be performed on the real-time operations, the real-time operation efficiency is increased, and thereby, a user can finely control the operation execution speed.

Description

technical field [0001] The invention relates to the application field of electronic technology, in particular to a Hadoop-oriented dynamic scheduling method which improves the Fair Scheduler algorithm of the Hadoop platform. Background technique [0002] With the rapid development of network information technology, billions of requests per day have resulted in PB-level database storage. As an open source large-scale data processing platform, Hadoop has become the best choice for cloud computing. Hadoop is a software framework capable of distributed processing of large amounts of data, because it can be deployed in heterogeneous cluster infrastructure and common hardware, making cloud computing easy to expand. Hadoop is currently the mainstream cloud computing solution, which solves the analysis problem of large data sets by implementing two simple functions. [0003] In the Hadoop framework, job scheduling is the key to job execution efficiency and user response. Existing ...

AI Technical Summary

Problems solved by technology

[0004] For the above-mentioned problems in the existing scheduling algorithm, the purpose of the present invention is to provide a dynamic scheduling method for Hadoop, to solve the problem that the high real-time job cannot be responded to in the prior art, the job execution speed is difficult to control, and the Fair Scheduler is based on the lack of resources. The preemption method makes the resources occupied by large jobs huge, resulting in the delay in obtaining resources for small jobs, which makes it difficult for small jobs to be scheduled in time

Images