Optimal localized task scheduling method based on MapReduce

Abstract

The invention provides a MapReduce task scheduling algorithm capable of working in an isomorphic cluster environment and a heterogeneous cluster environment at the same time and belongs to the technical field of computers. According to the scheduling algorithm, the processing performance of each computational node in a cluster can be comprehensively considered, computational nodes and computational tasks are abstracted into a bipartite graph, and through proper expansion of the bipartite graph and combination with a KM weighted optimal matching algorithm, a final overall task scheduling scheme is formed. Experiment data show that the localization degree of data in the Map stage can be increased to almost 100% and overall execution time of MapReduce operation can be optimally reduced by 67.1%.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to an optimal localization task scheduling method based on MapReduce. Background technique [0002] MapReduce task scheduling directly affects the execution time of MapReduce computing jobs, and an efficient scheduling algorithm can effectively improve job execution efficiency. [0003] The degree of data localization directly affects the execution efficiency of MapReduce jobs. The MapReduce job is mainly composed of the Map stage and the Reduce stage. The intermediate output data generated by the computing nodes in the Map stage needs to be transmitted through the network to the computing nodes in the Reduce stage as their input data. This intermediate stage is called Shuffle. The resource consumption of network bandwidth brought about by the data transmission in the Shuffle stage and the persistent storage of data in the Reduce stage is inevitable. Under the conditi...

AI Technical Summary

Problems solved by technology

[0004] There are various scheduling methods to improve the degree of data localization in the Map stage, but there are some problems such as low practicability and limited scope of application.

Zaharia et al. proposed a delay scheduling algorithm that can effectively improve data localization ("Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," in Proceedings of the 5th European conference on Computer systems. ACM, 2010, pp.265–278.), but this delayed scheduling method is based on the loss of execution efficiency of local jobs, and this scheduling algorithm is not widely applicable, when only one or a few jobs are running, Can not achieve optimal data localization and job overall execution time

Xie et al proposed a method to distribute data in advance according to the performance of computing nodes (“Improving mapreduce performance through data placement in heterogeneous hadoop clusters,” in Parallel & Distributed Processing, Workshops and PhD Forum (IPDPSW), 2010IEEE International Symposium on.IEEE , 2010, pp.1–9.), this method needs to measure the performance of each computing node in advance, and this method is not very practical under the MapReduce platform where the computing resources of computing nodes can be dynamically set by adjusting parameters

Images