An optimal localization task scheduling method based on mapreduce

Abstract

The invention proposes a MapReduce task scheduling algorithm that can work simultaneously in homogeneous and heterogeneous cluster environments, belonging to the field of computer technology. The scheduling algorithm can comprehensively consider the processing performance of each computing node in the cluster, abstract the computing nodes and computing tasks into a bipartite graph, and form the final global task scheduling scheme by appropriately expanding the bipartite graph and combining the KM weighted optimal matching algorithm. Experimental data show that the scheduling algorithm can improve the data localization degree in the Map stage to nearly 100%, and the overall execution time of the MapReduce job can be reduced by 67.1%.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to an optimal localization task scheduling method based on MapReduce. Background technique [0002] MapReduce task scheduling directly affects the execution time of MapReduce computing jobs, and an efficient scheduling algorithm can effectively improve job execution efficiency. [0003] The degree of data localization directly affects the execution efficiency of MapReduce jobs. The MapReduce job is mainly composed of the Map stage and the Reduce stage. The intermediate output data generated by the computing nodes in the Map stage needs to be transmitted through the network to the computing nodes in the Reduce stage as their input data. This intermediate stage is called Shuffle. The resource consumption of network bandwidth brought about by the data transmission in the Shuffle stage and the persistent storage of data in the Reduce stage is inevitable. Under the conditi...

AI Technical Summary

Problems solved by technology

[0004] There are various scheduling methods to improve the degree of data localization in the Map stage, but there are some problems such as low practicability and limited scope of application.

Zaharia et al. proposed a delay scheduling algorithm that can effectively improve the degree of data localization (“Delay scheduling: a simple technique for achieving locality and fairness including cluster scheduling,” in Proceedings of the 5th European conference on Computersystems.ACM,2010,pp. 265–278.), but this method of delayed scheduling is based on the loss of execution efficiency of local jobs, and this scheduling algorithm is not widely applicable, when only one or a few jobs are running, it cannot Achieve optimal data localization and overall job execution time

Xie et al proposed a method to distribute data in advance according to the performance of computing nodes (“Improving mapreduce performance through data placement intensive hadoop clusters,” in Parallel & Distributed Processing, Workshops and PhD Forum (IPDPSW), 2010IEEE International Symposium on.IEEE, 2010, pp .1–9.), this method needs to measure the performance of each computing node in advance, and this method is not very practical under the MapReduce platform where computing resources of computing nodes can be dynamically set by adjusting parameters

Images