A way to achieve mapreduce data localization within a job

Abstract

The invention provides a method for realizing MapReduce data localization in an operation, and belongs to the technical field of computers. In the present invention, by changing the task scheduling algorithm in the job, the localization degree of the task data of the job can be further improved. For a specified job, when scheduling a computing task to a computing node, the method of the present invention comprehensively considers the number of remaining localized data blocks of the computing node on the job and the number of localized data blocks on the computing node calculated through a series of processes. The number of tasks that are expected to be processed in the future is used to allocate different tasks. The method provided by the present invention does not need to measure the computing performance of each computing node in advance, it is flexible and convenient to implement, and does not affect the execution efficiency of local jobs, and can reduce the network bandwidth occupation in the Map stage to the greatest extent, thereby improving the parallelism of cluster jobs. The overall execution time of the job is also significantly reduced.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to an optimization method for realizing the localization of MapReduce data in a job. Background technique [0002] With the development of the distributed computing model, after the MapReduce distributed computing model, many other distributed computing models have emerged, such as Spark and Storm. These models have their own emphasis on data processing, so in large Internet companies, these distributed models are all carried on a physical cluster at the same time. Although each distributed computing model can be isolated from each other, the network bandwidth resources of the entire physical cluster are shared. Therefore, improving the network bandwidth consumption of the MapReduce computing model and reducing the network bandwidth consumption of the MapReduce computing model are not only for the same physical cluster. Other computing models of the cluster are benefi...

AI Technical Summary

Problems solved by technology

[0004] 1. The network bandwidth consumption caused by users uploading data from local to HDFS;

[0005] 2. The network bandwidth consumption caused by transferring the output of the Map stage to the Reduce stage in the Shuffle stage;

[0006] 3. The network bandwidth consumption caused by storing the processing results in HDFS in the Reduce stage;

[0007] 4. Network bandwidth consumption caused by non-localized tasks

In addition, through experiments, it is found that most of the non-localization tasks appear after the start of the Shuffle phase, and the non-localization tasks that appear at this time will compete with the Shuffle phase for network bandwidth resources, thereby delaying the execution progress of the job itself

[0009] There are many scheduling strategies for improving the data localization degree in the Map stage, but there are some problems such as low practicability and limited scope of application.

Zaharia et al. proposed a delay scheduling algorithm that can effectively improve the degree of data localization (“Delay scheduling: a simple technique for achieving locality and fairness including cluster scheduling,” in Proceedings of the 5th European conference on Computersystems.ACM,2010,pp.265 –278.), but this method of delayed scheduling is based on the loss of execution efficiency of local jobs, and this scheduling algorithm is not widely applicable, and cannot be obtained when only one or a few jobs are running Optimal data localization and overall job execution time

Xie et al proposed a method to distribute data in advance according to the performance of computing nodes (“Improving mapreduce performance through data placement intensive hadoop clusters,” in Parallel & Distributed Processing, Workshops and PhD Forum (IPDPSW), 2010 IEEE International Symposium on.IEEE, 2010, pp .1–9.), this method needs to measure the performance of each computing node in advance, and this method is not very practical under the MapReduce platform where computing resources of computing nodes can be dynamically set by adjusting parameters

Images