Mapreduce data localization method based on dynamic marker priority value

Abstract

The present invention provides a kind of MapReduce data localization method based on dynamic label priority value, comprises the following steps: the localization data block initialization mark value on each calculation node is 2n, calculates the quantity of each calculation node localization data block; According to The number of data blocks of each computing node is sorted in ascending order; starting from the computing node with the least number of data blocks in turn, subtract a certain number DecS from the tag value of the subsequent backup data block of the data block on each computing node; The data block with the largest value is scheduled; after task scheduling, the number of data blocks of each computing node is adjusted again and sorted in ascending order. After sorting, the tag value is adjusted, and data processing is finally completed. The present invention improves the data localization degree of jobs by marking data blocks and setting different priority values, thereby improving the execution efficiency of jobs on MapReduce and reducing the bandwidth occupation rate of jobs.

Description

technical field [0001] The invention relates to computer large-scale data calculation, especially to the field of MapReduce calculation, in particular to a MapReduce data localization method based on a dynamic mark priority value. Background technique [0002] The MapReduce computing framework is the core component of the Hadoop platform. All computing tasks on the Hadoop platform are completed on MapReduce. Therefore, the computing efficiency and throughput of the Hadoop platform are closely related to the job execution efficiency and throughput of MapReduce. [0003] On the Hadoop platform, the MapReduce distributed computing framework is built on the distributed file system HDFS that stores data blocks in redundant form. HDFS stores user data in the form of data blocks and the default redundancy number of data blocks is 3, that is to say, one When the file size corresponding to the job is 100 data blocks (in HDFS, the default size of the data block is 64MB), the number of...

AI Technical Summary

Problems solved by technology

[0005] However, this data localization method cannot achieve a high degree of data localization. The main reasons are: 1) the data blocks of a job are relatively unevenly distributed on each computing node; 2) the computing performance of each computing node is different, Not exactly isomorphic machines

In the original scheduling method, MapReduce considers that the machines in the cluster are isomorphic, and does not consider the unbalanced distribution of data blocks; 3) Each computing node does not have a priority when selecting local data blocks

Data blocks have redundant backups, which may lead to vicious preemption of localization tasks