Self-adaptive load balancing method for Reduce ends in parallel computing framework

Abstract

The invention relates to a self-adaptive load balancing method for Reduce ends in a parallel computing framework. The distribution of data input by tasks is predicted through adopting a dynamic hash function division method; and a static hash function is produced according to the distribution characteristics of the predicted data, so that under the effect of the static hash function, all data can be evenly distributed into corresponding computing nodes as much as possible. Therefore, during task scheduling, the allocation of the data and computing resources can be dynamically and self-adaptively adjusted according to the data distribution condition, the deflection occurring during the computation is reduced and the efficiency is improved.

Description

technical field [0001] The invention belongs to the field of information technology, and relates to a method for balancing loads of distributed nodes in a distributed computing environment, in particular to a method for self-adaptive load balancing of Reduce terminals in a parallel computing framework. Background technique [0002] With the growth of data volume and the increase in demand for data processing capabilities, traditional parallel computing can no longer cope with distributed computing under large data volumes. [0003] At present, the Map-Reduce computing framework can better solve the problem of task allocation and scheduling of distributed computing under the large amount of data through the random allocation of data and tasks and the parallel utilization of hardware resources. However, because the task allocation of Map-Reduce depends on Static hash function settings and the number of parallel calculations make the distributed calculations not uniform enough,...

AI Technical Summary

Problems solved by technology

[0018] It is precisely because of this simple division method that when the number of records corresponding to different keys is unevenly distributed or the size of the value corresponding to the key is also different, some of the data obtained by Reduce is too large, and some of the obtained data is too large. small, resulting in a skewed

In addition, since the amount of data output by Map is different from the amount of data input by Map in real tasks, users cannot estimate the amount of data to be processed by Reduce and set an appropriate number of Reduce, and cannot automatically adapt to data with different distributions.

Images