A hadoop load balancing task scheduling method based on hybrid metaheuristic algorithm

Abstract

The invention relates to a Hadoop load balancing task scheduling method based on a hybrid meta-heuristic algorithm. A resource slot pressure model is established. The model aims to make the calculation pressure of all Slave nodes in the cluster at the same level. The hybrid meta-heuristic algorithm of particle swarm optimization is used to solve the optimal task scheduling scheme, and realize the load balancing task scheduling in the Hadoop cluster environment. Further realize the parallel programming of the algorithm through the high-performance and widely portable message passing interface MPICH (MPI over CHameleon), transfer the calculation process of the heuristic optimization algorithm to an additional computing node, and solve the problem at the same time through multiple groups to reduce the burden on the Master node. The calculation pressure is improved, and the ability to solve the optimal task scheduling scheme per unit time is improved. The invention can overall allocate the computing resources of the Hadoop cluster, balance the loads of the nodes of the cluster, avoid the waste of computing resources of the nodes, and maximize the profit of the equipment input in the data center.

Description

technical field [0001] The invention relates to the field of task scheduling under the Hadoop MapReduce structure. More specifically, using particle swarm optimization and a hybrid meta-heuristic algorithm based on simulated annealing and particle swarm optimization, as well as the MPICH parallel programming method, a Hadoop task scheduling algorithm targeting cluster load balancing. Background technique [0002] With the rapid development of mobile smart devices, the development of the information age has become more and more rapid. At the same time, with the use of the network by users, a large amount of data is actively or passively generated. These data are passed through Traditional statistics or calculation methods are usually unable to dig out the value, but once the potential value behind these data can be tapped, it can bring huge benefits to enterprises and governments. Analyze the user’s product preferences and needs, and at the same time push the products on the...

AI Technical Summary

Problems solved by technology

However, since the number of nodes in the cluster is often very large, when using the heuristic algorithm to solve the problem, the solution process is very cumbersome and will cause a lot of extra overhead. In the traditional Hadoop task scheduling mode, the heuristic task scheduling will give the master node (Master node, responsible for job scheduling and distribution in Hadoop) creates a heavy burden, which affects the stability of the cluster. At the same time, heuristic algorithms such as particle swarms, ant colonies, and simulated annealing are prone to fall into local optimal problems. The performance during the scheduling process is also not stable

Images