Storm cluster load balancing method and system based on discrete particle swarms

Abstract

The invention discloses a Storm cluster load balancing method and system based on discrete particle swarms. The Storm cluster load balancing method comprises the steps of obtaining the number s of working nodes and the number t of tasks to be distributed; initializing a particle swarm; obtaining the Pbest of each primary particle and the Gbest of the particle swarm; updating each task allocation method; updating the Pbest of each particle and the Gbest of the particle swarm after iterative updating; obtaining a global historical optimal task allocation method Gbest until the number of iterations reaches a preset maximum number of iterations; running a Storm cluster according to a global historical optimal task allocation method Gbest; according to the method, the strategy of Storm clusterscheduling by adopting the particle swarm algorithm can improve the performance utilization rate of the Storm cluster, and the situation that the CPU of a certain working node is fully loaded and theother working node is unloaded is avoided.

Description

technical field [0001] The invention relates to the field of big data real-time processing, in particular to a Storm cluster load balancing method and system based on discrete particle swarms. Background technique [0002] After the client submits the Topology task, Nimbus, the control node of the Storm cluster, receives the Topology task. Based on the received Topology task, Nimbus calculates the number of runtime instance tasks that need to be processed, and uses polling according to the number of worker nodes contained in the current Storm cluster. Scheduling (Round Robin scheduling) algorithm, which evenly distributes the runtime instance tasks contained in the topology submitted by the user to each working node worker. During the allocation process, it will always ask from the first working node, so that the first The load pressure of the working node is getting bigger and bigger, which makes the load pressure of the working node much larger than other nodes, resulting ...

AI Technical Summary

Problems solved by technology

[0002] After the client submits the Topology task, Nimbus, the control node of the Storm cluster, receives the Topology task. Based on the received Topology task, Nimbus calculates the number of runtime instance tasks that need to be processed, and uses polling according to the number of worker nodes contained in the current Storm cluster. Scheduling (Round Robin scheduling) algorithm, which evenly distributes the runtime instance tasks contained in the topology submitted by the user to each working node worker. During the allocation process, it will always ask from the first working node, so that the first The load pressure of the working node is getting bigger and bigger, which makes the load pressure of the working node much larger than other nodes, resulting in different loads of each working node worker in the Storm cluster, and the round-robin scheduling algor

Images