Storm cluster load balancing method and system based on discrete particle swarms

Abstract

The invention discloses a Storm cluster load balancing method and system based on discrete particle swarms. The Storm cluster load balancing method comprises the steps of obtaining the number s of working nodes and the number t of tasks to be distributed; initializing a particle swarm; obtaining the Pbest of each primary particle and the Gbest of the particle swarm; updating each task allocation method; updating the Pbest of each particle and the Gbest of the particle swarm after iterative updating; obtaining a global historical optimal task allocation method Gbest until the number of iterations reaches a preset maximum number of iterations; running a Storm cluster according to a global historical optimal task allocation method Gbest; according to the method, the strategy of Storm clusterscheduling by adopting the particle swarm algorithm can improve the performance utilization rate of the Storm cluster, and the situation that the CPU of a certain working node is fully loaded and theother working node is unloaded is avoided.

Description

technical field [0001] The invention relates to the field of big data real-time processing, in particular to a Storm cluster load balancing method and system based on discrete particle swarms. Background technique [0002] After the client submits the Topology task, Nimbus, the control node of the Storm cluster, receives the Topology task. Based on the received Topology task, Nimbus calculates the number of runtime instance tasks that need to be processed, and uses polling according to the number of worker nodes contained in the current Storm cluster. Scheduling (Round Robin scheduling) algorithm, which evenly distributes the runtime instance tasks contained in the topology submitted by the user to each working node worker. During the allocation process, it will always ask from the first working node, so that the first The load pressure of the working node is getting bigger and bigger, which makes the load pressure of the working node much larger than other nodes, resulting ...

AI Technical Summary

Problems solved by technology

[0002] After the client submits the Topology task, Nimbus, the control node of the Storm cluster, receives the Topology task. Based on the received Topology task, Nimbus calculates the number of runtime instance tasks that need to be processed, and uses polling according to the number of worker nodes contained in the current Storm cluster. Scheduling (Round Robin scheduling) algorithm, which evenly distributes the runtime instance tasks contained in the topology submitted by the user to each working node worker. During the allocation process, it will always ask from the first working node, so that the first The load pressure of the working node is getting bigger and bigger, which makes the load pressure of the working node much larger than other nodes, resulting in different loads of each working node worker in the Storm cluster, and the round-robin scheduling algorithm does not consider the different nodes. The difference in performance leads to low utilization of node resources. In addition, the default scheduling pays more attention to CPU resources for node resources, while ignoring other types of resources such as memory usage, computer bandwidth usage, and performance perception rate. Insufficient memory, network congestion and other problems

When the load of the Storm cluster cannot be balanced, the working status of different working nodes of the computer will be different within a certain time range, and the performance of the Storm cluster cannot be fully utilized, and the service life of the working nodes under high pressure for a long time will be reduced. prone to failure

The performance utilization rate of the working nodes under low pressure is low, which has a great impact on the efficiency of the Storm cluster

Images