A work-slot-aware storm platform job-sharing scheduling method

Abstract

The invention discloses a work groove perceptive Storm platform operation equipartition dispatching method. The method includes the steps that 1, a set of topologies needing task dispatching in a cluster and a set of idle and available working grooves in the cluster are obtained; 2, the working grooves of the same working node in the working groove set are divided into the same group; 3, the working grooves are taken out of the group with the most remaining working grooves currently and put into a groove perceiving sorting sequence, and the number of the working grooves in each group is updated until the working groove set is null; 4, for the topologies in the topology set, the number Na of the working grooves needing to be dispatched to the topology according to the process number set for the topology, the total number of the working grooves and the number of the working grooves dispatched already; 5, the working grooves are sequentially taken out of the sequence, and it is controlled in real time that in the working grooves dispatched to the topology, the ratio of the number n of the working grooves belonging to the same working node to the total number of the topology working grooves is smaller than or equal to a set ratio alpha until Na working grooves are taken out and dispatched to the topology.

Description

technical field [0001] The invention belongs to Storm, a mainstream big data real-time streaming computing platform, and relates to the load balancing problem of job scheduling on the Storm platform, in particular to a fine-grained work slot-aware Storm platform job equalization scheduling method. Background technique [0002] In the era of big data, it is a challenge to extract valuable information from large-scale, rapidly and continuously generated data sets in real time. Hadoop, a traditional distributed big data processing system, mainly performs batch calculations, which is suitable for offline batch processing of static data and requires pre-storing the processed input data in a distributed file system. As a real-time distributed big data stream computing platform, Storm is suitable for real-time stream processing of dynamic data. It does not require pre-storage of data but directly performs real-time calculation of data in memory. Since data streaming computing is d...

AI Technical Summary

Problems solved by technology

By analyzing the sorting results, we can see that the sort-slots method has limitations. In the sorting results, some working nodes are always sorted first, and some working nodes are sorted later.

[0019] 2) Due to the problem of the sort-slots method of the class A equal-partition scheduler 1), when multiple topologies are scheduled and allocated, the topological tasks will always be assigned to some of the top-ranked working nodes

During the operation of the Storm cluster, if the working node fails (fast failure), the working process fails, the topology is killed by the user or the new working node is expanded, etc., the working slot resources in the cluster will be released-recycled, causing the topology The tasks are no longer evenly distributed on the working nodes in the cluster, and the cluster resource load is unbalanced

Once this situation is exploited by malicious users in the cluster, it will pose a security threat to the Storm cluster

[0021] 4) In the normal use of the Storm cluster, although the B-type equal share scheduler can realize the equal share of cluster resources in a single topology and multiple topologies, it still belongs to the node-level scheduling strategy

In particular, if the situation mentioned in 3) occurs in the cluster, it is obvious that the current A / B class evenly distributed scheduler that comes with Storm cannot solve the problem of load imbalance

This node-level round-robin scheduling strategy still distributes the new topology tasks equally to each worker node, which will aggravate the load imbalance

Images