A Storm Task Scaling and Scheduling Algorithm Based on Data Flow Prediction

Abstract

The invention relates to a Storm task expansion scheduling algorithm based on data stream prediction, and belongs to the field of data exchange networks. Through a monitoring module, the real-time operation data of a Topology task submitted by a user can be obtained, the degree of parallelism of a connected component in Topology under a situation that a component load is met is solved, and then, the degrees of parallelism of all components in the Topology can be solved through iteration. A time series model is used for predicting a data size which needs to be processed by the Topology, the optimal degree of parallelism of a startup component spout in the Topology under the situation is solved, the optimal degree of parallelism of each component in the Topology under a prediction condition is obtained, and scheduling is carried out. In scheduling, an on-line scheduling algorithm is used to reduce inter-node network communication to a largest degree and guarantee the load balance of a cluster. By use of the algorithm, a deficiency that relevance among all components in the Topology is not fully considered is overcome, the deficiency that the optimal degree of parallelism of each component in the Topology submitted by the user can not be quickly and efficiently solved is mad up, and the algorithm has the advantages that change is predicted in advance, handling capacity is improved and handling time delay is lowered.

Description

technical field [0001] The invention belongs to the field of data exchange networks, and relates to a Storm task scaling algorithm based on data flow prediction. Background technique [0002] The popularization and promotion of emerging technologies and application models such as cloud computing, Internet of Things, social media, and mobile Internet have led to a sharp increase in the amount of global data and pushed human society into the era of big data. In the context of big data, data contains rich connotations and values, the timeliness of data is becoming more and more important, the streaming characteristics of data are becoming more and more prominent, and the importance of streaming computing is also becoming more and more prominent. The industry has launched streaming computing frameworks such as S4, Spark, and Storm. Storm is a real-time, distributed and highly fault-tolerant computing system. Storm can process large batches of data, and can also make the proces...

AI Technical Summary

Problems solved by technology

[0006] In the existing Storm elastic scaling, insufficient consideration is given to the correlation between components in the Topology. At the same time, during elastic scaling, simply add or reduce the parallelism of each component until a better parallelism of each component is obtained. In this process, multiple task scheduling may be performed, and each task scheduling has time overhead, so the processing delay of tuple is increased to a certain extent.

At the same time, the existing scaling adjustments only adjust the parallelism of each component in the topology submitted by the user when the system load changes, and each adjustment takes a certain amount of time, thus reducing the system to a certain extent. throughput

Images