Network flow optimization-based Storm task scheduling method

Abstract

The invention provides a network flow optimization-based Storm task scheduling method, and relates to the field of data exchange networks. The method comprises the following steps of: carrying out statistical analysis on flows between processes; carrying out network k division by adoption of a genetic algorithm according to a communication relationship matrix and optimizing flows between Worker processes in machines; and triggering task distribution through a Storm command rebalance. According to the method, flows between nodes are decreased, the using efficiency of existing resources is improved, much unnecessary flow transmission is decreased, the system delay is reduced, the throughput capacity is improved, the problem that flow information between tasks in Storm task scheduling is insufficiently considered is solved, the problem that flows between machines in systems and flows between processes in the machines are unbalanced is solved, the system delay is reduced and the throughputis improved.

Description

technical field [0001] The invention relates to the field of data exchange networks, in particular to a Storm task scheduling method. Background technique [0002] With the development of computers, real-time streaming data such as sensor data, financial data, and online transaction data continue to increase. There are many types of data, large volume, continuous flow, and strong timeliness. These real-time data are generated quickly, disappear quickly, and have a short effective time. Current real-time data contains rich value, and the efficient processing of real-time data ensures the rapid processing of transactions and real-time acquisition of information. Currently, distributed real-time streaming data processing frameworks such as Storm and Spark have been launched in the industry. Storm is a distributed, scalable, fault-tolerant real-time stream processing system. These characteristics of Storm make it widely used in the industry. But Storm is still in the proces...

AI Technical Summary

Problems solved by technology

[0005] In order to overcome the deficiencies of the existing technology, the existing scheduling method of Storm does not take into account the communication overhead between tasks, resulting in unbalanced system data traffic, large traffic between nodes, high delay and decreased throughput. The present invention provides a The Storm task scheduling method based on network traffic optimization monitors the process traffic when the task is running, analyzes the traffic relationship between processes, and provides an optimized scheduling plan, without changing the number of existing worker processes and thread tasks as much as possible. Undergo, communication traffic optimization

Images