Storm task deployment and configuration platform with cluster adaptability

Abstract

The invention discloses a Storm task deployment and configuration platform with cluster adaptability, and belongs to the field of real-time streaming data computing and processing. Through the platform, a Storm cluster can sense internal communication traffic among nodes and remaining resources, and perform running process number configuration self-adjustment according to a topology task demand published by a user and the cluster remaining resources, so that the limitation of the need of artificially specifying a process number in a conventional Storm scheduling method is broken through. A friendly centralized communication traffic monitoring interface is provided for the user by the platform, and is convenient for the user to call in a task program, so that load and resource awareness is realized. Moreover, a one-phase commitment scheduling method which is different from a conventional Storm two-phase commitment scheduling method is implemented on the platform in an embedding way, and communication traffic optimization among different processes at the same node is realized. Through adoption of the Storm task deployment and configuration platform, optimal scheduling can be realized by setting of basic optimization threshold parameters, and great convenience is brought to a cluster user and a manager.

Description

technical field [0001] It relates to a Storm task deployment and configuration platform with cluster adaptability, belonging to the fields of massive data processing and real-time stream computing. Background technique [0002] With the development of information technology, information presents explosive growth. In many information processing problems, it is necessary to perform real-time complex calculations on streaming big data. This is a new data model. Different from traditional data modeling methods, this type of data is suitable for transient data flow modeling. Scenarios such as Weibo popularity, shopping recommendations, and statistics of router datagrams all require complex decision-making on real-time streaming data. [0003] In the traditional data processing mode, the data is often independent of the application, and the system is responsible for storing the data centrally on the disk, and the data is a static and fixed collection. The core value of stream co...

AI Technical Summary

Problems solved by technology

If the number of running processes is set too much, the executors will be distributed to more nodes, which will inevitably increase the communication traffic between nodes. No matter how optimized it is, it is difficult to achieve an ideal scheduling scheme

[0007] (2) If the number of running processes is set too small, the executors will be concentrated in a small number of one or several workers, which may cause thread context switching overhead on the one hand, and more importantly, may cause some nodes to run due to The executor is overloaded and causes downtime

Although these scheduling algorithms can alleviate problems such as node overload and internal communication overhead to a certain extent, they cannot fundamentally solve this problem.

Because users cannot control the global state information of the cluster in real time, blindly setting the number of running processes for tasks in this case will inevitably have a more serious impact on the cluster processing performance

Images