Storm-based stream computing bipartite graph task scheduling method

Abstract

The invention discloses a Storm-based stream computing bipartite graph task scheduling method. The method is characterized in that Storm DAJG (Directed Acyclic Job Graph) nodes and cluster physical machine UNG (Undirected Node Graph) nodes are seen as two types of vertexes of a bipartite graph; a bipartite graph model is built; and in combination with computing capabilities of nodes of physical machines in a cluster and network transmission delay in the cluster, and according to schedulable relationships between tasks and node resources, task scheduling is performed by adopting an algorithm for solving maximum weight matching of the bipartite graph. According to the method, under the condition of ensuring resource load balance of the physical machines in the cluster, the network delay in adata stream migration process during task execution is shortened, so that the overall performance of a system is improved.

Description

technical field [0001] The invention relates to a Storm-based stream computing bipartite graph task scheduling method. Background technique [0002] With the rapid development of information science and technology, applications in the cloud computing and Internet of Things environment present the characteristics of large data volume, continuous data flow, multi-source concurrency, and real-time processing. The real-time processing of these streaming data is called streaming data processing or streaming computing. In the big data streaming computing system Storm, multi-task scheduling is a key factor affecting the performance of the streaming computing system Storm. Tasks in a streaming computing system have two typical characteristics: [0003] (1) Multi-task and multi-stage features. [0004] From a theoretical model point of view, the tasks submitted to the system can be represented by a directed acyclic graph (DAG) during processing [1]. That is to say, task schedulin...

AI Technical Summary

Problems solved by technology

[0010] (1) A certain type of task is sensitive to CPU or memory. If the same CPU-sensitive task is scheduled on the same physical machine, it may make the multi-dimensional resource utilization of the machine unbalanced. For example, the CPU resource load is too high, while the memory resources are idle;

[0011] (2) In a heterogeneous cluster, different physical machines have different resources (CPU, memory, network bandwidth, etc.), and a simple even distribution strategy for task threads may lead to excessive physical loads with scarce resources, resulting in system throughput. decline;

[0012] (3) The data of a certain task is on node A, but it is scheduled to be executed on node B, which undoubtedly increases the bandwidth and delay overhead of reading data. The influence in can not be ignored;

Images