Task scheduling and resource allocation method and system for real-time cloud platform

Abstract

The invention relates to a task scheduling and resource allocation method and system for a real-time cloud platform. According to the task scheduling and resource allocation method and system for the real-time cloud platform, a global state storage module obtains the running state of the cloud platform and reports the running state to a global state monitoring module; the global state monitoring module develops a corresponding scheduling strategy through a task allocation matrix, a task adjacency matrix and a mask matrix on the basis of the running state; node-driven and/or task-driven task scheduling and resource allocation are performed on the real-time cloud platform on the basis of the scheduling strategy. The task scheduling and resource allocation method and system for the real-time cloud platform can take the relationship among tasks into full consideration during task allocating and reduce traffic among nodes and bandwidth pressure so as to improve the performance of the platform, can perfectly adapt to all situations of dynamic scheduling of the cloud platform to ensure that the cloud platform can maintain high computing performance and resource utilization rate at all times during operation, and are low in time complexity and applicable to cloud environments with large-scale nodes and large amount of tasks.

Description

technical field [0001] The invention relates to the field of real-time cloud computing, in particular to a method and system for task scheduling and resource allocation oriented to a real-time cloud platform. Background technique [0002] The amount of data in today's society is expanding day by day, and more and more data appears in the form of large-scale and continuous streams. The value of data decreases over time, so it is required to process the data as soon as they appear, rather than cache them for batch processing. For example, search engines process tens of thousands of queries per second, and each page contains multiple advertisements. In order to process user feedback in a timely manner, a low-latency, scalable, and highly reliable processing engine is required. The traditional DBMS or the method of using Map / Reduce for real-time data stream processing can hardly meet the application requirements. [0003] For this reason, many stream computing platforms have e...

AI Technical Summary

Problems solved by technology

[0005] 1. Existing real-time cloud platforms, such as Twitter's Storm, assign tasks as independent units without considering the relationship between tasks. In fact, from the perspective of improving platform efficiency, interrelated tasks should be assigned to the same or adjacent nodes;

[0006] 2. The existing real-time cloud platform only considers the CPU and memory usage of tasks, but does not consider the communication volume between tasks and the upstream and downstream relationships of tasks;

[0007] 3. The existing real-time cloud platform only considers the initial or static allocation problem, but ignores the important feature that the platform is open and tasks and nodes change dynamically. The allocation strategy during the operation of the platform will limit its efficiency. an important factor;

[0008] 4. The classic multi-core task allocation algorithm has high complexity and has advantages when the number of cores is small and the amount of tasks is small. However, the data volume, task volume, and node scale of the cloud platform are beyond the processing range of traditional algorithms, making The allocation algorithm of the real-time cloud platform is urgent and necessary

Images