A hybrid scheduling method and system for wide-node scientific workflow on cloud platform

Abstract

The invention discloses a mixed type scheduling method and system for a cloud platform wide node scientific workflow. The method comprises the following steps that: when each scheduling stage is initialized, utilizing bond energy algorithm transformation to construct a task into a task multilayer clustering tree structure; in a global initialization stage, utilizing the bond energy algorithm transformation and basic dichotomy multi-layer division to construct scheduling resources into a cloud resource multilayer tree structure; according to the mapping deadline of a task group and the each-dimension resource distribution of scheduled resources, carrying out a mapping operation on the node of a task subtree in the multilayer clustering tree structure and the node in a cloud resource multilayer tree structure; at the scheduling tail of the stage, judging whether the amount of tasks which finish the mapping operation exceeds a set threshold value or not; and if the amount of tasks which finish the mapping operation exceeds the set threshold value, classifying tasks which do not finish mapping to a next scheduling stage, and re-distributing the deadline of residual tasks. By use of themethod or the system, the scheduling method of the dynamic scheduling and the static scheduling of dual features is fused, and the energy cost optimization of a cloud resource provider is realized onthe basis of meeting user execution time requirements.

Description

technical field [0001] The invention relates to the field of cloud computing task scheduling and resource allocation, in particular to a hybrid scheduling method and system for wide-node scientific workflow on a cloud platform. Background technique [0002] The research on task scheduling and collaborative allocation of resources in cloud data centers is of great significance. For cloud resource renters, it can reduce rental fees. For cloud computing data centers, it can optimize resource utilization, improve user satisfaction, and reduce energy consumption. , has been a research hotspot in recent years. Research on cloud scientific workflow scheduling is an important part of cloud scheduling research. Scientific workflow is a type of computing task that can be fully or partially automatically executed. According to a series of rules, data and tasks can be transferred between different executors. delivery and execution. Compared with the traditional workflow running enviro...

AI Technical Summary

Problems solved by technology

[0003] (1) Adopt static scheduling method or dynamic scheduling method, but it is difficult to apply to the scheduling of data-intensive wide-node scientific workflow

An unavoidable defect of the static algorithm is that the execution time of the task cannot be accurately estimated, resulting in poor generalization and robustness in practical applications, and the wide-node scientific workflow includes more tasks that require distributed batch processing group, task execution time is more difficult to estimate

Although the dynamic scheduling method has better adaptability to runtime state changes, but at the same time it is difficult to achieve global overall optimization because it cannot grasp the global information, that is, there is a so-called short-sighted defect

For data-intensive wide-node scientific workflows, this defect is even more serious. If global data dependencies are not considered during scheduling, frequent network data transmissions will be caused, which will seriously affect the overall execution efficiency.

[0004] (2) The current deadline decomposition algorithm only performs deadline generation for independent tasks, and does not consider the batch task set structure

Deadline decomposition is the basis for workflow task scheduling. The traditional method distributes the overall deadline of the workflow in proportion to the size of a single task. Direct application to wide-node scientific workflow will lead to unbalanced distribution, resulting in nodes containing a large number of short tasks. Components have tight deadlines, while node components containing a large number of long tasks have loose deadlines, resulting in an increase in the deadline default rate and waste of computing resources

[0005] (3) For workflow scheduling involving massive batch tasks, existing methods based on data-dependent clustering to reduce time-consuming data transmission still need some improvement

Most of these methods aim to optimize the frequency of data transmission. For workflows involving massive batch processing tasks, the data scale is larger and the size of the data sets varies widely. The transmission frequency cannot represent the transmission cost, and additional data transmission volume and network bandwidth are required. factors to consider

Images