Scientific workflow scheduling method and device

Abstract

The invention discloses a scientific workflow scheduling method and device under a network environment. The method comprises the following steps that heterogeneous computing resources used as targets are inquired, and computing power PCj of each heterogeneous computing resource is recorded; all schedulable tasks in workflows used as targets are inquired; the task ratio p (ti, rj) of each task ti in all the schedulable tasks on the j heterogeneous computing resource rj is computed, and ct (ti, rj) = ext (ti, rj) + rt (ti, rj); in all the obtained task ratios, the largest task ratio p (tm, rn) is obtained, and the m task tm is correspondingly scheduled to the n heterogeneous computing resource rn to be executed. By means of the scientific workflow scheduling method and device, good resource load balance is achieved, the method and device can be adaptive to static and moving environments, scheduling time is short, total execution time is short, and combination property is excellent in the existing scheduling algorithms.

Description

【Technical field】 [0001] The invention relates to the field of data processing, in particular to a scientific workflow scheduling processing method and device. 【Background technique】 [0002] Grid is a global network infrastructure that provides virtual services for scientific research and business operations by integrating large-scale, dispersed, and heterogeneous computing resources, storage resources, and data resources. In recent years, more and more scientific fields such as biomedicine, geography, astrophysics, etc. have begun to use grids to share, manage and process large data sets within and between disciplines. In this environment of big data and intensive computing, the application of scientific workflow in grid becomes more and more important. The main purpose of adopting scientific workflow is to modularize and opaque the complex process of processing big data, to realize the simple call and reuse of the process that requires multiple calculations and repeated ...

AI Technical Summary

Problems solved by technology

Since the DAG scheduling problem is a problem of NP complexity, it is impossible to use strict mathematical derivation to obtain the optimal scheduling strategy. At present, many scholars at home and abroad have proposed many heuristic and meta-heuristic algorithms

The Myopic algorithm is the simplest scheduling algorithm. It arbitrarily schedules a schedulable task to an optimal resource each time, but the result is often a long execution time and poor resource load balance. The Min-Min algorithm Execute the task with the minimum expected execution time among all executable tasks, and allocate corresponding resources until all task scheduling is completed. The scheduling result can generally shorten the total execution time, but when the resource difference is large, the load is not balanced. The corresponding total execution time will be longer; the Max-Min algorithm is an improvement on the Min-Min algorithm. During the execution process, only the task with the maximum expected execution time of all executable tasks is executed each time, and will It is mapped to the resource that takes the least time, and has good load balance, but it is not as good as the Min-Min algorithm in the case of fewer short tasks and more long tasks

The Sufferage algorithm looks for the task with the largest scheduling loss to be executed first. Generally speaking, the balance is good, but the execution effect is not good in a multi-cluster environment; DCP (Dynamic Critical Path, dynamic critical path algorithm) first calculates the earliest start of each task If the time and the latest start time are equal, the tasks on the critical path are considered to be executed first. Generally, the scheduling algorithm takes a longer time than the previous ones, and the total execution time is also longer in the case of irregular workflows; The genetic algorithm is a meta-heuristic algorithm, which needs to search for the optimal solution in the global scope, so the algorithm itself takes the longest time to execute, and sometimes it will fall into the local optimum due to improper setting of the fitness function and other conditions, resulting in unsatisfactory results and relatively poor dynamic characteristics. Poor, the result is not as good as the above algorithms

Images