Task and data collaborative scheduling method for wide-area high-performance computing environment

Abstract

The invention provides a task and data collaborative scheduling method for a wide-area high-performance computing environment. The method comprises the following steps: firstly, constructing a task completion time model according to conditions such as a global resource state, a computing task demand and a data layout condition; secondly, according to an optimal scheme selection mechanism, selecting an optimal center-level cooperative scheduling scheme based on a task completion time estimated value and a global resource use state; and then, realizing data redundancy layout based on the data access frequency in the scheduling process. And finally, generating a queue-level scheduling scheme based on a task stealing mechanism so as to correct estimated value deviation of task completion time.According to the method, the computing tasks and the corresponding data can be cooperatively scheduled among a plurality of cross-domain centers and task queues, task allocation and data layout are efficiently and reasonably carried out, and the system time performance and the global resource utilization rate are both considered, so that efficient computing is realized.

Description

technical field [0001] The invention discloses a task and data collaborative scheduling method for a wide-area high-performance computing environment, relates to challenges faced by wide-area high-performance computing, and belongs to the field of computer technology. Background technique [0002] In the high-performance computing environment, for many parallel applications such as weather forecasting and geological observation, the calculation data comes from remote sensing satellites or observation points distributed all over the world, resulting in data storage in geographically distributed computing centers (including supercomputing centers, data processing centers, etc.) center, etc.) and wide area sharing. At the same time, with the increasing complexity of scientific and engineering problems, parallel applications have gradually increased the demand for storage and computing resources. To address the above challenges, the current trend is to execute applications in w...

AI Technical Summary

Problems solved by technology

[0007] The main problems of traditional wide-area scheduling methods are: most task scheduling methods do not consider data migration, but data migration costs account for a large proportion of task completion time, which leads to a large amount of global data migration and task completion time. and long waiting times for results

In addition, most of the existing data scheduling methods focus on the efficient access and migration of data, and seldom consider the correlation between data and computing tasks, which leads to the incomplete matching of data layout and computing task distribution, which leads to relatively Low global resource utilization and long task completion times

However, the existing collaborative scheduling methods only focus on optimizing the scheduling process through one aspect of data layout or task scheduling. The relatively single consideration of scheduling factors leads to unbalanced global resource utilization, so that the system still has a relatively high task completion rate. time and waiting time

Images