High-throughput task processing method based on super computer

Abstract

The invention relates to a supercomputer-based high-throughput task processing method, and the method comprises the following steps: obtaining the task parameters of a to-be-executed task, wherein the task parameters comprise task physical information and task execution information, the task physical information comprises a job chain type and an application program type, and the task execution information comprises a job chain number M corresponding to a to-be-executed task, a logical relationship between job chains, a job number N of each job chain, a job structure file, a logical relationship between jobs and execution precision; obtaining a corresponding workflow template from a preset database according to the task parameters of the to-be-executed task to generate M workflows in one-to-one correspondence with the M job chains; distributing the M workflows to the computing nodes corresponding to the super computer according to the current computing resources for parallel execution, and storing a final task execution result in the database. According to the method, the calculation efficiency of the high-throughput calculation task is improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a supercomputer-based high-throughput task processing method. Background technique [0002] Existing supercomputers are usually used to perform large-scale, long-term high-performance computing tasks, which usually only include a few job chains, but each job chain requires a lot of computing resources and a long execution cycle, so High-performance computing usually focuses on providing huge computing power in a short period of time, such as climate weather forecasting, oil exploration, etc. In actual use, there are many users who need to perform computing tasks with a small amount of tasks and short running time. Such tasks usually include many job chains that need to be executed in parallel at the same time, usually more than a thousand. Tasks are called high-throughput tasks. Compared with high-performance computing job chains, these job chains occupy relatively less compu...

AI Technical Summary

Problems solved by technology

If the high-performance computing technology of supercomputers is directly used to perform high-throughput computing tasks, the utilization efficiency of supercomputing systems will be greatly reduced, and the processing efficiency of high-throughput tasks will be low.

Images