Mixed task arrangement method based on message system

Abstract

The invention discloses a mixed task arrangement method based on a message system. The method comprises the steps that a user uploads a calculation task to the message system; job information of the first job is obtained through a task submission module, the job information is stored in a database, and a job state is set to be a first state; the user uploads an input file of the first job to the database and then sets the job state to be a second state; after discovering the first job of which the job state is the second state, the task submission module sends job information to a message system and sets the job state to be a third state; the task submission module receives the jobs, sets the job state as a fourth state, inquires the dependency relationship of the first job through the database, monitors the job state of the first job through the storage resource module if the dependency relationship exists, transmits the output file of the first job to the second job when the first job is completed, and transmits the output file to the second job after the transmission is completed. And setting the working state of the second operation as a second state, continuing to execute the operation calculation, and ending until the current operation does not have the dependency relationship.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of high performance computing, in particular to a hybrid task orchestration method based on a message system. BACKGROUND

[0002] A lightweight workflow engine of a high performance computing (HPC) environment is a task flow management tool specially designed for an HPC scene, aiming to solve the pain points of multi-task dependence scheduling, resource dynamic allocation, and heterogeneous job cooperation in an HPC environment with a "light, efficient, and decoupled" architecture, and support complex workflow automatic execution in fields such as scientific computing, engineering simulation, and data mining. The existing task submission mode of the high performance computing environment only supports the submission of a single job task, and cannot operate a workflow task containing multiple jobs.

[0003] Therefore, a hybrid task orchestration method based on a message system is needed. SUMMARY

[0004] The purpose of the present application is to provide a hybrid task orchestration method based on a message system, which analyzes the dependence relationship between jobs, listens to the job state, and after the current subsequent job execution is completed, transmits the output file to all subsequent jobs and starts and completes them.

[0005] To achieve the above purpose, in a first aspect, the present application provides a hybrid task orchestration method based on a message system, comprising:

[0006] A user uploads a computing task to a message system, parses the computing task through a task submission module to determine the dependence relationship between all jobs in the computing task, and allocates a high performance computing node to each job in the computing task through a scheduling algorithm;

[0007] The task submission module obtains the job information of a first job in the computing task, stores the job information in a database, and sets the job state in the job information to a first state; the user uploads an input file of the first job to the database and sets the job state to a second state; the task submission module queries the job information regularly, finds the first job with the job state of the second state, sends the job information to the message system, and sets the job state to a third state;

[0008] The high performance computing task scheduling and execution module receives the job information from the message system, downloads the input file from the working directory of the first job, sends the input file to an HPC cluster, submits a job execution request to the HPC cluster, receives the job number returned by the HPC cluster, and sends the job number to the message system;

[0009] The task submitting module receives the job number and sets the job state to a fourth state, queries the dependency relationship of the first job through a database, if there is a second job dependent on the first job, issues a listening task to the storage resource module to listen to the job state of the first job, when the first job is completed and the job state is set to a fifth state, transmits the output file of the first job to the working directory of the second job, after the transmission is completed, sets the working state of the second job to the second state, and continues to execute the job calculation until the current job has no dependency relationship.

[0010] Specifically, the computing task includes a single job task and a workflow task.

[0011] Preferably, the first state is a submitting state, the second state is a preparing to start state, the third state is a starting state, the fourth state is a started state, and the fifth state is a job completion state. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 A whole framework diagram of a hybrid task arrangement method based on a message system provided by the embodiment of the application is provided.

[0013] Figure 2 A flowchart of a hybrid task arrangement method based on a message system provided by the embodiment of the application is provided.

[0014] Figure 3 A core module schematic diagram of a hybrid task arrangement method based on a message system provided by the embodiment of the application is provided.

[0015] Figure 4 A HPC task arrangement timing diagram based on a message system provided by the embodiment of the application is provided.

[0016] Figure 5 A workflow execution schematic diagram of a HPC node provided by the embodiment of the application is provided. DETAILED DESCRIPTION

[0017] The technical solutions of the application will be further described in detail below with the drawings and embodiments.

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described below with reference to the drawings. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0019] In the description of embodiments of the present application, the words "exemplary", "for example", or "e.g." are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary", "for example", or "e.g." is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary", "for example", or "e.g." is intended to present concepts in a concrete manner.

[0020] The task submission mode of the existing high-performance computing environment only supports the submission of a single job task, and cannot operate a workflow task containing multiple jobs.

[0021] In order to overcome the deficiencies in the prior art, a hybrid task orchestration method based on a message system is proposed. By analyzing the dependency relationship between jobs, listening to the job state, and after the current subsequent job execution is completed, the output file is transmitted to all subsequent jobs and started.

[0022] Figure 1 The overall framework diagram of the hybrid task orchestration method based on the message system provided by the embodiments of the present application is shown in the figure. The HPC layer is a physical / virtual resource pool that actually executes computing tasks. The front-end machine (FS) layer includes three types of services: file service, resource service, and job service. The file service (BackendProcessor, DataTransferService) is responsible for file transmission between HPC and the system, the resource service (hpcExplorer) collects HPC job states at regular intervals, and the job service (hpcExec) receives requests and submits jobs to HPC and returns job numbers. The message system layer is the asynchronous communication hub: it undertakes information transmission between modules, such as hpcExplorer sending HPC states, storeResource collecting state updates to DB, and job submission. Cross-layer notifications of key events such as these are implemented through it. The center of operations (CS) layer includes four types of modules: resource service (resourceService, StoreResource) receives job states from the message system and updates the DB, manages resource allocation logic, file service (FrontendProxy, DataTransferGlobalProxy) cooperates with the file service of the front-end machine to implement global file transmission scheduling, scheduling service and job service (jobSubmit) analyze job requests (including dependency relationships), call scheduling services to allocate resources and trigger job submission processes, and DB stores core data such as jobs, resources, and states.

[0023] Figure 2 The flowchart of the hybrid task orchestration method based on the message system provided by the embodiments of the present application is shown in Figure 1 The method mainly includes the following steps:

[0024] Step S101: The user uploads a computing task to the message system, and a task submission module is used to parse the computing task to determine the dependency relationship between all jobs in the computing task, and a scheduling algorithm is used to assign a high-performance computing node to each job in the computing task.

[0025] In one embodiment, the computing task includes a single job task and a workflow task.

[0026] For example, the user uploads a computing task to the message system, and the message system receives a single job task request or a workflow task from the user. A parser is used to convert the format of the single job into a JSON format, and a jobSubmit module (task submission module) is used to parse the single job task. Since it is a single job, there is no dependency relationship, while the workflow task includes multiple jobs, and the dependency relationship between the multiple jobs is parsed. A scheduling algorithm is used to assign a high-performance computing node to the job.

[0027] Step S102: The job information of a first job in the computing task is obtained through the task submission module, the job information is stored in a database, and the job state in the job information is set to a first state. After the user uploads an input file of the first job to the database, the job state is set to a second state. The task submission module queries the job information regularly, finds the first job with the job state of the second state, sends the job information to the message system, and sets the job state to a third state.

[0028] In one embodiment, the first state is a submission state, the second state is a ready-to-start state, and the third state is a starting state.

[0029] For example, the job information of a single job is obtained through the jobSubmit module, the job information is stored in a database, and the job state is set to 1 (submission state). After the user uploads an input file, the job state is set to 4 (the job is ready, ready to start). The jobSubmit module queries the job information regularly, finds the job with the state of 4, sends the job information to the message system, and updates the job state to 8 (starting, waiting for scheduling).

[0030] Step S103: The high-performance computing task scheduling and execution module receives the job information from the message system, downloads the input file from the working directory of the first job, sends the input file to the HPC cluster, submits a job execution request to the HPC cluster, receives the job number returned by the HPC cluster, and sends the job number to the message system.

[0031] For example, the hpcExec module (high performance computing task scheduling and execution module) receives job information from the message system, downloads input files from a specified working directory, and sends them to an HPC, then submits a job to the HPC, obtains a job number on the HPC, and sends the job number to the message system.

[0032] Step S104: The task submission module receives the job number and sets the job state to the fourth state, queries the dependency relationship of the first job through a database, if there is a second job dependent on the first job, issues a listening task to the storage resource module to listen to the job state of the first job, when the first job is completed and the job state is set to the fifth state, transmits the output file of the first job to the working directory of the second job, after the transmission is completed, sets the working state of the second job to the second state, and continues to execute job calculation until there is no dependency relationship for the current job.

[0033] In one embodiment, the fourth state is a started state, and the fifth state is a job completed state.

[0034] For example, the jobSubmit module receives a job submission result (i.e., the above-mentioned job number), updates the state to 10 (successfully submitted to the HPC), queries the database whether it needs to be listened to, because a single job has no dependency relationship and does not need to be listened to, so there is no subsequent operation, if it is a workflow task, it needs to find all jobs dependent on the job from the database, and transmit the output file of the job to the working directory of all jobs dependent on the job, after the transmission is completed, update the job state to 4 (the job is ready and ready to start), the jobSubmit module queries the job information in a timely manner, finds the job in the state of 4, sends the job information to the message system, and updates the job state to 8 (starting, waiting for scheduling), and the cycle is repeated until there is no dependency relationship for the current job.

[0035] Figure 3A core module schematic diagram of a hybrid task scheduling method based on a message system provided by the embodiment of the application is shown in the figure. In the workflow processing, CWL, WDL, JSON and other standard workflow formats and other custom formats are supported, format conversion is completed through a workflow parser, and uniform "workflow uniform description" that can be recognized by the system is obtained; then, the workflow information is parsed and persisted, the computing resources are pre-allocated according to a scheduling algorithm, and finally, the job is submitted to a specified queue. In the single job submission process, after the job uniform description is completed, the job is submitted to the specified queue, the job submission result is received and the database is queried, and it is judged whether listening is needed - if it is a single job without dependence, the process is ended; if it is a job with dependence, the "send listening request" operation is performed, and the listening request is sent through the message bus. In the task management (StoreResource) process, the job state is monitored through the message bus after receiving the listening request, and it is judged whether the job reaches the DONE (job completion) state; if it is completed, the subsequent task dependent on the job is found, the output file is transmitted to the subsequent task working directory, the file state is updated, and the closed loop triggered by the dependence is completed.

[0036] Figure 4 A HPC task scheduling timing diagram based on a message system provided by the embodiment of the application is shown in the figure,

[0037] 1. First, the HPC cluster is initialized, including creating a queue (queue), configuring a user map (usermap) and other basic environments, to ensure the compatibility of the HPC and the system interaction;

[0038] Then, a timing collection mechanism is started, and information such as queue load, job running state and user permission is obtained from the HPC at a fixed period (such as 10 minutes for cluster state collection and 1 minute for job state collection), and the collected data is synchronized to the system through the message bus, to provide a real-time HPC resource view for subsequent job scheduling.

[0039] 2. The jobSubmit module drives the state closed loop:

[0040] After the user submits a job request, jobSubmit parses the request information (including job type, resource requirement, input file path, etc.), stores it in the database, and sets the initial state to 1 (submission state);

[0041] After the user uploads the input file to the specified directory, jobSubmit detects that the file is ready, and updates the state to 4 (the job is ready to start);

[0042] jobSubmit queries the database at a fixed time, finds the job with state 4, sends the job information to hpcExec through the message bus, and updates the state to 8 (starting, waiting for scheduling);

[0043] hpcExec module executes the submission: receives job information from the message bus, generates an HPC identifiable execution file (such as a Slurm script), downloads input files from the specified working directory and transmits them to the HPC cluster; after submitting the job to the HPC, the unique job number returned by the HPC (used to track the execution process) is obtained, and the job number is returned through the message bus;

[0044] After jobSubmit receives the job number, it confirms that the job has been successfully submitted to the HPC, and updates the status to 10 (execution has started).

[0045] 3. The storeResource module realizes the dependence linkage:

[0046] If the job is a workflow task (there are subsequent dependent jobs), jobSubmit will issue a "listening task" to storeResource through the message bus before submission, specifying the job ID to be listened to and the trigger condition (usually "job status changes to 20");

[0047] storeResource continuously receives HPC job status synchronized by hpcExplorer through the message bus, and when it listens to the target job status changing to 20 (job completion), it triggers the subsequent operation:

[0048] Query all subsequent tasks dependent on the job from the database;

[0049] Call the file service to transfer the output files of the previous job to the working directory of the subsequent job;

[0050] After the transmission is completed, the status of the subsequent job is updated to 4 (the job is ready and is ready to start), so that it enters the job submission and status flow stage, and repeats the execution logic of the jobSubmit module driven state closed loop.

[0051] Figure 5A working flow execution schematic diagram of an HPC node provided by the embodiment of the present application is shown in the figure. Input working flow task description is provided, and the scheduling algorithm is used to allocate the optimal HPC node for each job (e.g., job 1 to node 1, job 2 to node 2, and job 3 to node 3, wherein the nodes 1, 2, and 3 can be the same or different) and match the corresponding application (APP1, APP2, and APP3). Then, job 1 uploads the input file data0.in to execute the output data1.out, job 2 receives data1.out to execute the output data2.out, and job 3 receives data1.out and data2.out to execute the output data3.out and download, the whole process is automatically connected through file dependency, the multi-HPC node cooperation is realized, the technical advantages of each node are maximized, and the whole process automation of complex scientific calculation is supported.

[0052] It can be understood that the method steps in the embodiment of the present application can be realized by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0053] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0054] The above detailed description of the specific embodiments of the present application further illustrates the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A hybrid task orchestration method based on a message system, comprising: Users upload computing tasks to the messaging system. The task submission module parses the computing tasks to determine the dependencies between all jobs in the computing tasks. A scheduling algorithm then allocates high-performance computing nodes to each job in the computing tasks. The task submission module obtains the job information of the first job in the computing task, stores the job information in the database, and sets the job status in the job information to the first status. After the user uploads the input file for the first job to the database, the job status is set to the second status; The task submission module periodically queries the job information. After discovering the first job with the job status in the second state, it sends the job information to the message system and sets the job status to the third state. The high-performance computing task scheduling and execution module receives the job information from the message system, downloads the input file from the working directory of the first job, sends the input file to the HPC cluster, submits a job execution request to the HPC cluster, receives the job number returned by the HPC cluster, and sends the job number to the message system. The task submission module receives the job number and sets the job status to the fourth state. It queries the database for the dependencies of the first job. If there is a second job that depends on the first job, it publishes a listening task to the storage resource module to monitor the job status of the first job. When the first job is completed and its job status is set to the fifth state, it transmits the output file of the first job to the working directory of the second job. After the transmission is completed, it sets the working status of the second job to the second state and continues to execute the job calculation until the current job has no dependencies.

2. The method according to claim 1, wherein, The computational tasks include individual job tasks and workflow tasks.

3. The method according to claim 1, wherein, The first state is the submission state, the second state is the ready-to-start state, the third state is the starting state, the fourth state is the started state, and the fifth state is the job completed state.

Images