Calculation task scheduling method and device, storage medium and electronic equipment

Abstract

The invention discloses a calculation task scheduling method and device, a storage medium and electronic equipment, an engine server side receives calculation tasks submitted by a user, task arrangement information and configuration files of the calculation tasks, and the configuration files comprise task types. According to the task arrangement information, the workflow of each calculation task is determined, and for each calculation task contained in the workflow, the calculation task is distributed to a target cluster according to the task type indicated by the configuration file of the calculation task, so that the target cluster executes the calculation task according to the configuration file of the calculation task. The target cluster is a computing cluster matched with the task type indicated by the configuration file of the computing task. According to the method, all the calculation tasks can be automatically scheduled according to the workflow, all the calculation tasks contained in the workflow can be scheduled to the target clusters matched with the task types of the calculation tasks to be executed according to the task types of the calculation tasks, and cross-cluster scheduling of the workflow is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a computing task scheduling method, device, storage medium and electronic device. Background Art

[0002] Nowadays, more and more scientific research projects and industrial applications require large-scale computing, such as material property determination, gene sequencing, protein analysis, etc. These large-scale calculations involve large amounts of computing data and complex calculation processes. They often include code execution, professional software operation, mirror operation and other different types of computing tasks. Each computing task has an execution order, and high-throughput computing needs to be performed according to the workflow of each computing task.

[0003] High Throughput Computing (HTC) is a computing model that uses parallel and distributed computing resources to efficiently process large-scale computing tasks. In high-throughput computing, it is necessary to rely on a task scheduling framework to split large-scale computing tasks into small tasks and assign each small task to different computing nodes in the computing cluster for parallel execution.

[0004] Existing task scheduling frameworks, such as Apache Airflow and Spark, are used to schedule computing tasks within a computing cluster, but cannot implement cross-cluster task scheduling. And the existing scheduling framework can only schedule computing tasks to the same computing cluster for execution, but cannot schedule computing tasks to multiple different computing clusters for execution, which limits the execution efficiency of computing tasks.

[0005] Therefore, this specification provides a computing task scheduling method. Summary of the invention

[0006] The present specification provides a computing task scheduling method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions: This specification provides a computing task scheduling method, which is applied to an engine server, where the engine server is deployed on any login node included in a plurality of computing clusters. The method includes: Receive multiple computing tasks submitted by a user, task scheduling information, and a configuration file of each computing task, wherein the configuration file includes a task type; Determining the workflow of the plurality of computing tasks according to the task scheduling information; The configuration file of the computing task is sent to the designated memory access area, and for each computing task included in the workflow, the computing task is assigned to the target cluster in turn according to the task type indicated by the configuration file of the computing task, so that the target cluster reads the configuration file of the computing task in the designated memory access area to execute the computing task, and the target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

[0008] Optionally, sending the configuration file of the computing task to a designated memory access area of ​​the target cluster specifically includes: The configuration files of the computing tasks are uploaded to a shared storage, and the multiple computing clusters have read and write permissions to the shared storage.

[0009] Optionally, execution of at least one computing task among the computing tasks requires input of a source data file; The method further comprises: The source data file submitted by the user is obtained, and the source data file is uploaded to the shared storage.

[0010] This specification provides another computing task scheduling method, which is applied to a target cluster, wherein the target cluster is a computing cluster adapted to a task type indicated by a configuration file of a computing task, determined by the engine server according to the computing task scheduling method applied to the engine server by the engine server, for each computing task included in a workflow determined by a plurality of computing tasks submitted by a user, the method comprising: Receiving the computing task assigned by the engine server; In a designated memory access area, read a configuration file of the computing task and execute the computing task, wherein the designated memory access area includes the configuration file of the computing task uploaded by the engine server; The calculation result of the calculation task is returned to the engine server.

[0011] Optionally, the execution of the computing task requires a source data file, and the configuration file of the computing task also includes a source file storage directory, and the configuration file is stored in a shared storage; Execute the computing task according to the configuration file of the computing task, specifically including: Read the configuration file of the computing task and determine the source file storage directory; Obtaining the source data file of the computing task from the source file storage directory of the shared storage; Execute the computing task according to the source data file of the computing task.

[0012] Optionally, the configuration file of the computing task further includes a result storage directory of the computing result of the computing task in the shared storage; Return the calculation result of the calculation task to the engine server, specifically including: Determine a calculation result of the calculation task, and determine a result storage directory of the calculation result of the calculation task in the shared storage according to a configuration file of the calculation task; According to the result storage directory, storing the calculation result in the shared storage; If the calculation result is numerical data, the calculation result is returned to the engine server; if the calculation result is file data, the result storage directory is returned to the engine server.

[0013] This specification provides a computing task scheduling device, which is applied to an engine server, and the engine server is deployed on any login node included in multiple computing clusters. The device includes: A task receiving module receives multiple computing tasks submitted by a user, task scheduling information, and a configuration file of each computing task, wherein the configuration file includes a task type; A task scheduling module, which determines the workflow of the plurality of computing tasks according to the task scheduling information; The scheduling module sequentially assigns each computing task included in the workflow to a target cluster according to the task type indicated by the configuration file of the computing task, and sends the configuration file of the computing task to a designated memory access area of ​​the target cluster, so that the target cluster reads the configuration file of the computing task in the designated memory access area to execute the computing task. The target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

[0014] This specification provides another computing task scheduling device, which is applied to a target cluster, wherein the target cluster is a computing cluster adapted to the task type indicated by a configuration file of the computing task, determined by the engine server according to the computing task scheduling method applied to the engine server by the engine server in accordance with the present specification, for each computing task included in a workflow determined by multiple computing tasks submitted by a user, and the device comprises: A task receiving module receives the computing task assigned by the engine server; The execution module reads the configuration file of the computing task in the specified memory access area and executes the computing task; The result returning module returns the calculation result of the calculation task to the engine server.

[0015] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned computing task scheduling method is implemented.

[0016] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned computing task scheduling method when executing the program.

[0017] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the computing task scheduling method provided in this specification, the engine server is deployed on any login node included in multiple computing clusters, and the engine server receives multiple computing tasks submitted by users, task scheduling information, and configuration files of each computing task, and the configuration files include task types. According to the task scheduling information, the workflow of multiple computing tasks is determined, and for each computing task included in the workflow, the computing task is assigned to a target cluster according to the task type indicated by the configuration file of the computing task, so that the target cluster executes the computing task according to the configuration file of the computing task, and the target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

[0018] In this method, not only can each computing task be automatically scheduled and executed according to the workflow, but also each computing task included in the workflow can be scheduled to a target cluster that is adapted to the respective task type for execution according to the task type of the computing task, thereby realizing cross-cluster scheduling of the workflow, making more full use of the computing resources of the computing cluster, and thus improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 A schematic diagram of a computing task scheduling system in this specification; Figure 2 An architecture diagram of a computing engine provided in an embodiment of this specification; Figure 3 A material calculation application scenario diagram provided in an embodiment of this specification; Figure 4 A flowchart of a computing task scheduling method provided in this specification; Figure 5 A flowchart of a computing task scheduling method provided in this specification; Figure 6 A flowchart of a computing task scheduling device provided in this specification; Figure 7 A flowchart of another computing task scheduling device provided in this specification; Figure 8 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0022] Figure 1 This is a schematic diagram of the interaction process of a high-throughput computing task scheduling system in this specification. The system includes an engine server and multiple computing clusters. The engine server is deployed on any login node included in the multiple computing clusters. The computing cluster can be any type of server cluster, such as a cloud server cluster, a high-performance computing cluster, etc. A computing cluster can have one login node, which is the entrance for the engine server to establish a communication connection with the computing cluster. The engine server is deployed on one of the login nodes of a computing cluster and is connected to the login nodes of other computing clusters through a network.

[0023] In this method, the engine server needs to assign the computing task to a computing cluster for execution, and the computing cluster assigned to execute the computing task serves as the target cluster of the computing task. Figure 1 The scheduling process of a computing task shows the interaction between the engine server and the target cluster. Figure 1 The interaction process between the engine server and the target cluster shown may include the following steps: S100: The engine server receives multiple computing tasks submitted by a user, task scheduling information, and a configuration file of each computing task, wherein the configuration file includes a task type.

[0024] A complex computing task includes multiple different types of computing tasks, such as code execution tasks, script execution tasks, software running tasks, virtual image tasks, etc. Different types of computing tasks have different operating environments. For example, the computing nodes of code execution tasks need to have the configuration environment required for code execution tasks, the computing resources of the computing nodes of software running tasks can support software running, and the computing nodes of virtual image tasks need to have Docker image services installed.

[0025] Different computing clusters have their own resource advantages. For example, high-performance computing clusters are suitable for scientific simulations such as materials science and fluid dynamics, while cloud server clusters are suitable for tasks such as big data analysis and script execution. In the current task scheduling method, multiple computing tasks are often scheduled on computing nodes within a computing cluster, that is, each computing task is scheduled to different servers in the same computing cluster for computing, which cannot fully utilize the resource advantages of different computing clusters and improve computing efficiency.

[0026] For example, in the simulation and calculation of material properties, it is necessary to first generate parameterized structures, that is, to design and generate tens of millions of microstructures according to various parameters such as material size, composition ratio, morphology, etc., and then simulate each microstructure to calculate the Young's modulus, Poisson's ratio and other properties of each microstructure, and then verify the calculated properties, and finally store the verified properties in the database. In this calculation work, parameterized structure generation, microstructure simulation, filling verification, and database storage need to be performed in sequence, and the task types of each calculation task are different, namely script execution task, simulation software running task, virtual mirror task, and code execution task. The execution environment of each calculation task is inconsistent and cannot be executed in the same computing cluster. Therefore, cross-cluster execution is required to complete the calculation of each calculation task in the material property simulation calculation work.

[0027] The engine server in this specification is used to interact with users to receive computing tasks submitted by users and configuration files required for computing task execution. There is a data dependency relationship between each computing task. Each computing task may be executed in a serial order or in a parallel order, or partially in a serial order and partially in a parallel order. The configuration file is used to indicate the execution steps of the computing task, including the task execution instructions of the computing task, and the task scheduling information is used to indicate the execution order of each computing task.

[0028] The task scheduling information may be a program instruction, in which a tree structure or a chain structure may be used to represent the execution order of various computing tasks. The task scheduling information may also be a flowchart submitted by a user through a push-and-drag connection operation. In the engine server, various task components operable by the user are included. Different task components represent computing tasks of different task types. The user can drag and connect various task components to configure a flowchart.

[0029] The configuration file of this specification contains the task type of the computing task. The engine server will determine the computing cluster that is suitable for the computing resources required by the computing task according to the task type of the computing task, so as to realize cross-cluster scheduling of different computing tasks.

[0030] S102: The engine server determines the workflow of the plurality of computing tasks according to the task scheduling information.

[0031] The engine server determines the workflow of each computing task according to the execution order of each task indicated in the task scheduling information submitted by the user.

[0032] After determining the workflow of each computing task, the engine server can schedule each computing task to the computing cluster in order according to the data dependency of each computing task in the workflow.

[0033] S104: The engine server sends the configuration file of the computing task to the designated memory access area, and sequentially allocates each computing task included in the workflow to a target cluster according to the task type indicated by the configuration file of the computing task.

[0034] Because in this manual, the user submits the computing task to the engine server, and the engine server schedules each computing task to different computing clusters for execution, and the computing cluster needs to read the configuration file and execute the computing task according to the instructions in the configuration file, so the engine server needs to enable the computing cluster that executes the computing task to access the configuration file of the computing task in order to execute the computing task.

[0035] Therefore, the engine server sends the configuration file of the computing task to the specified memory access area.

[0036] In one embodiment, the designated memory access area may be a shared storage, and each computing cluster mounts the shared storage, so each computing cluster has read and write permissions to the shared storage. This specification does not limit the specific implementation of the shared storage, and the shared storage can be constructed through an object storage service (OSS), a network file system (NFS), etc.

[0037] The engine server monitors the execution status of each computing task in real time. For a computing task, when the engine server determines that the computing task has no predecessor task according to the workflow, or determines that the predecessor tasks of the computing task have been completed according to the workflow, it starts scheduling the computing task.

[0038] Specifically, when a computing task is the first computing task that needs to be executed according to the workflow, the engine server determines that the computing task has no predecessor task and starts scheduling the computing task. When a computing task is not the first computing task that needs to be executed, that is, the computing task has a predecessor task, the engine server needs to determine that all predecessor tasks of the computing task have been executed and start scheduling the computing task.

[0039] The engine server allocates each computing task to the target cluster according to the execution status of each computing task in the workflow and the task type indicated in the configuration file of the computing task. And according to the resource usage status of each computing node in the target cluster, the computing task is allocated to the computing node of the target cluster for execution based on the load balancing principle or the application of the Slurm scheduling framework.

[0040] Different types of computing tasks require different computing resources and operating environments. The engine server determines the task type of the computing task based on the configuration file of the computing task, and determines the computing cluster that matches the computing resources required by the computing task as the target cluster of the computing task based on the task type of the computing task. The engine server assigns the computing task to the target cluster so that the target cluster executes the computing task.

[0041] S106: The target cluster receives the computing task assigned by the engine server.

[0042] S108: The target cluster reads the configuration file of the computing task in the designated memory access area and executes the computing task.

[0043] S110: Return the calculation result of the calculation task to the engine server.

[0044] After the target cluster receives the computing task assigned by the engine server, it reads the configuration file of the computing task in the specified memory access area, executes the computing task according to the configuration file, and returns the computing result to the engine server after the computing task is completed. After the engine server receives the computing result of the computing task, it can determine that the computing task is completed and start scheduling the next computing task of the target task in the workflow.

[0045] Specifically, the target cluster reads the configuration file in the specified memory access area, determines the task execution instruction of the computing task, and executes the computing task according to the task execution instruction of the computing task.

[0046] Different computing requirements correspond to different task execution instructions. Task execution instructions are execution instructions designed by users based on the computing objectives of computing tasks. After a computing cluster is assigned a computing task, it reads the configuration file of the computing task and executes the task execution instructions of the computing task to implement the execution of the computing task.

[0047] For example, the task execution instructions corresponding to the code execution task are program codes written by the user, but the program codes of different computing tasks are different. The task execution instructions corresponding to the script execution task are script commands written by the user. The task execution instructions corresponding to the software running task include the software to be run, the software running environment, the software installation package, etc. The task execution instructions corresponding to the virtual image task may include Docker container creation instructions, services supported by different Docker containers, etc.

[0048] like Figure 1 In the scheduling system shown, the engine server is deployed on any login node included in multiple computing clusters. The engine server receives multiple computing tasks, task scheduling information and configuration files of each computing task submitted by the user, and the configuration file includes the task type. According to the task scheduling information, the workflow of multiple computing tasks is determined, and each computing task included in the workflow is assigned to the target cluster in turn according to the task type indicated by the configuration file of the computing task, so that the target cluster executes the computing task according to the configuration file of the computing task, and the target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task. In this method, not only can the automatic scheduling and execution of each computing task be realized according to the workflow, but also each computing task included in the workflow can be scheduled to the target cluster adapted to the respective task type according to the task type of the computing task, so as to realize the cross-cluster scheduling of the workflow.

[0049] In the above step S102, the engine server can also receive a user's input for changing the task, which can be adding a computing task or updating a configuration file corresponding to a computing task. The engine server adjusts the workflow according to the change input, replaces the previous configuration file with the changed configuration file, and executes the task scheduling method in this specification according to the changed configuration file.

[0050] In the above step S104, the designated memory access area may also be the internal storage area of ​​each computing cluster. In another embodiment of the present specification, the engine server copies multiple copies of the configuration file of the computing task and sends the configuration file of the computing task to the internal storage area of ​​each computing cluster respectively. When the computing task is assigned to a computing cluster for execution, the computing cluster assigned to the computing task can read the configuration file of the computing task in the internal storage area.

[0051] In one or more embodiments of the present specification, if there is at least one computing task among each computing task whose execution requires corresponding input data, the user can submit the source data file required by the computing task that requires input data in the workflow through the engine server, and the engine server stores the source data file in the designated memory access area so that the computing cluster can obtain the input data of the at least one computing task when executing the at least one computing task.

[0052] When the designated storage area is shared storage, the engine server uploads the source data file to the shared storage.

[0053] In one embodiment, the storage location of the source data file of the at least one computing task in the shared storage can be specified by the user, and the storage location of the source data file of the at least one computing task in the shared storage is included in a configuration file written by the user. When the target cluster executes the at least one computing task, the storage location of the source data file in the shared storage is determined by reading the configuration file. Thus, the target cluster can obtain the source data file in the shared storage.

[0054] In another embodiment, the storage location of the source data file in the shared storage can be randomly determined, and the storage directory of the source data file in the shared storage is obtained by the engine server. When the target cluster is assigned to a computing task that requires input data, the engine server sends the storage directory of the source data file of the computing task in the shared storage to the target cluster, and the target cluster can read the source data file in the shared storage according to the storage directory to execute the computing task.

[0055] In one or more embodiments of the present specification, the configuration file of the computing task also includes a result storage directory of the computing result of the computing task in the shared storage. Then in the above step S110, after the target cluster completes the computing task, the result storage directory of the computing result of the computing task in the shared storage is determined according to the configuration file of the computing task, and the computing result of the computing task is stored in the shared storage according to the result storage directory.

[0056] After the computing task is completed, the target cluster also needs to return the computing results to the engine server so that the engine server can determine that the computing task is completed. The engine server can also store the computing results of the computing task or perform further data processing operations.

[0057] In one or more embodiments of the present specification, when the calculation result of the calculation task is numerical data, the target cluster returns the calculation result itself to the engine server. When the calculation result of the calculation task is file data, the target cluster returns the result storage directory of the calculation result in the shared storage to the engine server.

[0058] Because in high-throughput computing, the calculation results of file data types often have large data volumes, such as microstructure simulation model files in the materials field, gene sequence files in the biological field, etc. If the file data is directly transmitted to the engine server, it will consume a lot of bandwidth and take a long time.

[0059] Through this embodiment, when the calculation result of the calculation task is file data, the target cluster first stores the calculation result in the shared storage. In this way, when the engine server needs to obtain the calculation result of the calculation task, it can read the calculation result of the calculation task in the shared storage according to the result storage directory of the calculation result in the shared storage.

[0060] Furthermore, this manual Figure 1 The scheduling system shown may also include a database. After the target cluster completes the computing task, the computing result of the computing task may be stored in the database. The computing cluster may store the computing result of each computing task in the result database, or store the result storage directory of the computing result of the file data type in the shared storage in the result database, and store the computing result of the numerical type in the result database, so as to reduce the amount of storage resources consumed by the computing result in the database.

[0061] In one embodiment of the present specification, the present specification Figure 1 The scheduling system shown may also include a management module. The engine server may create an execution queue for each computing task according to the workflow, monitor the execution status of each computing task, and record the execution status of each computing task in the execution queue in real time and store it in the management module, so that the user can know the execution status of the computing task in the workflow at any time by checking the records of the management module.

[0062] When a computing task in the workflow fails to execute, the target cluster of the computing task returns the error reason to the engine server. Specifically, the target cluster can return the error information of the execution software or operating system during the execution of the computing task to the engine server in real time so that the engine server can notify the user in time. At the same time, the target cluster can also store the error information in the management module for users to view at any time.

[0063] In order to facilitate users to view the computing task workflows submitted to the scheduling system multiple times, the management module can also record the workflow determined based on the computing tasks submitted by the user each time, as well as the computing tasks and subtasks of each computing task in the workflow. The subtasks of a computing task are relatively independent execution steps included in the execution of the computing task. The engine server can monitor the execution status of each subtask and notify the user of the execution status of the subtask in a visual way, so that the user can view the execution progress of the workflow in a more real-time manner.

[0064] The engine server can divide computing tasks and subtasks into task tables and subtask tables for storage. The task table records the execution status of computing tasks, task titles set by users, and error reasons when tasks fail to execute. The subtask table records the execution status of subtasks, task titles set by users, and error reasons when tasks fail to execute.

[0065] Among them, for a computing task including multiple subtasks, when the engine server determines that all subtasks of the computing task have been executed, the execution status of the computing task is updated to the execution completion status.

[0066] The scheduling system in this specification can be deployed on a high-throughput computing engine. Figure 2 Schematic diagram of a computing engine provided in the embodiments of this specification. Figure 2 As shown, the computing engine is configured with an engine server, a computing cluster, a shared storage and a management module. The computing cluster includes two different computing sub-clusters: a high-performance computing cluster or a cloud server cluster.

[0067] The engine server can receive computing tasks submitted by users. The computing engine can schedule the computing tasks submitted by users to high-performance computing clusters or cloud server clusters in sequence according to the execution order indicated by the workflow, thereby realizing the automatic scheduling and execution of computing tasks in the workflow.

[0068] The engine server stores the configuration files of each computing task uploaded by the user to the shared storage, so that different computing clusters can access the shared storage to realize data interaction between different computing tasks. The management module can store the execution status of computing tasks or subtasks, the task title set by the user, the error reason when the task execution fails, etc.

[0069] exist Figure 2 In the computing engine shown, each computing task can be automatically scheduled according to the execution order based on the workflow, and cross-cluster scheduling of different types of computing tasks in the same workflow can be achieved through a high-performance computing cluster or a cloud server cluster. The scheduling of computing tasks is more flexible, and computing resources are reasonably allocated according to the characteristics of the computing tasks, which speeds up the computing efficiency.

[0070] Figure 3 This is a material calculation application scenario diagram provided in the embodiments of this specification. Figure 3 As shown in the figure, the user submits multiple computing tasks to the engine server, and the engine server determines the workflow of each computing task based on the user's task scheduling information. The engine server routes and forwards the computing tasks to the computing cluster in the order of the workflow, and completes the execution of each computing task through the computing cluster.

[0071] Figure 3The workflow shown contains two computing tasks, computing task 1 is parameterized structure generation, and computing task 2 is microstructure simulation. Parameterized structure generation is a script execution task, which is suitable for cloud server execution. The engine server will assign computing task 1 to the cloud server cluster for execution according to the task type indicated in the configuration file of computing task 1. Computing task 2 is a simulation software execution task with a huge amount of calculation, which is suitable for high-performance computing cluster execution. The engine server will assign computing task 2 to the high-performance computing cluster for execution according to the task type indicated in the configuration file of computing task 2.

[0072] Computation Task 1 and Computation Task 2 have a data dependency relationship, and the input of Computation Task 2 is the output of Computation Task 1. The engine server monitors the execution status of Computation Task 1, and starts the task scheduling of Computation Task 2 after the execution of Computation Task 1 is completed.

[0073] like Figure 3 As shown in the figure, the calculation result of calculation task 1 is of STL file type. After the calculation of calculation task 1 is completed, the cloud server cluster stores the calculation result of calculation task 1 in shared storage and writes the storage location of the calculation result of calculation task 1 in shared storage to the database. The calculation result of calculation task 2 is microstructure property, which is of numerical type. The high-performance computing cluster writes the calculation result of calculation task 2 to the database.

[0074] This manual also provides Figure 1 The corresponding computing task scheduling method is applied to the engine server, which is deployed on any login node included in multiple computing clusters, such as Figure 4 shown.

[0075] Figure 4 A flowchart of a computing task scheduling method provided in this specification includes: S200: Receive multiple computing tasks submitted by a user, task scheduling information, and a configuration file of each computing task, wherein the configuration file includes a task type.

[0076] S202: Determine the workflow of the plurality of computing tasks according to the task scheduling information.

[0077] S204: Send the configuration file of the computing task to the designated memory access area, and assign each computing task included in the workflow to a target cluster in turn according to the task type indicated by the configuration file of the computing task, so that the target cluster reads the configuration file of the computing task in the designated memory access area to execute the computing task, and the target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

[0078] Figure 4The specific contents of the corresponding steps S200 to S204 can refer to the above Figure 1 The description is not repeated in this manual.

[0079] This manual also provides Figure 1 The corresponding computing task scheduling method is applied to the target cluster, which is the engine server based on Figure 4 The computing task scheduling method shown in the figure sequentially determines a computing cluster that is adapted to the task type indicated by the configuration file of the computing task for each computing task included in the workflow determined by the multiple computing tasks submitted by the user. The method is as follows Figure 5 shown.

[0080] Figure 5 A flowchart of a computing task scheduling method provided in this specification includes: S300: Receive the computing task assigned by the engine server.

[0081] S302: Reading a configuration file of the computing task in a designated memory access area and executing the computing task, wherein the designated memory access area includes the configuration file of the computing task uploaded by the engine server.

[0082] S304: Return the calculation result of the calculation task to the engine server.

[0083] Figure 5 The specific contents of the corresponding steps S300 to S304 can refer to the above Figure 1 The description is not repeated in this manual.

[0084] The above is the computing task scheduling method provided in this specification. Based on the same idea, this specification also provides a corresponding computing task scheduling device, such as Figure 6 shown.

[0085] Figure 6 This is a schematic diagram of a computing task scheduling device provided in this specification. The device is applied to an engine server, and the engine server is deployed on any login node included in multiple computing clusters. The device includes: The task receiving module 400 is used to receive multiple computing tasks submitted by a user, task scheduling information and configuration files of each computing task, wherein the configuration files include task types; A task scheduling module 402, configured to determine the workflow of the plurality of computing tasks according to the task scheduling information; The scheduling module 404 is used to assign each computing task included in the workflow to a target cluster in turn according to the task type indicated by the configuration file of the computing task, and send the configuration file of the computing task to a designated memory access area of ​​the target cluster, so that the target cluster reads the configuration file of the computing task in the designated memory access area to execute the computing task. The target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

[0086] Optionally, the scheduling module 404 is specifically configured to upload the configuration files of the computing tasks to a shared storage, and the multiple computing clusters have read and write permissions to the shared storage.

[0087] Optionally, execution of at least one computing task among the computing tasks requires input of a source data file, and the task receiving module 400 is specifically used to obtain the source data file submitted by the user and upload the source data file to the shared storage.

[0088] This specification also provides another computing task scheduling device, such as Figure 7 As shown, Figure 7 This is another schematic diagram of a computing task scheduling device provided in this specification. The device is applied to a target cluster, which is an engine server based on Figure 2 The computing task scheduling method shown in the figure sequentially determines a computing cluster adapted to the task type indicated by the configuration file of the computing task for each computing task included in the workflow determined by the multiple computing tasks submitted by the user, and the device includes: The task receiving module 500 is used to receive the computing task assigned by the engine server; An execution module 502 is used to read the configuration file of the computing task in the specified memory access area and execute the computing task; The result returning module 504 is used to return the calculation result of the calculation task to the engine server.

[0089] Optionally, the execution of the computing task requires a source data file, and the configuration file of the computing task also includes a source file storage directory. The configuration file is stored in a shared storage. The execution module 502 is specifically used to read the configuration file of the computing task, determine the source file storage directory, obtain the source data file of the computing task under the source file storage directory of the shared storage, and execute the computing task according to the source data file of the computing task.

[0090] Optionally, the configuration file of the computing task also includes a result storage directory of the computing result of the computing task in the shared storage. The result return module 504 is specifically used to determine the computing result of the computing task, and determine the result storage directory of the computing result of the computing task in the shared storage according to the configuration file of the computing task. According to the result storage directory, the computing result is stored in the shared storage. If the computing result is numerical data, the computing result is returned to the engine server; if the computing result is file data, the result storage directory is returned to the engine server.

[0091] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides a computing task scheduling method.

[0092] This manual also provides Figure 8 The schematic structure diagram of the electronic device shown in FIG. Figure 8 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0093] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0094] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0095] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0096] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0103] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0107] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0108] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A computing task scheduling method, characterized in that: Applied to an engine server, the engine server is deployed on any login node included in multiple computing clusters, and the method includes: Receive multiple computing tasks submitted by a user, task scheduling information, and a configuration file of each computing task, wherein the configuration file includes a task type; Determining the workflow of the plurality of computing tasks according to the task scheduling information; The configuration file of the computing task is sent to the designated memory access area, and for each computing task included in the workflow, the computing task is assigned to the target cluster in turn according to the task type indicated by the configuration file of the computing task, so that the target cluster reads the configuration file of the computing task in the designated memory access area to execute the computing task, and the target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

2. The method according to claim 1, characterized in that Sending the configuration file of the computing task to the designated memory access area of ​​the target cluster specifically includes: The configuration files of the computing tasks are uploaded to a shared storage, and the multiple computing clusters have read and write permissions to the shared storage.

3. The method according to claim 2, characterized in that The execution of at least one computing task among the computing tasks requires input of a source data file; The method further comprises: The source data file submitted by the user is obtained, and the source data file is uploaded to the shared storage.

4. A computing task scheduling method, characterized in that: Applied to a target cluster, the target cluster being a computing cluster adapted to a task type indicated by a configuration file of a computing task, determined by the engine server according to any one of claims 1 to 3, for each computing task included in a workflow determined by a plurality of computing tasks submitted by a user, the method comprising: Receiving the computing task assigned by the engine server; In a designated memory access area, read a configuration file of the computing task and execute the computing task, wherein the designated memory access area includes the configuration file of the computing task uploaded by the engine server; The calculation result of the calculation task is returned to the engine server.

5. The method according to claim 4, characterized in that The execution of the computing task requires a source data file, and the configuration file of the computing task also includes a source file storage directory, and the configuration file is stored in a shared storage; Execute the computing task according to the configuration file of the computing task, specifically including: Read the configuration file of the computing task and determine the source file storage directory; Obtaining the source data file of the computing task from the source file storage directory of the shared storage; Execute the computing task according to the source data file of the computing task.

6. The method according to claim 5, characterized in that The configuration file of the computing task also includes a result storage directory of the computing result of the computing task in the shared storage; Return the calculation result of the calculation task to the engine server, specifically including: Determine a calculation result of the calculation task, and determine a result storage directory of the calculation result of the calculation task in the shared storage according to a configuration file of the calculation task; According to the result storage directory, storing the calculation result in the shared storage; If the calculation result is numerical data, the calculation result is returned to the engine server; if the calculation result is file data, the result storage directory is returned to the engine server.

7. A high-throughput computing task scheduling device, characterized in that: Applied to an engine server, the engine server is deployed on any login node included in multiple computing clusters, and the device includes: A task receiving module receives multiple computing tasks submitted by a user, task scheduling information, and a configuration file of each computing task, wherein the configuration file includes a task type; A task scheduling module, which determines the workflow of the plurality of computing tasks according to the task scheduling information; The scheduling module sequentially assigns each computing task included in the workflow to a target cluster according to the task type indicated by the configuration file of the computing task, and sends the configuration file of the computing task to a designated memory access area of ​​the target cluster, so that the target cluster reads the configuration file of the computing task in the designated memory access area to execute the computing task. The target cluster is a computing cluster adapted to the task type indicated by the configuration file of the computing task.

8. A high-throughput computing task scheduling device, characterized in that: Applied to a target cluster, the target cluster is a computing cluster adapted to the task type indicated by a configuration file of the computing task, determined by the engine server according to any one of claims 1 to 3, for each computing task included in a workflow determined by multiple computing tasks submitted by a user, the device comprising: A task receiving module receives the computing task assigned by the engine server; The execution module reads the configuration file of the computing task in the specified memory access area and executes the computing task; The result returning module returns the calculation result of the calculation task to the engine server.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images