Task scheduling method and device, task scheduling system and medium

Abstract

The invention discloses a task scheduling method and device, a task scheduling system and a medium, and the method comprises the steps: obtaining a plurality of sub-tasks through a first module, and enabling the plurality of sub-tasks to have an execution sequence; the subtasks are sequentially stored in first middleware according to the sequential execution sequence through the first module, and the subtasks in the first middleware are stored in a disk; distributing the current subtask stored in the first middleware to an agent module through the first middleware; and executing the current subtask through the agent module. According to the method, the first middleware stores the subtasks on the disk, and the capacity of the disk can be expanded according to actual conditions, so that sufficient storage resources can be provided for task scheduling, support is provided for realizing high-throughput data processing, and the processing performance of task scheduling is improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing, and in particular to a task scheduling method, device, task scheduling system and medium. Background Art

[0002] In big data development, task scheduling is a critical capability. Task scheduling mainly involves splitting the task process into different subtasks to achieve the purpose of orderly execution of these subtasks.

[0003] Currently, mainstream task scheduling systems are all developed at a low level. Task scheduling uses native queues to cache and distribute tasks. The subtasks in the queue are stored in memory, occupying memory storage capacity. Due to the limited capacity of memory, when the number of subtasks is large, the above task scheduling method may cause the system to crash. Summary of the invention

[0004] The present application provides a task scheduling method, device, task scheduling system and medium, which are used to solve the problem that the existing scheduling method requires a large amount of memory and has low processing performance when the number of subtasks is large.

[0005] In a first aspect, the present application provides a task scheduling method, comprising:

[0006] Acquire multiple subtasks through the first module, and the multiple subtasks have a sequential execution order;

[0007] The plurality of subtasks are sequentially stored in the first middleware according to the execution order through the first module, and the subtasks in the first middleware are stored in the disk;

[0008] Distributing the current subtask stored in the first middleware to the proxy module through the first middleware;

[0009] The current subtask is executed by the proxy module.

[0010] In an embodiment of the present application, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware, which can reduce the coupling degree of data transmission between the first module and the proxy module. The first middleware stores the subtask on the disk. Since the capacity of the disk can be expanded according to actual conditions, sufficient storage resources can be provided for task scheduling, and support can be provided for high-throughput data processing, thereby improving the processing performance of task scheduling.

[0011] In an embodiment of the present application, the acquiring of multiple subtasks through the first module includes:

[0012] Receiving a first input, the first input is used to select a process template, the process template includes a plurality of task nodes for implementing a target task;

[0013] A process instance is generated based on the process template through the first module, and the process instance includes a subtask corresponding to each task node.

[0014] In the embodiment of the present application, the user can set the process template according to the actual situation, and the first module generates a process instance based on the process template selected by the user to obtain multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, realizing batch processing of process instances and improving the generation efficiency of process instances.

[0015] In an embodiment of the present application, the first middleware includes an executor and a message queue;

[0016] The distributing the current subtask stored in the first middleware to the proxy module through the first middleware includes:

[0017] Sending task information of the current subtask stored in the message queue to the proxy module through the executor;

[0018] The current subtask is obtained from the message queue through the proxy module.

[0019] In an embodiment of the present application, the executor sends the task information of the current subtask to the proxy module, and the proxy module can obtain the current subtask from the message queue according to actual conditions, such as when the proxy module is idle, to avoid crashing due to high operating pressure on the proxy module.

[0020] In one embodiment of the present application, there are multiple proxy modules;

[0021] The step of sending the task information of the current subtask stored in the message queue to the proxy module by the executor includes:

[0022] Sending task information of the current subtask stored in the message queue to the plurality of proxy modules through the executor;

[0023] The obtaining the current subtask from the message queue through the proxy module includes:

[0024] Obtaining the current subtask from the message queue through a first proxy module among the plurality of proxy modules, and adding the current subtask to a shared queue;

[0025] The executing the current subtask through the proxy module includes:

[0026] When a second proxy module among the plurality of proxy modules acquires a distributed lock, taking out the current subtask from the shared queue through the second proxy module;

[0027] The current subtask is executed through the second proxy module, and the execution result of the current subtask is fed back to the executor.

[0028] In an embodiment of the present application, multiple proxy modules process subtasks in a shared queue, and use distributed locks to ensure that only one proxy module can obtain subtasks from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different proxy modules, resulting in repeated execution of subtasks.

[0029] In an embodiment of the present application, executing the current subtask by the second agent module includes:

[0030] Encapsulating the current subtask according to the task type of the current subtask by the second proxy module to obtain an encapsulation command;

[0031] The target module is called according to the encapsulation command by the second proxy module to execute the current subtask.

[0032] In the embodiment of the present application, the current subtask may be processed differently based on different task types to optimize the processing of the current subtask.

[0033] In an embodiment of the present application, the process of the target module executing the current subtask includes:

[0034] In the case where the task type is the first type, the target module executes the current subtask locally;

[0035] When the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0036] In an embodiment of the present application, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution to optimize the processing of the current subtask.

[0037] In an embodiment of the present application, the process of the target module executing the current subtask includes:

[0038] The shared variables obtained during the execution of the current subtask and the execution status of the current subtask are stored in the second middleware through the target module.

[0039] In an embodiment of the present application, the shared variables and execution status obtained during the execution of the current subtask are stored in the second middleware. The first module can obtain the execution status of the current subtask from the second middleware as needed, thereby realizing the transmission of shared variables between multiple subtasks of the target task, and also facilitating other modules to monitor the execution status of the current subtask.

[0040] In a second aspect, an embodiment of the present application provides a task scheduling device, including:

[0041] An acquisition module, used for acquiring a plurality of subtasks through the first module, wherein the plurality of subtasks have a sequential execution order;

[0042] A storage module, used for storing the plurality of subtasks in the first middleware in the execution order through the first module, wherein the subtasks in the first middleware are stored in a disk;

[0043] a distribution module, configured to distribute the current subtask stored in the first middleware to the proxy module through the first middleware;

[0044] An execution module is used to execute the current subtask through the proxy module.

[0045] In an embodiment of the present application, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware, which can reduce the coupling degree of data transmission between the first module and the proxy module. The first middleware stores the subtask on the disk. Since the capacity of the disk can be expanded according to actual conditions, sufficient storage resources can be provided for task scheduling, and support can be provided for high-throughput data processing, thereby improving the processing performance of task scheduling.

[0046] In an embodiment of the present application, the acquisition module includes:

[0047] A receiving module, configured to receive a first input, wherein the first input is used to select a process template, wherein the process template includes a plurality of task nodes for implementing a target task;

[0048] A generation module is used to generate a process instance based on the process template through the first module, and the process instance includes a subtask corresponding to each task node.

[0049] In the embodiment of the present application, the user can set the process template according to the actual situation, and the first module generates a process instance based on the process template selected by the user to obtain multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, realizing batch processing of process instances and improving the generation efficiency of process instances.

[0050] In an embodiment of the present application, the first middleware includes an executor and a message queue;

[0051] The distribution module comprises:

[0052] A sending submodule, used for sending the task information of the current subtask stored in the message queue to the proxy module through the executor;

[0053] The first acquisition submodule is used to acquire the current subtask from the message queue through the proxy module.

[0054] In an embodiment of the present application, the executor sends the task information of the current subtask to the proxy module, and the proxy module can obtain the current subtask from the message queue according to actual conditions, such as when the proxy module is idle, to avoid crashing due to high operating pressure on the proxy module.

[0055] In one embodiment of the present application, there are multiple proxy modules;

[0056] The sending submodule comprises:

[0057] A sending unit, configured to send the task information of the current subtask stored in the message queue to the plurality of proxy modules through the executor;

[0058] The acquisition submodule includes:

[0059] an acquiring unit, configured to acquire the current subtask from the message queue through a first agent module among the plurality of agent modules;

[0060] an adding unit, configured to add the current subtask to a shared queue through the first proxy module;

[0061] The execution module comprises:

[0062] A second acquisition submodule, configured to, when a second proxy module among the plurality of proxy modules acquires a distributed lock, take out the current subtask from the shared queue through the second proxy module;

[0063] an execution submodule, configured to execute the current subtask through the second agent module;

[0064] A feedback submodule is used to feed back the execution result of the current subtask to the executor through the second agent module.

[0065] In an embodiment of the present application, multiple proxy modules process subtasks in a shared queue, and use distributed locks to ensure that only one proxy module can obtain subtasks from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different proxy modules, resulting in repeated execution of subtasks.

[0066] In an embodiment of the present application, the execution module includes:

[0067] A packaging submodule, used for packaging the current subtask according to the task type of the current subtask through the second proxy module to obtain a packaging command;

[0068] An execution submodule is used to call a target module according to the encapsulation command through the second proxy module to execute the current subtask.

[0069] In the embodiment of the present application, the current subtask may be processed differently based on different task types to optimize the processing of the current subtask.

[0070] In an embodiment of the present application, the process of the target module executing the current subtask includes:

[0071] In the case where the task type is the first type, the target module executes the current subtask locally;

[0072] When the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0073] In an embodiment of the present application, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution to optimize the processing of the current subtask.

[0074] In an embodiment of the present application, the process of the target module executing the current subtask includes:

[0075] The shared variables obtained during the execution of the current subtask and the execution status of the current subtask are stored in the second middleware through the target module.

[0076] In an embodiment of the present application, the shared variables and execution status obtained during the execution of the current subtask are stored in the second middleware. The first module can obtain the execution status of the current subtask from the second middleware as needed, thereby realizing the transmission of shared variables between multiple subtasks of the target task, and also facilitating other modules to monitor the execution status of the current subtask.

[0077] In a third aspect, an embodiment of the present application provides a task scheduling system, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the task scheduling method described in the first aspect.

[0078] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the task scheduling method described in the first aspect are implemented.

[0079] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The features, advantages and technical effects of exemplary embodiments of the present application will be described below with reference to the accompanying drawings.

[0081] Figure 1 A flowchart of a task scheduling method provided in an embodiment of the present application;

[0082] Figure 2 A schematic diagram of the structure of a task scheduling system provided in an embodiment of the present application;

[0083] Figure 3 Another flowchart of the task scheduling method provided in the embodiment of the present application;

[0084] Figure 4 A schematic diagram of the structure of a task scheduling device provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of the hardware structure of the task scheduling system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0087] Unless otherwise defined, all technical and scientific terms used in this application have the same meanings as those commonly understood by technicians in the technical field of this application; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order or a primary and secondary relationship.

[0088] The following will introduce the task scheduling method, device, task scheduling system and medium provided by the embodiment of the present application in detail with reference to the accompanying drawings. The task scheduling method disclosed in the embodiment of the present application can be applied to the technical field of distributed task scheduling system.

[0089] Figure 1 A flowchart of a task scheduling method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method is applied to a task scheduling system, and the method includes steps 101 to 104, wherein:

[0090] Step 101: acquiring a plurality of subtasks through a first module, wherein the plurality of subtasks have a sequential execution order.

[0091] The task scheduling system includes a first module, a first middleware and a proxy module, and each component cooperates with each other to complete task scheduling. Among them, the first module can also be called a core module, which is used to obtain multiple subtasks, which are subtasks of the same task, and there is a sequence of execution between the multiple subtasks. For example, the first module generates a corresponding directed graph based on the flowchart drawn by the front-end user, and the in-degree of the process start node is 0, and the out-degree of the end node is 0. From the task start node, the next task node is parsed in turn, and each task node can be encapsulated into a corresponding subtask.

[0092] The information of each subtask, such as the parameters of the subtask, the execution order of the subtasks, etc., can be stored in the database of the task scheduling system.

[0093] Step 102: The plurality of subtasks are sequentially stored in the first middleware according to the execution sequence through the first module, and the subtasks in the first middleware are stored in a disk.

[0094] The first module obtains information of each subtask from the database, and stores the multiple subtasks in the first middleware in sequence according to the execution order of the multiple subtasks. The first middleware is also called message middleware.

[0095] Step 103: distribute the current subtask stored in the first middleware to the proxy module through the first middleware.

[0096] For example, the first middleware may include an executor and a message queue. The subtask sent by the first module to the first middleware is stored in the message queue. The executor sends the task information of the subtask received by the first middleware to the proxy module. After the proxy module receives the task information of the subtask, it obtains the subtask from the message queue of the first middleware.

[0097] Step 104: Execute the current subtask through the proxy module.

[0098] After acquiring the current subtask, the proxy module executes the current subtask, obtains the execution result, and feeds back the execution result to the executor. The executor further feeds back the execution result to the first module, and the first module stores the execution result in the database of the task scheduling system. The execution result can be the execution completion status of the current subtask, for example, the execution has been completed or is not completed. The execution result can also be the running result of the current subtask. The first module can determine whether to send the subtask whose execution order is after the current subtask to the first middleware according to the execution result of the current subtask.

[0099] In this embodiment, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware, which can reduce the coupling degree of data transmission between the first module and the proxy module. Since the proxy module needs to be responsible for the execution of the subtask, the processing efficiency of the proxy module is lower than that of the first module. Storing the subtask through the first middleware can alleviate the problem of inconsistent processing efficiency between the first module and the proxy module when processing subtasks. In addition, the first middleware stores the subtask on the disk. Since the capacity of the disk can be expanded according to actual conditions, it can provide sufficient storage resources for task scheduling and provide support for high-throughput data processing, thereby improving the processing performance of task scheduling.

[0100] In an embodiment of the present application, the acquiring of multiple subtasks through the first module includes:

[0101] Receiving a first input, the first input is used to select a process template, the process template includes a plurality of task nodes for implementing a target task;

[0102] A process instance is generated based on the process template through the first module, and the process instance includes a subtask corresponding to each task node.

[0103] Specifically, the user selects the process template by dragging and dropping in the visual interface, and sets the corresponding template parameters. The process template can be preset by the user. The process template can be understood as an abstract model that defines a series of steps, conditions and rules. The process template can complete complex tasks composed of multiple steps.

[0104] After selecting a process template, you also need to configure parameters for the process template. The parameters can be configured by the user or by default. There is no limitation here. For example, the process template is used to calculate the oxidation potential of a molecule. The input parameter is the molecule. You can also configure the environmental parameters for the subtask operation. Different parameters in the input value process template can result in different process instances. For example, if you input 1,000 different molecules into the process module, you can get 1,000 process instances.

[0105] The process template includes multiple task nodes for achieving the target task. These multiple task nodes constitute a directed graph, and the in-degree of the start node is 0 and the out-degree of the end node is 0. The first module parses each task node in turn from the task start node, and encapsulates each task node into a corresponding subtask. According to the directed graph, the dependency relationship between each task node can be obtained, thereby obtaining the execution order of the subtasks corresponding to the task node.

[0106] In this embodiment, the user can set the process template according to the actual situation, and the first module generates a process instance based on the process template selected by the user to obtain multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, realizing batch processing of process instances and improving the generation efficiency of process instances.

[0107] In yet another embodiment of the present application, the first middleware includes an executor and a message queue;

[0108] The distributing the current subtask stored in the first middleware to the proxy module through the first middleware includes:

[0109] Sending task information of the current subtask stored in the message queue to the proxy module through the executor;

[0110] The current subtask is obtained from the message queue through the proxy module.

[0111] Specifically, the first middleware may include multiple executors. The first module sends the subtasks in the process instance to different executors in sequence according to the execution order. These executors are responsible for notifying the agent module of the task information of the subtasks, monitoring the task completion status, and implementing a task failure retry mechanism.

[0112] In this embodiment, the executor sends the task information of the current subtask to the proxy module, and the proxy module can obtain the current subtask from the message queue according to actual conditions, such as when the proxy module is idle, to avoid crashing due to high operating pressure of the proxy module.

[0113] In yet another embodiment of the present application, there are multiple proxy modules;

[0114] The step of sending the task information of the current subtask stored in the message queue to the proxy module by the executor includes:

[0115] Sending task information of the current subtask stored in the message queue to the plurality of proxy modules through the executor;

[0116] The obtaining the current subtask from the message queue through the proxy module includes:

[0117] Obtaining the current subtask from the message queue through a first proxy module among the plurality of proxy modules, and adding the current subtask to a shared queue;

[0118] The executing the current subtask through the proxy module includes:

[0119] When a second proxy module among the plurality of proxy modules acquires a distributed lock, taking out the current subtask from the shared queue through the second proxy module;

[0120] The current subtask is executed through the second proxy module, and the execution result of the current subtask is fed back to the executor.

[0121] In this embodiment, there are multiple proxy modules. In a distributed system, one proxy module can be set on one device, and multiple proxy modules on multiple devices process multiple subtasks in the first middleware. The executor sends the task information of the current subtask to each proxy module. After these proxy modules obtain the task information, the first proxy module takes out the current subtask from the message queue and adds the current subtask to the shared queue. The first proxy module can be the proxy module that takes out the current subtask earliest among the multiple proxy modules.

[0122] A shared queue is a queue shared by multiple proxy modules. When each proxy module executes a subtask, it takes the subtask from the shared queue. When taking the subtask, in order to ensure data consistency in a distributed environment, only one proxy module needs to obtain exclusive access to the shared queue at the same time. In this embodiment, a distributed lock is set, and the proxy module that obtains the distributed lock is called the second proxy module. The second proxy module takes the current subtask from the shared queue and executes the current subtask.

[0123] In this embodiment, multiple proxy modules process subtasks in a shared queue, and use distributed locks to ensure that only one proxy module can obtain subtasks from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different proxy modules, resulting in repeated execution of subtasks.

[0124] In yet another embodiment of the present application, executing the current subtask by the second agent module includes:

[0125] Encapsulating the current subtask according to the task type of the current subtask by the second proxy module to obtain an encapsulation command;

[0126] The target module is called according to the encapsulation command by the second proxy module to execute the current subtask.

[0127] For example, the second proxy module, according to the task type of the current subtask, is installed as different Python commands to call the target module, and the target module executes the current subtask and releases the distributed lock after the execution is completed.

[0128] The task type may include a first type and a second type. The first type may be a common task type, which generally refers to tasks with small computational complexity and low resource consumption. The second type may be a high-performance task type, which generally refers to tasks with large resource consumption and long execution time.

[0129] The encapsulation command may include the task type of the current subtask. When the target module is called according to the encapsulation command, the task type of the current subtask in the encapsulation command may be passed to the target module, and the target module executes the current subtask based on different task types.

[0130] In this example, the current subtask may be processed differently based on different task types to optimize the processing of the current subtask.

[0131] In another embodiment of the present application, the process of the target module executing the current subtask includes:

[0132] In the case where the task type is the first type, the target module executes the current subtask locally;

[0133] When the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0134] Specifically, the target module executes the current subtask according to the task type. If the task type is the first type, for example, a common task type, the target module executes the current subtask locally; if the task type is the second type, for example, a high-performance task type, the target module sends the current subtask to the target scheduling system, and the target scheduling coefficient executes the current subtask on the high-performance computer. In this way, hardware resources can be allocated to each subtask as much as possible, so that the high-performance computer can give full play to its capabilities, while also improving the execution efficiency of tasks.

[0135] The target scheduling system may be a slurm task scheduling system. Through the allocation of the slurm task scheduling system and the support of high-performance computers, the execution efficiency of the current subtask can be effectively improved.

[0136] In this embodiment, the current subtask may be processed differently based on different task types. The current subtask may be executed locally or handed over to the target scheduling system for execution, so as to optimize the processing of the current subtask.

[0137] In another embodiment of the present application, the process of the target module executing the current subtask includes:

[0138] The shared variables obtained during the execution of the current subtask and the execution status of the current subtask are stored in the second middleware through the target module.

[0139] Specifically, the second middleware can be redis, and the execution status can refer to the status of the current subtask being suspended, running, completed, or error. Storing the shared variables and execution status obtained during the execution of the current subtask in the second middleware can enable the shared variables to be transferred between multiple subtasks of the target task, and also facilitate other modules to monitor the execution status of the current subtask. For example, the first module can obtain the execution status of the current subtask from the second middleware as required.

[0140] The task scheduling method implemented in the embodiment of the present application can be based on the current mainstream distributed scheduling system, and the message middleware (i.e., the first middleware) can be introduced to improve it and increase the high concurrency capability of the system. The workflow template is imported by the user, and the parameters of the tasks of each node in the template can be configured. The first module constructs an executable process instance through the workflow template, and then assigns the subtasks in the process instance to different executors in turn according to the node relationship. These executors are responsible for notifying the agent module of the task and monitoring the completion of the task and performing a task failure retry mechanism. After receiving the task, the agent module executes the subtasks in turn, and uses distributed locks to resolve concurrency conflicts during execution.

[0141] The following is an example of the task scheduling method provided in the embodiment of the present application.

[0142] like Figure 2 , which is a schematic diagram of the structure of the task scheduling system provided in an embodiment of the present application.

[0143] The first module: Generates a directed graph based on the flowchart drawn by the front-end user, and the in-degree of the process start node is 0, and the out-degree of the end node is 0. The next task node is parsed from the task start node in turn, and encapsulated into different subtasks according to the structure of the graph, and then sent to the agent module through the kafka module. The first module is also responsible for the retry function of subtask failure.

[0144] Agent module: It is mainly used to obtain subtasks and encapsulate the corresponding environment, and then call pythonWrapper to execute specific subtasks and ensure the ACID characteristics of transactions when executing tasks. Since the agent module serves clusters and uses multi-threaded task submission, a distributed lock is also introduced to ensure that only one node can execute the task at the same time to avoid the problem of repeated execution of tasks.

[0145] The target module, also known as the pythonWrapper module, is mainly responsible for the execution of specific subtasks. If it is a high-performance task, the task is submitted to the slurm scheduling system through a remote connection for execution; if it is an ordinary task, it is executed directly locally.

[0146] Message middleware includes:

[0147] The first middleware is implemented using Kafka: it is mainly responsible for subtask caching and system decoupling.

[0148] Redis: Mainly provides publish-subscribe mode, responsible for notification of thread suspension and resumption.

[0149] The database includes:

[0150] The second middleware uses the Redis database: it mainly stores some intermediate shared variables and the execution status of subtasks. The status of the distributed lock is also placed in the second middleware. For example, the distributed lock in the second middleware is identified as 0, indicating that the distributed lock is in an idle state. The distributed lock is identified as 1, indicating that the distributed lock is in use.

[0151] The first database uses a Mongo database: it mainly stores workflow node information (ie, subtask information) and workflow instances of process templates, for example, the execution order of each subtask.

[0152] Slurm scheduler: responsible for the scheduling and execution of high-performance tasks and the rational allocation of resources, so that these tasks can be efficiently executed on high-performance computers.

[0153] like Figure 3 , which is a flowchart of a task scheduling method provided in an embodiment of the present application.

[0154] Step 301, receiving user input, the user generates a process template by dragging and dropping in the visual interface, and sets the corresponding system parameters, and the user clicks to execute the operation.

[0155] Step 302: When the process template is clicked to execute, the first module generates a process instance according to the transmitted process template.

[0156] Step 303: The first module sends the subtasks of the process instance to the proxy module in sequence. For example, the first module applies the strategy pattern, the observer pattern, and the responsibility chain design pattern to distribute each specific subtask in the process instance to the proxy module through Kafka in sequence. At the same time, the first module can also query the execution status of each process instance.

[0157] Step 304: After the proxy module acquires the distributed lock, it takes out the subtask from the shared queue for execution and releases the distributed lock after completion. Specifically, after the proxy module receives the corresponding task message, it mainly goes through the following steps:

[0158] First add the task to the shared queue.

[0159] Multiple proxy modules cyclically acquire the distributed lock and determine whether there are executable tasks in the shared queue.

[0160] If there is a task to be executed, it is encapsulated into different python commands according to different task types to call pythonWrapper to execute the task, and the distributed lock is released after the execution is completed.

[0161] Step 305, the target module executes the subtask according to the task type of the subtask. If it is a common task type, it is directly executed locally. If it is a high-performance task type, it is handed over to the slurm scheduling system for execution on a high-performance computer. Through the allocation of slurm and the support of high-performance computers, the execution efficiency of the task can be effectively improved.

[0162] In the above, the use of message middleware and design patterns reduces the coupling degree of the system, improves the scalability of the system, and enables the system to run stably in a high-concurrency environment. At the same time, the intermediate variables of each subtask are stored on the second middleware to facilitate parameter transfer. According to different task types, they are executed locally or by the slurm scheduling system, which effectively improves the execution efficiency and resource utilization of tasks.

[0163] See also Figure 4 , is a schematic diagram of the structure of the task scheduling device provided in the embodiment of the present application, such as Figure 4 As shown, the task scheduling device 400 includes:

[0164] An acquisition module, used for acquiring a plurality of subtasks through the first module, wherein the plurality of subtasks have a sequential execution order;

[0165] A storage module, used for storing the plurality of subtasks in the first middleware in the execution order through the first module, wherein the subtasks in the first middleware are stored in a disk;

[0166] a distribution module, configured to distribute the current subtask stored in the first middleware to the proxy module through the first middleware;

[0167] An execution module is used to execute the current subtask through the proxy module.

[0168] In an embodiment of the present application, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware, which can reduce the coupling degree of data transmission between the first module and the proxy module. The first middleware stores the subtask on the disk. Since the capacity of the disk can be expanded according to actual conditions, sufficient storage resources can be provided for task scheduling, and support can be provided for high-throughput data processing, thereby improving the processing performance of task scheduling.

[0169] In an embodiment of the present application, the acquisition module includes:

[0170] A receiving module, configured to receive a first input, wherein the first input is used to select a process template, wherein the process template includes a plurality of task nodes for implementing a target task;

[0171] A generation module is used to generate a process instance based on the process template through the first module, and the process instance includes a subtask corresponding to each task node.

[0172] In the embodiment of the present application, the user can set the process template according to the actual situation, and the first module generates a process instance based on the process template selected by the user to obtain multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, realizing batch processing of process instances and improving the generation efficiency of process instances.

[0173] In an embodiment of the present application, the first middleware includes an executor and a message queue;

[0174] The distribution module comprises:

[0175] A sending submodule, used for sending the task information of the current subtask stored in the message queue to the proxy module through the executor;

[0176] The first acquisition submodule is used to acquire the current subtask from the message queue through the proxy module.

[0177] In an embodiment of the present application, the executor sends the task information of the current subtask to the proxy module, and the proxy module can obtain the current subtask from the message queue according to actual conditions, such as when the proxy module is idle, to avoid crashing due to high operating pressure on the proxy module.

[0178] In one embodiment of the present application, there are multiple proxy modules;

[0179] The sending submodule comprises:

[0180] A sending unit, configured to send the task information of the current subtask stored in the message queue to the plurality of proxy modules through the executor;

[0181] The acquisition submodule includes:

[0182] an acquiring unit, configured to acquire the current subtask from the message queue through a first agent module among the plurality of agent modules;

[0183] an adding unit, configured to add the current subtask to a shared queue through the first proxy module;

[0184] The execution module comprises:

[0185] A second acquisition submodule, configured to, when a second proxy module among the plurality of proxy modules acquires a distributed lock, take out the current subtask from the shared queue through the second proxy module;

[0186] an execution submodule, configured to execute the current subtask through the second agent module;

[0187] A feedback submodule is used to feed back the execution result of the current subtask to the executor through the second agent module.

[0188] In an embodiment of the present application, multiple proxy modules process subtasks in a shared queue, and use distributed locks to ensure that only one proxy module can obtain subtasks from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different proxy modules, resulting in repeated execution of subtasks.

[0189] In an embodiment of the present application, the execution module includes:

[0190] A packaging submodule, used for packaging the current subtask according to the task type of the current subtask through the second proxy module to obtain a packaging command;

[0191] An execution submodule is used to call a target module according to the encapsulation command through the second proxy module to execute the current subtask.

[0192] In the embodiment of the present application, the current subtask may be processed differently based on different task types to optimize the processing of the current subtask.

[0193] In an embodiment of the present application, the process of the target module executing the current subtask includes:

[0194] In the case where the task type is the first type, the target module executes the current subtask locally;

[0195] When the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0196] In an embodiment of the present application, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution to optimize the processing of the current subtask.

[0197] In an embodiment of the present application, the process of the target module executing the current subtask includes:

[0198] The shared variables obtained during the execution of the current subtask and the execution status of the current subtask are stored in the second middleware through the target module.

[0199] In an embodiment of the present application, the shared variables and execution status obtained during the execution of the current subtask are stored in the second middleware. The first module can obtain the execution status of the current subtask from the second middleware as needed, thereby realizing the transmission of shared variables between multiple subtasks of the target task, and also facilitating other modules to monitor the execution status of the current subtask.

[0200] The task scheduling device 400 provided in the embodiment of the present application can implement each process implemented in the aforementioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0201] Figure 5 A hardware structure diagram of a task scheduling system provided in an embodiment of the present application is shown.

[0202] The task scheduling system may include a processor 501 and a memory 502 storing computer program instructions.

[0203] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0204] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In some instances, the memory 402 may include a removable or non-removable (or fixed) medium, or the memory 502 is a non-volatile solid-state memory. In some embodiments, the memory 502 may be inside or outside the battery device.

[0205] In some examples, the memory 502 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0206] The memory 502 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0207] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement Figure 1 The method in the embodiment shown in the figure is achieved Figure 1 The corresponding technical effects achieved by executing the methods / steps in the example shown are not repeated here for the sake of brevity.

[0208] In one example, the task scheduling system may further include a communication interface 503 and a bus 504. Figure 5As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 504 and communicate with each other.

[0209] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0210] Bus 504 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), Hyper Transport (Hyper Transport, HT) interconnection, industry standard architecture (Industry Standard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 504 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0211] In addition, the embodiments of the present application may be implemented by providing a computer storage medium having computer program instructions stored thereon; when the computer program instructions are executed by a processor, any one of the task scheduling methods in the above embodiments is implemented.

[0212] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0213] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0214] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0215] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device, equipment and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A task scheduling method, characterized in that: include: Acquire multiple subtasks through the first module, and the multiple subtasks have a sequential execution order; The plurality of subtasks are sequentially stored in the first middleware according to the execution order through the first module, and the subtasks in the first middleware are stored in the disk; Distributing the current subtask stored in the first middleware to the proxy module through the first middleware; The current subtask is executed by the proxy module.

2. The task scheduling method according to claim 1, characterized in that: The obtaining of multiple subtasks through the first module includes: Receiving a first input, the first input is used to select a process template, the process template includes a plurality of task nodes for implementing a target task; A process instance is generated based on the process template through the first module, and the process instance includes a subtask corresponding to each task node.

3. The task scheduling method according to claim 1, characterized in that: The first middleware includes an executor and a message queue; The distributing the current subtask stored in the first middleware to the proxy module through the first middleware includes: Sending task information of the current subtask stored in the message queue to the proxy module through the executor; The current subtask is obtained from the message queue through the proxy module.

4. The task scheduling method according to claim 3, characterized in that: There are multiple proxy modules; The step of sending the task information of the current subtask stored in the message queue to the proxy module through the executor includes: Sending task information of the current subtask stored in the message queue to the plurality of proxy modules through the executor; The obtaining the current subtask from the message queue through the proxy module includes: Obtaining the current subtask from the message queue through a first proxy module among the plurality of proxy modules, and adding the current subtask to a shared queue; The executing the current subtask through the proxy module includes: When a second proxy module among the plurality of proxy modules acquires a distributed lock, taking out the current subtask from the shared queue through the second proxy module; The current subtask is executed through the second proxy module, and the execution result of the current subtask is fed back to the executor.

5. The task scheduling method according to claim 4, characterized in that: The executing the current subtask by the second agent module includes: Encapsulating the current subtask according to the task type of the current subtask by the second proxy module to obtain an encapsulation command; The target module is called according to the encapsulation command by the second proxy module to execute the current subtask.

6. The task scheduling method according to claim 5, characterized in that: The process of the target module executing the current subtask includes: In the case where the task type is the first type, the target module executes the current subtask locally; When the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

7. The task scheduling method according to claim 5, characterized in that: The process of the target module executing the current subtask includes: The shared variables obtained during the execution of the current subtask and the execution status of the current subtask are stored in the second middleware through the target module.

8. A task scheduling device, characterized in that: include: An acquisition module, used for acquiring a plurality of subtasks through the first module, wherein the plurality of subtasks have a sequential execution order; A storage module, used for storing the plurality of subtasks in the first middleware in the execution order through the first module, wherein the subtasks in the first middleware are stored in a disk; a distribution module, configured to distribute the current subtask stored in the first middleware to the proxy module through the first middleware; An execution module is used to execute the current subtask through the proxy module.

9. A task scheduling system, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the task scheduling method as claimed in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the task scheduling method according to any one of claims 1 to 7 are implemented.

Images