Task scheduling method and device, task scheduling system and medium

Abstract

The application discloses a task scheduling method and device, a task scheduling system and a medium. The method comprises the following steps: acquiring a plurality of sub-tasks through a first module, and the plurality of sub-tasks have an execution sequence; storing the plurality of sub-tasks in a first middleware in sequence through the first module according to the execution sequence, and the sub-tasks in the first middleware are stored in a disk; distributing a current sub-task stored in the first middleware to an agent module through the first middleware; and executing the current sub-task through the agent module. In the above, the first middleware stores the sub-tasks in the disk. Since the capacity of the disk can be expanded according to actual conditions, sufficient storage resources can be provided for task scheduling, support can be provided for high-throughput data processing, and the processing performance of task scheduling is improved.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to a task scheduling method, apparatus, task scheduling system and medium. Background Technology

[0002] In big data development, task scheduling is a crucial capability. Task scheduling mainly involves breaking down the task process into different sub-tasks, so as to achieve the goal of executing these sub-tasks in an orderly manner.

[0003] Currently, most mainstream task scheduling systems are developed at a lower level. Task scheduling uses native queues for task caching and distribution. Subtasks in the queue are stored in memory, occupying memory storage capacity. Due to the limited memory capacity, the above task scheduling method may cause system crashes when the number of subtasks is large. Summary of the Invention

[0004] This application provides a task scheduling method, apparatus, system, and medium to solve the problems of existing scheduling methods requiring a large amount of memory and having low processing performance when the number of subtasks is large.

[0005] Firstly, this application provides a task scheduling method, including:

[0006] The first module obtains multiple subtasks, which have a sequential execution order.

[0007] The first module stores multiple subtasks in the first middleware according to the order of execution, and the subtasks in the first middleware are stored on the disk.

[0008] The current subtask stored in the first middleware is distributed to the agent module through the first middleware.

[0009] The current subtask is executed through the proxy module.

[0010] In this embodiment, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware. This can reduce the coupling of data transmission between the first module and the proxy module. The first middleware stores the subtask on the disk. Since the disk capacity can be expanded according to the actual situation, it can provide sufficient storage resources for task scheduling, support the realization of high-throughput data processing, and thus improve the processing performance of task scheduling.

[0011] In one embodiment of this application, obtaining multiple subtasks through the first module includes:

[0012] Receive a first input, which is used to select a process template, the process template including multiple task nodes for implementing the target task;

[0013] The first module generates a process instance based on the process template, and the process instance includes subtasks corresponding to each task node.

[0014] In this embodiment, the user can set a process template according to the actual situation. The first module generates process instances based on the process template selected by the user, resulting in multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, enabling batch processing of process instances and improving the generation efficiency of process instances.

[0015] In one embodiment of this application, the first middleware includes an executor and a message queue;

[0016] The step of distributing the current subtask stored in the first middleware to the agent module through the first middleware includes:

[0017] The executor sends the task information of the current subtask stored in the message queue to the agent module.

[0018] The agent module retrieves the current subtask from the message queue.

[0019] In this embodiment, the executor sends the task information of the current subtask to the agent module. The agent module can retrieve the current subtask from the message queue according to the actual situation, such as when the agent module is idle, so as to avoid the agent module crashing due to high operating pressure.

[0020] In one embodiment of this application, the number of the proxy modules is multiple;

[0021] The step of sending the task information of the current subtask stored in the message queue to the agent module through the executor includes:

[0022] The executor sends the task information of the current subtask stored in the message queue to multiple agent modules.

[0023] The step of retrieving the current subtask from the message queue through the proxy module includes:

[0024] The current subtask is obtained from the message queue by the first proxy module among the multiple proxy modules, and the current subtask is added to the shared queue;

[0025] The execution of the current subtask through the proxy module includes:

[0026] If the second proxy module among the multiple proxy modules acquires the distributed lock, the current subtask is retrieved from the shared queue through the second proxy module;

[0027] The second agent module executes the current subtask and feeds back the execution result of the current subtask to the executor.

[0028] In this embodiment, multiple agent modules process subtasks in a shared queue, and a distributed lock is used to ensure that only one agent module can obtain a subtask from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different agent modules and causing duplicate execution of the subtask.

[0029] In one embodiment of this application, executing the current subtask through the second proxy module includes:

[0030] The second proxy module encapsulates the current subtask according to its task type to obtain an encapsulation command;

[0031] The second agent module invokes the target module to execute the current subtask according to the encapsulation command.

[0032] In this embodiment, different processing can be performed on the current subtask based on different task types to optimize the processing of the current subtask.

[0033] In one embodiment of this application, the process by which the target module executes the current subtask includes:

[0034] If the task type is the first type, the target module executes the current subtask locally.

[0035] If the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0036] In this embodiment, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution, so as to optimize the processing of the current subtask.

[0037] In one embodiment of this application, the process by which the target module executes the current subtask includes:

[0038] The target module stores the shared variables obtained during the execution of the current subtask and the execution status of the current subtask into the second middleware.

[0039] In this embodiment, the shared variables and execution status obtained during the execution of the current subtask are stored in the second middleware. The first module can obtain the execution status of the current subtask from the second middleware as needed, realize the transmission of shared variables between multiple subtasks of the target task, and facilitate other modules to monitor the execution status of the current subtask.

[0040] Secondly, embodiments of this application provide a task scheduling device, comprising:

[0041] The acquisition module is used to acquire multiple subtasks through the first module, wherein the multiple subtasks have a sequential execution order.

[0042] A storage module is used to store multiple subtasks in a first middleware in the order of their execution, with the subtasks in the first middleware stored on a disk.

[0043] The distribution module is used to distribute the current subtask stored in the first middleware to the agent module through the first middleware;

[0044] An execution module is used to execute the current subtask through the proxy module.

[0045] In this embodiment, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware. This can reduce the coupling of data transmission between the first module and the proxy module. The first middleware stores the subtask on the disk. Since the disk capacity can be expanded according to the actual situation, it can provide sufficient storage resources for task scheduling, support the realization of high-throughput data processing, and thus improve the processing performance of task scheduling.

[0046] In one embodiment of this application, the acquisition module includes:

[0047] A receiving module is used to receive a first input, which is used to select a process template, the process template including multiple task nodes for implementing the target task;

[0048] The generation module is used to generate a process instance based on the process template through the first module. The process instance includes subtasks corresponding to each task node.

[0049] In this embodiment, the user can set a process template according to the actual situation. The first module generates process instances based on the process template selected by the user, resulting in multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, enabling batch processing of process instances and improving the generation efficiency of process instances.

[0050] In one embodiment of this application, the first middleware includes an executor and a message queue;

[0051] The distribution module includes:

[0052] A sending submodule is used to send the task information of the current subtask stored in the message queue to the agent module through the executor;

[0053] The first acquisition submodule is used to acquire the current subtask from the message queue through the proxy module.

[0054] In this embodiment, the executor sends the task information of the current subtask to the agent module. The agent module can retrieve the current subtask from the message queue according to the actual situation, such as when the agent module is idle, so as to avoid the agent module crashing due to high operating pressure.

[0055] In one embodiment of this application, the number of the proxy modules is multiple;

[0056] The sending submodule includes:

[0057] A sending unit is used to send the task information of the current subtask stored in the message queue to multiple agent modules through the executor;

[0058] The acquisition submodule includes:

[0059] The acquisition unit is used to acquire the current subtask from the message queue through a first proxy module among the multiple proxy modules;

[0060] An adding unit is used to add the current subtask to the shared queue through the first agent module;

[0061] The execution module includes:

[0062] The second acquisition submodule is used to retrieve the current subtask from the shared queue through the second agent module when the second agent module among the multiple agent modules acquires the distributed lock;

[0063] An execution submodule is used to execute the current subtask through the second proxy module;

[0064] The feedback submodule is used to feed back the execution result of the current subtask to the executor through the second agent module.

[0065] In this embodiment, multiple agent modules process subtasks in a shared queue, and a distributed lock is used to ensure that only one agent module can obtain a subtask from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different agent modules and causing duplicate execution of the subtask.

[0066] In one embodiment of this application, the execution module includes:

[0067] The encapsulation submodule is used to encapsulate the current subtask according to the task type of the current subtask through the second proxy module to obtain the encapsulation command;

[0068] The execution submodule is used to invoke the target module to execute the current subtask according to the encapsulation command through the second proxy module.

[0069] In this embodiment, different processing can be performed on the current subtask based on different task types to optimize the processing of the current subtask.

[0070] In one embodiment of this application, the process by which the target module executes the current subtask includes:

[0071] If the task type is the first type, the target module executes the current subtask locally.

[0072] If the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0073] In this embodiment, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution, so as to optimize the processing of the current subtask.

[0074] In one embodiment of this application, the process by which the target module executes the current subtask includes:

[0075] The target module stores the shared variables obtained during the execution of the current subtask and the execution status of the current subtask into the second middleware.

[0076] In this embodiment, the shared variables and execution status obtained during the execution of the current subtask are stored in the second middleware. The first module can obtain the execution status of the current subtask from the second middleware as needed, realize the transmission of shared variables between multiple subtasks of the target task, and facilitate other modules to monitor the execution status of the current subtask.

[0077] Thirdly, embodiments of this application provide a task scheduling system, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the task scheduling method as described in the first aspect.

[0078] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the task scheduling method as described in the first aspect.

[0079] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0080] The features, advantages, and technical effects of exemplary embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Figure 1 A flowchart illustrating the task scheduling method provided in this application embodiment;

[0082] Figure 2 This is a schematic diagram of the structure of the task scheduling system provided in the embodiments of this application;

[0083] Figure 3 Another flowchart illustrating the task scheduling method provided in this application embodiment;

[0084] Figure 4 This is a schematic diagram of the structure of the task scheduling device provided in the embodiments of this application;

[0085] Figure 5 This is a schematic diagram of the hardware structure of the task scheduling system provided in an embodiment of this application. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0087] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used in the description of this application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the description, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the description, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a specific order or hierarchy.

[0088] The task scheduling method, apparatus, system, and medium provided in this application will be described in detail below with reference to the accompanying drawings. The task scheduling method disclosed in this application can be applied to the field of distributed task scheduling system technology.

[0089] Figure 1 A flowchart illustrating a task scheduling method provided in an embodiment of this application is shown below. Figure 1 As shown, the method is applied to a task scheduling system, and the method includes steps 101-104, wherein:

[0090] Step 101: Obtain multiple subtasks through the first module, and the multiple subtasks have a sequential execution order.

[0091] The task scheduling system comprises a first module, a first middleware, and a proxy module. These components work together to complete task scheduling. The first module, also known as the core module, is used to obtain multiple subtasks. These subtasks are subtasks of the same task and have a sequential execution order. For example, the first module generates a directed graph based on a flowchart drawn by the front-end user, with the start node having an in-degree of 0 and the end node having an out-degree of 0. Starting from the task start node, the system sequentially parses the next task node, and each task node can be encapsulated into a corresponding subtask.

[0092] Information about each subtask, such as its parameters and the order in which they are executed, can be stored in the database of the task scheduling system.

[0093] Step 102: The first module stores the multiple subtasks in the first middleware in the order of their execution, and the subtasks in the first middleware are stored on the disk.

[0094] The first module obtains information about each subtask from the database and stores the multiple subtasks in the first middleware, also known as the message middleware, according to their execution order.

[0095] Step 103: Distribute the current subtask stored in the first middleware to the agent module through the first middleware.

[0096] For example, the first middleware may include an executor and a message queue. Subtasks sent by the first module to the first middleware are stored in the message queue. The executor sends the task information of the subtasks received by the first middleware to the agent module. After receiving the task information of the subtasks, the agent module retrieves the subtasks from the message queue of the first middleware.

[0097] Step 104: Execute the current subtask through the proxy module.

[0098] After obtaining the current subtask, the proxy module executes it, obtains the execution result, and feeds it back to the executor. The executor then feeds its result back to the first module, which stores it in the task scheduling system's database. The execution result can be the completion status of the current subtask (e.g., completed or not completed) or the execution result of the current subtask. Based on the execution result of the current subtask, the first module can determine whether to send subtasks whose execution order follows the current subtask to the first middleware.

[0099] In this embodiment, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware. This reduces the coupling of data transmission between the first module and the proxy module. Since the proxy module is responsible for executing the subtask, its processing efficiency is lower than that of the first module. By storing the subtask in the first middleware, the problem of inconsistent processing efficiency between the first module and the proxy module when processing subtasks can be alleviated. In addition, the first middleware stores the subtask on the disk. Since the disk capacity can be expanded according to the actual situation, it can provide sufficient storage resources for task scheduling, support high-throughput data processing, and thus improve the processing performance of task scheduling.

[0100] In one embodiment of this application, obtaining multiple subtasks through the first module includes:

[0101] Receive a first input, which is used to select a process template, the process template including multiple task nodes for implementing the target task;

[0102] The first module generates a process instance based on the process template, and the process instance includes subtasks corresponding to each task node.

[0103] Specifically, users select a process template in the visual interface by dragging and dropping, and set the corresponding template parameters. The process template can be pre-set by the user. It can be understood as an abstract model that defines a series of steps, conditions, and rules, and can complete complex tasks composed of multiple steps.

[0104] After selecting a workflow template, you also need to configure its parameters. These parameters can be configured by the user or use default parameters; there is no restriction here. For example, if the workflow template is used to calculate the oxidation potential of molecules, the input parameter is the molecule. You can also configure the environmental parameters for running subtasks. Different input values ​​and parameters in the workflow template will result in different workflow instances. For example, inputting 1000 different molecules into the workflow module will result in 1000 workflow instances.

[0105] The process template includes multiple task nodes for achieving the target task. These task nodes form a directed graph, with the start node having an in-degree of 0 and the end node having an out-degree of 0. The first module parses each task node sequentially from the task start node and encapsulates each task node into a corresponding subtask. Based on the directed graph, the dependencies between each task node can be obtained, thus determining the execution order of the subtasks corresponding to each task node.

[0106] In this embodiment, the user can set a process template according to the actual situation. The first module generates process instances based on the process template selected by the user, resulting in multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, enabling batch processing of process instances and improving the efficiency of process instance generation.

[0107] In another embodiment of this application, the first middleware includes an executor and a message queue;

[0108] The step of distributing the current subtask stored in the first middleware to the agent module through the first middleware includes:

[0109] The executor sends the task information of the current subtask stored in the message queue to the agent module.

[0110] The agent module retrieves the current subtask from the message queue.

[0111] Specifically, the first middleware may include multiple executors. The first module sends the subtasks in the process instance to different executors in sequence according to their execution order. These executors are responsible for notifying the agent module of the task information of the subtasks, monitoring the task completion status, and implementing a task failure retry mechanism.

[0112] In this embodiment, the executor sends the task information of the current subtask to the agent module. The agent module can retrieve the current subtask from the message queue according to the actual situation, such as when the agent module is idle, so as to avoid the agent module crashing due to excessive running pressure.

[0113] In yet another embodiment of this application, the number of the proxy modules is multiple;

[0114] The step of sending the task information of the current subtask stored in the message queue to the agent module through the executor includes:

[0115] The executor sends the task information of the current subtask stored in the message queue to multiple agent modules.

[0116] The step of retrieving the current subtask from the message queue through the proxy module includes:

[0117] The current subtask is obtained from the message queue by the first proxy module among the multiple proxy modules, and the current subtask is added to the shared queue;

[0118] The execution of the current subtask through the proxy module includes:

[0119] If the second proxy module among the multiple proxy modules acquires the distributed lock, the current subtask is retrieved from the shared queue through the second proxy module;

[0120] The second agent module executes the current subtask and feeds back the execution result of the current subtask to the executor.

[0121] In this embodiment, there are multiple proxy modules. In a distributed system, one proxy module can be set up on one device, and multiple proxy modules on multiple devices process multiple subtasks in the first middleware. The executor sends the task information of the current subtask to each proxy module. After receiving the task information, the first proxy module retrieves the current subtask from the message queue and adds it to the shared queue. The first proxy module can be the first among the multiple proxy modules to retrieve the current subtask.

[0122] The shared queue is a queue shared by multiple agent modules. When each agent module executes a subtask, it retrieves the subtask from the shared queue. To ensure data consistency in a distributed environment, only one agent module needs to have exclusive access to the shared queue at any given time. In this embodiment, a distributed lock is set up, and the agent module that acquires the distributed lock is called the second agent module. The second agent module retrieves the current subtask from the shared queue and executes the current subtask.

[0123] In this embodiment, multiple agent modules process subtasks in the shared queue, and a distributed lock is used to ensure that only one agent module can obtain a subtask from the shared queue at the same time, so as to avoid the same subtask in the shared queue being executed by different agent modules, resulting in duplicate execution of the subtask.

[0124] In another embodiment of this application, executing the current subtask through the second proxy module includes:

[0125] The second proxy module encapsulates the current subtask according to its task type to obtain an encapsulation command;

[0126] The second agent module invokes the target module to execute the current subtask according to the encapsulation command.

[0127] For example, the second proxy module packages different Python commands based on the task type of the current subtask to call the target module, which then executes the current subtask and releases the distributed lock after execution.

[0128] Task types can include a first type and a second type. The first type can be a normal task, which generally refers to tasks with low computational load and low resource consumption. The second type can be a high-performance task, which generally refers to tasks with high resource consumption and long execution time.

[0129] The encapsulation command can include the task type of the current subtask. When calling the target module according to the encapsulation command, the task type of the current subtask in the encapsulation command can be passed to the target module, and the target module will execute the current subtask based on the different task types.

[0130] In this example, the current subtask can be processed differently based on different task types to optimize the processing of the current subtask.

[0131] In yet another embodiment of this application, the process by which the target module executes the current subtask includes:

[0132] If the task type is the first type, the target module executes the current subtask locally.

[0133] If the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0134] Specifically, the target module executes the current subtask based on the task type. If the task type is type one, such as a normal task, the target module executes the current subtask locally. If the task type is type two, such as a high-performance task, the target module sends the current subtask to the target scheduling system, which then executes it on the high-performance computer. This maximizes the allocation of hardware resources to each subtask, allowing the high-performance computer to fully utilize its capabilities and improving task execution efficiency.

[0135] The target scheduling system can be the Slurm task scheduling system. With the allocation of the Slurm task scheduling system and the support of high-performance computers, the execution efficiency of the current subtask can be effectively improved.

[0136] In this embodiment, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution, so as to optimize the processing of the current subtask.

[0137] In yet another embodiment of this application, the process by which the target module executes the current subtask includes:

[0138] The target module stores the shared variables obtained during the execution of the current subtask and the execution status of the current subtask into the second middleware.

[0139] Specifically, the second middleware can be Redis, and the execution status can refer to the current subtask's status such as suspended, running, completed, or encountering an error. Storing shared variables and execution status obtained during the execution of the current subtask in the second middleware allows shared variables to be passed between multiple subtasks of the target task. It also facilitates other modules in monitoring the execution status of the current subtask. For example, the first module can obtain the execution status of the current subtask from the second middleware as needed.

[0140] The task scheduling method implemented in this application can be improved based on the current mainstream distributed scheduling system by introducing a message middleware (i.e., the first middleware) to enhance the system's high concurrency capabilities. Users can import workflow templates and configure parameters for tasks at each node within the template. The first module constructs an executable process instance using the workflow template, and then assigns the subtasks in the process instance to different executors according to node relationships. These executors are responsible for notifying the agent module of the tasks, monitoring task completion status, and implementing a retry mechanism for task failures. After receiving the tasks, the agent module executes the subtasks sequentially, using distributed locks to resolve concurrency conflicts during execution.

[0141] The task scheduling method provided in the embodiments of this application is illustrated by the following examples.

[0142] like Figure 2 The diagram shown is a structural schematic of the task scheduling system provided in an embodiment of this application.

[0143] The first module generates a directed graph based on the flowchart drawn by the front-end user, with the start node having an in-degree of 0 and the end node having an out-degree of 0. It sequentially parses the next task node from the start node, encapsulates it into different subtasks according to the graph structure, and then sends them to the agent module via the Kafka module. This first module also handles subtask retry functionality upon failure.

[0144] The agent module is primarily used to obtain subtasks and encapsulate the corresponding environments, then call `pythonWrapper` to execute the specific subtasks, ensuring the ACID properties of transactions during task execution. Since the agent module serves a cluster and uses multi-threaded task submission, a distributed lock is also introduced to ensure that only one node can execute the task at a time, preventing duplicate task execution.

[0145] The target module, also known as the pythonWrapper module, is mainly responsible for executing specific subtasks. If it is a high-performance task, it submits the task to the slurm scheduling system for execution via a remote connection; if it is a normal task, it executes it directly on the local machine.

[0146] Message middleware includes:

[0147] The first middleware, implemented using Kafka, is primarily responsible for caching subtasks and decoupling the system.

[0148] Redis primarily provides a publish-subscribe model and is responsible for notifying threads when they are suspended or resumed.

[0149] The database includes:

[0150] The second middleware uses a Redis database: it mainly stores some intermediate shared variables and the execution status of subtasks. The state of the distributed lock is also stored in the second middleware. For example, a distributed lock flag of 0 in the second middleware indicates that the distributed lock is in an idle state, and a distributed lock flag of 1 indicates that the distributed lock is in use.

[0151] The first database, using MongoDB, primarily stores workflow node information (i.e., subtask information) and workflow instances, such as the execution order of each subtask.

[0152] The Slurm scheduler is responsible for scheduling and executing high-performance tasks and allocating resources efficiently, enabling these tasks to run efficiently on high-performance computers.

[0153] like Figure 3 The diagram shown is a flowchart of a task scheduling method provided in an embodiment of this application.

[0154] Step 301: Receive user input. The user generates a process template in the visual interface by dragging and dropping, and sets the corresponding system parameters. The user then clicks to execute the operation.

[0155] Step 302: After the process template is clicked to execute, the first module generates a process instance based on the passed process template.

[0156] Step 303: The first module sequentially sends the subtasks of the process instance to the proxy module. For example, the first module applies the strategy pattern, observer pattern, and chain of responsibility design pattern to distribute each specific subtask in the process instance to the proxy module via Kafka. Simultaneously, the first module can also query the execution status of each process instance.

[0157] Step 304: After acquiring the distributed lock, the proxy module retrieves a subtask from the shared queue, executes it, and releases the distributed lock upon completion. Specifically, after receiving the corresponding task message, the proxy module mainly performs the following steps:

[0158] First, add the task to the shared queue.

[0159] Multiple agent modules circulate to acquire the distributed lock and determine whether there are any executable tasks in the shared queue.

[0160] If a task needs to be executed, it is encapsulated into different Python commands according to different task types, and the PythonWrapper is called to execute the task. After the task is completed, the distributed lock is released.

[0161] Step 305: The target module executes the subtasks according to their task types. If it's a normal task, it's executed locally. If it's a high-performance task, it's scheduled by the Slurm system and executed on a high-performance computer. Through Slurm's allocation and the support of high-performance computers, task execution efficiency can be effectively improved.

[0162] In the above approach, message middleware and design patterns are used to reduce system coupling, improve system scalability, and enable the system to run stably in high-concurrency environments. Meanwhile, intermediate variables for each subtask are stored in a second middleware, facilitating parameter passing. Depending on the task type, tasks are executed either locally or by the Slurm scheduling system, effectively improving task execution efficiency and resource utilization.

[0163] Please see Figure 4 This is a schematic diagram of the task scheduling device provided in the embodiments of this application, as shown below. Figure 4 As shown, the task scheduling device 400 includes:

[0164] The acquisition module is used to acquire multiple subtasks through the first module, wherein the multiple subtasks have a sequential execution order.

[0165] A storage module is used to store multiple subtasks in a first middleware in the order of their execution, with the subtasks in the first middleware stored on a disk.

[0166] The distribution module is used to distribute the current subtask stored in the first middleware to the agent module through the first middleware;

[0167] An execution module is used to execute the current subtask through the proxy module.

[0168] In this embodiment, the first module sends the subtask to the first middleware, and the proxy module obtains the subtask from the first middleware. This can reduce the coupling of data transmission between the first module and the proxy module. The first middleware stores the subtask on the disk. Since the disk capacity can be expanded according to the actual situation, it can provide sufficient storage resources for task scheduling, support the realization of high-throughput data processing, and thus improve the processing performance of task scheduling.

[0169] In one embodiment of this application, the acquisition module includes:

[0170] A receiving module is used to receive a first input, which is used to select a process template, the process template including multiple task nodes for implementing the target task;

[0171] The generation module is used to generate a process instance based on the process template through the first module. The process instance includes subtasks corresponding to each task node.

[0172] In this embodiment, the user can set a process template according to the actual situation. The first module generates process instances based on the process template selected by the user, resulting in multiple subtasks for achieving the target task. By selecting a process template, multiple process instances can be obtained by setting different parameters, enabling batch processing of process instances and improving the generation efficiency of process instances.

[0173] In one embodiment of this application, the first middleware includes an executor and a message queue;

[0174] The distribution module includes:

[0175] A sending submodule is used to send the task information of the current subtask stored in the message queue to the agent module through the executor;

[0176] The first acquisition submodule is used to acquire the current subtask from the message queue through the proxy module.

[0177] In this embodiment, the executor sends the task information of the current subtask to the agent module. The agent module can retrieve the current subtask from the message queue according to the actual situation, such as when the agent module is idle, so as to avoid the agent module crashing due to high operating pressure.

[0178] In one embodiment of this application, the number of the proxy modules is multiple;

[0179] The sending submodule includes:

[0180] A sending unit is used to send the task information of the current subtask stored in the message queue to multiple agent modules through the executor;

[0181] The acquisition submodule includes:

[0182] The acquisition unit is used to acquire the current subtask from the message queue through a first proxy module among the multiple proxy modules;

[0183] An adding unit is used to add the current subtask to the shared queue through the first agent module;

[0184] The execution module includes:

[0185] The second acquisition submodule is used to retrieve the current subtask from the shared queue through the second agent module when the second agent module among the multiple agent modules acquires the distributed lock;

[0186] An execution submodule is used to execute the current subtask through the second proxy module;

[0187] The feedback submodule is used to feed back the execution result of the current subtask to the executor through the second agent module.

[0188] In this embodiment, multiple agent modules process subtasks in a shared queue, and a distributed lock is used to ensure that only one agent module can obtain a subtask from the shared queue at the same time, thereby preventing the same subtask in the shared queue from being executed by different agent modules and causing duplicate execution of the subtask.

[0189] In one embodiment of this application, the execution module includes:

[0190] The encapsulation submodule is used to encapsulate the current subtask according to the task type of the current subtask through the second proxy module to obtain the encapsulation command;

[0191] The execution submodule is used to invoke the target module to execute the current subtask according to the encapsulation command through the second proxy module.

[0192] In this embodiment, different processing can be performed on the current subtask based on different task types to optimize the processing of the current subtask.

[0193] In one embodiment of this application, the process by which the target module executes the current subtask includes:

[0194] If the task type is the first type, the target module executes the current subtask locally.

[0195] If the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

[0196] In this embodiment, the current subtask can be processed differently based on different task types. The current subtask can be executed locally or handed over to the target scheduling system for execution, so as to optimize the processing of the current subtask.

[0197] In one embodiment of this application, the process by which the target module executes the current subtask includes:

[0198] The target module stores the shared variables obtained during the execution of the current subtask and the execution status of the current subtask into the second middleware.

[0199] In this embodiment, the shared variables and execution status obtained during the execution of the current subtask are stored in the second middleware. The first module can obtain the execution status of the current subtask from the second middleware as needed, realize the transmission of shared variables between multiple subtasks of the target task, and facilitate other modules to monitor the execution status of the current subtask.

[0200] The task scheduling device 400 provided in this application embodiment can implement the various processes implemented in the aforementioned method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0201] Figure 5 A schematic diagram of the hardware structure of the task scheduling system provided in an embodiment of this application is shown.

[0202] The task scheduling system may include a processor 501 and a memory 502 storing computer program instructions.

[0203] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0204] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In some instances, memory 502 may include removable or non-removable (or fixed) media, or memory 502 may be a non-volatile solid-state memory. In some embodiments, memory 502 may be internal or external to a battery device.

[0205] In some instances, memory 502 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0206] Memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0207] The processor 501 reads and executes computer program instructions stored in the memory 502 to achieve... Figure 1 The method in the illustrated embodiment achieves... Figure 1 The technical effects achieved by executing the methods / steps shown in the examples are not elaborated here for the sake of brevity.

[0208] In one example, the task scheduling system may also include a communication interface 503 and a bus 504. Wherein, as... Figure 5As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 504 and complete communication with each other.

[0209] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0210] Bus 504 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 504 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0211] Alternatively, embodiments of this application can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the task scheduling methods described in the above embodiments.

[0212] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0213] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0214] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0215] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A task scheduling method, characterized in that, include: The first module obtains information about multiple subtasks, which have a sequential execution order. The information of each subtask includes the parameters of the subtask and the sequential execution order of the subtasks. The first module stores multiple subtasks in the first middleware according to the order of execution, and the subtasks in the first middleware are stored on the disk. The current subtask stored in the first middleware is distributed to the agent module through the first middleware. The current subtask is executed through the agent module.

2. The task scheduling method according to claim 1, characterized in that, The process of obtaining information about multiple subtasks through the first module includes: Receive a first input, which is used to select a process template, the process template including multiple task nodes for implementing the target task; The first module generates a process instance based on the process template, and the process instance includes information about the subtasks corresponding to each task node.

3. The task scheduling method according to claim 1, characterized in that, The first middleware includes an executor and a message queue; The step of distributing the current subtask stored in the first middleware to the agent module through the first middleware includes: The executor sends the task information of the current subtask stored in the message queue to the agent module. The agent module retrieves the current subtask from the message queue.

4. The task scheduling method according to claim 3, characterized in that, There are multiple proxy modules; The step of sending the task information of the current subtask stored in the message queue to the agent module through the executor includes: The executor sends the task information of the current subtask stored in the message queue to multiple agent modules. The step of retrieving the current subtask from the message queue through the proxy module includes: The current subtask is obtained from the message queue by the first proxy module among the multiple proxy modules, and the current subtask is added to the shared queue; The execution of the current subtask through the proxy module includes: If the second proxy module among the multiple proxy modules acquires the distributed lock, the current subtask is retrieved from the shared queue through the second proxy module; The second agent module executes the current subtask and feeds back the execution result of the current subtask to the executor.

5. The task scheduling method according to claim 4, characterized in that, The execution of the current subtask through the second agent module includes: The second proxy module encapsulates the current subtask according to its task type to obtain an encapsulation command; The second agent module invokes the target module to execute the current subtask according to the encapsulation command.

6. The task scheduling method according to claim 5, characterized in that, The process by which the target module executes the current subtask includes: If the task type is the first type, the target module executes the current subtask locally. If the task type is the second type, the target module sends the current subtask to the target scheduling system for execution.

7. The task scheduling method according to claim 5, characterized in that, The process by which the target module executes the current subtask includes: The target module stores the shared variables obtained during the execution of the current subtask and the execution status of the current subtask into the second middleware.

8. A task scheduling device, characterized in that, include: The acquisition module is used to acquire information about multiple subtasks through the first module. The multiple subtasks have a sequential execution order. The information of each subtask includes the parameters of the subtask and the sequential execution order between the subtasks. A storage module is used to store multiple subtasks in a first middleware in the order of their execution, with the subtasks in the first middleware stored on a disk. The distribution module is used to distribute the current subtask stored in the first middleware to the agent module through the first middleware; An execution module is used to execute the current subtask through the proxy module.

9. A task scheduling system, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the task scheduling method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the task scheduling method as described in any one of claims 1 to 7.

Images