A task scheduling method, device, equipment and storage medium

A hybrid scheduling approach combining parallel and serial engines addresses the challenge of serial task dependencies, enabling efficient execution of both task types.

CN114647491BActive Publication Date: 2024-12-03CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011503319.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-17
Publication Date
2024-12-03
Estimated Expiration
2040-12-17

AI Technical Summary

Technical Problem

Existing parallel scheduling engines fail to handle serial tasks due to the lack of consideration for dependencies between schedules, leading to incomplete task execution.

Method used

Introduce a serial scheduling engine to handle serial tasks, registering them chronologically and ensuring dependencies are respected, while integrating with a parallel scheduling engine to manage both task types effectively.

Benefits of technology

Enables the scheduling of both parallel and serial tasks, ensuring serial tasks are executed in chronological order and dependencies are maintained, improving task processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114647491B_ABST
    Figure CN114647491B_ABST
Patent Text Reader

Abstract

The present application discloses a task scheduling method, device, equipment and storage medium, the method comprising: when determining that the first task to be scheduled is a serial task, registering the first task to be scheduled to a serial scheduling engine; using the serial scheduling engine to schedule the target task in the first task to be scheduled, so as to send the target task to the calculation engine for calculation processing to obtain a calculation result; if the calculation result indicates that the calculation is successful, using the serial scheduling engine to schedule the next task in the first task to be scheduled. In this way, a serial scheduling engine is added on the basis of the existing parallel scheduling engine, and the serial task can be sent to the serial scheduling engine for scheduling, that is, the serial scheduling engine is used to implement the scheduling of the serial task, and the dependence relationship between the serial tasks is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a task scheduling method, device, equipment and storage medium. Background Art

[0002] There are usually multiple tasks to be processed in data processing tools. Since these tasks are periodic, a distributed scheduling engine is needed to schedule the corresponding tasks according to the set time period. After scheduling, they are sent to the computing engine for computing and processing.

[0003] Task types generally include parallel tasks and serial tasks. Existing parallel scheduling engines only schedule parallel tasks, that is, they schedule tasks strictly according to the set time period without considering the dependencies between schedules. Serial tasks mean that the next task can only be executed after one task is completed, that is, the dependencies between schedules need to be considered. Therefore, when the tasks obtained by the existing parallel scheduling engine include serial tasks, the serial tasks cannot be scheduled. Summary of the invention

[0004] In order to solve the above technical problems, the present application hopes to provide a task scheduling method, device, equipment and storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, a task scheduling method is provided, the method comprising:

[0007] When determining that the first task to be scheduled is a serial task, registering the first task to be scheduled to a serial scheduling engine;

[0008] Using the serial scheduling engine to schedule the target task in the first task to be scheduled, so as to send the target task to the calculation engine for calculation processing to obtain a calculation result;

[0009] If the calculation result indicates that the calculation is successful, the serial scheduling engine is used to schedule the next task in the first task to be scheduled.

[0010] In the above scheme, before registering the first task to be scheduled with the serial scheduling engine, the method also includes: creating a corresponding serial scheduling queue in the serial scheduling engine based on the serial task type of the first task to be scheduled; registering the first task to be scheduled with the serial scheduling engine includes: writing the first task to be scheduled into the corresponding serial scheduling queue based on the serial task type of the first task to be scheduled.

[0011] In the above scheme, the use of the serial scheduling engine to schedule the target task in the first task to be scheduled includes: determining the target serial scheduling queue corresponding to the first task to be scheduled based on the serial task type of the first task to be scheduled; and using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling.

[0012] In the above scheme, before registering the first task to be scheduled to the serial scheduling engine, the method also includes: creating a corresponding serial scheduling thread in the serial scheduling engine based on the serial task type of the first task to be scheduled; using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling, including: executing the serial scheduling thread to obtain the first exclusive lock corresponding to the first task to be scheduled; if the first exclusive lock is successfully obtained before scheduling the first task to be scheduled, executing the serial scheduling thread to retrieve the target task from the corresponding serial scheduling queue in turn for scheduling.

[0013] In the above scheme, the use of the serial scheduling engine to schedule the next task in the first task to be scheduled includes: if the serial scheduling queue is not empty, using the serial scheduling engine to schedule the next task in the first task to be scheduled; the method also includes: if the serial scheduling queue is empty, releasing the first exclusive lock corresponding to the first task to be scheduled.

[0014] In the above scheme, before determining that the first task to be scheduled is a serial task, the method also includes: using a parallel scheduling engine to obtain the first task to be scheduled and registering the first task to be scheduled; using the parallel scheduling engine to determine the task type of the first task to be scheduled; when determining that the first task to be scheduled is a serial task, registering the first task to be scheduled to the serial scheduling engine, including: when using the parallel scheduling engine to determine that the first task to be scheduled is a serial task, using the parallel scheduling engine to register the first task to be scheduled to the serial scheduling engine.

[0015] In the above solution, the method further includes: when the parallel scheduling engine is used to determine that the first task to be scheduled is a parallel task, the first task to be scheduled is directly sent to the computing engine.

[0016] In the above scheme, before sending the target task to the computing engine for computing processing, the method further includes: converting the format of the target task from json format to yaml format.

[0017] In the above solution, the method further includes: if the calculation result indicates a calculation failure, stopping the scheduling of the first task to be scheduled.

[0018] The stopping of scheduling of the first task to be scheduled includes: using the serial scheduling engine to send a pause scheduling instruction to the parallel scheduling engine, so that the parallel scheduling engine responds to the pause scheduling instruction and stops scheduling of the first task to be scheduled.

[0019] In the above scheme, after stopping the scheduling of the first task to be scheduled, the method also includes: determining unscheduled tasks and / or scheduling failed tasks, and using the unscheduled tasks and / or scheduling failed tasks as the second task to be scheduled; when determining that the second task to be scheduled is a serial task, registering the second task to be scheduled to the serial scheduling engine.

[0020] In the above scheme, the determination of unscheduled tasks and / or scheduling failed tasks includes: based on a preset period, executing a make-up period producer thread to periodically scan historical scheduled tasks; and determining unscheduled tasks and / or scheduling failed tasks from the historical scheduled tasks.

[0021] In the above scheme, when determining that the second task to be scheduled is a serial task, registering the second task to be scheduled to the serial scheduling engine includes: when determining that the second task to be scheduled is a serial task, writing the second task to be scheduled into the corresponding make-up cycle queue.

[0022] In the above scheme, after writing the second task to be scheduled into the corresponding make-up cycle queue, the method also includes: executing the make-up cycle consumer thread to obtain the second exclusive lock corresponding to the second task to be scheduled; if the second exclusive lock is successfully acquired before the second task to be scheduled is scheduled, executing the make-up cycle consumer thread to sequentially retrieve the target tasks in the second task to be scheduled from the corresponding make-up cycle queue for scheduling.

[0023] In the above solution, the method further includes: when it is determined that the second task to be scheduled is a parallel task, sending the second task to be scheduled to the computing engine.

[0024] In a second aspect, a task scheduling device is provided, which includes: a parallel scheduling engine, a serial scheduling engine and a calculation scheduling engine; wherein:

[0025] The parallel scheduling engine is used to register the first task to be scheduled with the serial scheduling engine when determining that the first task to be scheduled is a serial task;

[0026] The serial scheduling engine is used to schedule the target task in the first task to be scheduled, so as to send the target task to the computing engine for computing processing;

[0027] The calculation scheduling engine is used to perform calculation processing on the target task. If the calculation result indicates that the calculation is successful, the serial scheduling engine is used to schedule the next task in the first task to be scheduled.

[0028] According to a third aspect, a task scheduling device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0029] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0030] By adopting the above technical solution, when it is determined that the first task to be scheduled is a serial task, the first task to be scheduled is registered to the serial scheduling engine; the target task in the first task to be scheduled is scheduled by using the serial scheduling engine to send the target task to the calculation engine for calculation processing to obtain the calculation result; if the calculation result indicates that the calculation is successful, the next task in the first task to be scheduled is scheduled by using the serial scheduling engine. In this way, a serial scheduling engine is added on the basis of the existing parallel scheduling engine, and the serial task can be sent to the serial scheduling engine for scheduling, that is, the serial scheduling engine is used to realize the scheduling of the serial task, and the dependence relationship between the serial tasks is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the first process of the task scheduling method in an embodiment of the present application;

[0032] Figure 2 This is a task scheduling framework diagram in an embodiment of the present application;

[0033] Figure 3 A second flow chart of task scheduling in an embodiment of the present application;

[0034] Figure 4 A first flow chart of scheduling unscheduled tasks and / or scheduling failed tasks in an embodiment of the present application;

[0035] Figure 5 A second flow chart of scheduling unscheduled tasks and / or scheduling failed tasks in an embodiment of the present application;

[0036] Figure 6 This is a schematic diagram of the structure of the task scheduling device in the embodiment of the present application;

[0037] Figure 7 This is a schematic diagram of the structure of the task scheduling device in the embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0039] The present application embodiment provides a task scheduling method, Figure 1 FIG. 1 is a first flow chart of the task scheduling method in an embodiment of the present application. Figure 1 As shown, the task scheduling method may specifically include:

[0040] Step 101: When determining that the first task to be scheduled is a serial task, registering the first task to be scheduled to a serial scheduling engine;

[0041] It should be noted that the first task to be scheduled is a task that is about to be scheduled. A serial task is a task that is executed in chronological order, or the next task can be executed only after one task is completed. Here, the action of determining the task type of the first task to be scheduled can be completed by the processor of the task scheduling device, or by the parallel scheduling engine.

[0042] It should be noted that the serial scheduling engine is used to automatically schedule the first task to be scheduled at a set periodic time, and the serial scheduling engine usually needs to obtain the first task to be scheduled before scheduling the first task to be scheduled. Therefore, the first task to be scheduled is registered to the serial scheduling engine here.

[0043] It should be noted that, since there may be at least one task that needs to be processed at a certain time point, the number of the first tasks to be scheduled may be one or more.

[0044] In some embodiments, before registering the first task to be scheduled with the serial scheduling engine, the method also includes: based on the serial task type of the first task to be scheduled, creating a corresponding serial scheduling queue in the serial scheduling engine; registering the first task to be scheduled with the serial scheduling engine includes: based on the serial task type of the first task to be scheduled, writing the first task to be scheduled into the corresponding serial scheduling queue.

[0045] It should be noted that the first task to be scheduled here is a serial task, and the target task in the first task to be scheduled needs to be executed in chronological order. The embodiment of the present application utilizes the first-in-first-out feature of the queue to store the serial tasks in the queue in chronological order, which can ensure that the serial scheduling engine picks them up from the queue in chronological order. Since different types of serial tasks are included in the actual application, it is necessary to create corresponding serial scheduling queues for different types of first tasks to be scheduled.

[0046] In some embodiments, before determining that the first task to be scheduled is a serial task, the method also includes: using a parallel scheduling engine to obtain the first task to be scheduled and registering the first task to be scheduled; using the parallel scheduling engine to determine the task type of the first task to be scheduled; when determining that the first task to be scheduled is a serial task, registering the first task to be scheduled to the serial scheduling engine, including: when using the parallel scheduling engine to determine that the first task to be scheduled is a serial task, using the parallel scheduling engine to register the first task to be scheduled to the serial scheduling engine.

[0047] It should be noted that the task types of the first task to be scheduled include serial tasks and parallel tasks. Here, the parallel scheduling engine is used to determine whether the first task to be scheduled is a serial task or a parallel task.

[0048] It should be noted that a comparison table is created in advance based on the name of the first task to be scheduled and the corresponding task type. After the parallel scheduling engine obtains the first task to be scheduled, the corresponding task type is queried from the comparison table according to the name of the first task to be scheduled.

[0049] Specifically, the first task to be scheduled is registered with the parallel scheduling engine. When the parallel scheduling engine determines that the first task to be scheduled is a serial task, the parallel scheduling engine cannot schedule the serial task, so the parallel scheduling engine needs to re-register the first task to be scheduled with the serial scheduling engine. In addition, if the parallel scheduling engine determines that the first task to be scheduled is a parallel task, the parallel scheduling engine can schedule the task to be scheduled.

[0050] Here, when the task types of the first task to be scheduled are different, the corresponding scheduling information may also be different. Therefore, when the first task to be scheduled is registered with the parallel scheduling engine, the scheduling information configured for the first task to be scheduled is also registered with the parallel scheduling engine. The scheduling information includes the start scheduling time, the end scheduling time and the scheduling period.

[0051] Afterwards, the parallel scheduling engine periodically retrieves the first task to be scheduled based on the scheduling information corresponding to the first period to be scheduled and tags the first task to be scheduled with data time information. The tag information of data time includes a name and a time point, for example: the name of the tag is datatime, and the value of the tag is 2020060101. In other words, the parallel scheduling engine specifically registers the identification information of the first task to be scheduled and the corresponding tag information to the serial scheduling engine. The identification information can be the name of the first task to be scheduled.

[0052] It should be noted that the tag information is used to identify the first task to be scheduled. The tag information facilitates the subsequent identification of the specific scheduling status of the first task to be scheduled at each scheduling time point, and the scheduling status includes: scheduling success, failure or unscheduled.

[0053] In some embodiments, the method further includes: when the parallel scheduling engine determines that the first task to be scheduled is a parallel task, directly sending the first task to be scheduled to the computing engine.

[0054] It should be noted that parallel tasks are tasks that are executed in parallel.

[0055] Exemplarily, the first to-be-scheduled tasks at the first time point, the second time point, and the third time point are sent to the computing engine at the same time.

[0056] Step 102: using the serial scheduling engine to schedule the target task in the first task to be scheduled, so as to send the target task to the calculation engine for calculation processing to obtain a calculation result;

[0057] It should be noted that the first task to be scheduled is scheduled multiple times according to the periodic time, so the first task to be scheduled includes multiple target tasks to be scheduled.

[0058] It should be noted that the computing engine is used to perform computing processing on tasks. Usually, after the computing engine completes the processing, it will return a result indicating the success or failure of the computing. The failure of the computing may be due to a crash or abnormality in the computing engine.

[0059] In some embodiments, the use of the serial scheduling engine to schedule the target task in the first task to be scheduled includes: determining the target serial scheduling queue corresponding to the first task to be scheduled based on the serial task type of the first task to be scheduled; and using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling.

[0060] It should be noted that, since the serial task types of the first task to be scheduled are different, the corresponding serial scheduling queues are different. Therefore, after obtaining the first task to be scheduled, first find the corresponding target serial scheduling queue from all serial scheduling queues according to its serial task type, and then use the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling.

[0061] In some embodiments, before registering the first task to be scheduled with the serial scheduling engine, the method also includes: creating a corresponding serial scheduling thread in the serial scheduling engine based on the serial task type of the first task to be scheduled; using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling, including: determining the target serial scheduling thread corresponding to the first task to be scheduled based on the serial task type of the first task to be scheduled; executing the target serial scheduling thread to obtain the first exclusive lock corresponding to the first task to be scheduled; if the first exclusive lock is successfully acquired before scheduling the first task to be scheduled, executing the target serial scheduling thread to retrieve the target tasks from the corresponding target serial scheduling queue in turn for scheduling.

[0062] It should be noted that the first exclusive lock means that the lock can only be held by one thread at a time. Here, a thread only needs to obtain the first exclusive lock corresponding to a task for the first time, and then pick up the tasks in the queue one by one, and release the exclusive lock only when the queue is empty.

[0063] It should be noted that, since the serial task types of the first tasks to be scheduled are different, the corresponding serial scheduling threads are different. Therefore, after the first tasks to be scheduled are obtained, the corresponding target serial scheduling thread is first found from all serial scheduling threads according to the serial task type. In other words, each first task to be scheduled corresponds to a serial scheduling thread.

[0064] The above-mentioned process of using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling is specifically as follows: before retrieving the target task, the target serial scheduling thread needs to be executed to obtain the first exclusive lock corresponding to the first task to be scheduled. After successfully acquiring the first exclusive lock, the target serial scheduling thread is executed to retrieve the target tasks from the target serial scheduling queue in turn for scheduling until the target serial scheduling queue is empty, and the scheduling ends.

[0065] In addition, it should be noted that before the serial scheduling engine sends the first task to be scheduled to the computing engine in sequence, it will convert the first task to be scheduled in JSON format into YAML format. This reduces the data packet size of the first task to be scheduled format and reduces the bandwidth pressure between the serial scheduling engine and the computing engine.

[0066] Step 103: If the calculation result indicates that the calculation is successful, the serial scheduling engine is used to schedule the next task in the first task to be scheduled.

[0067] It should be noted that even if the time point for scheduling the next task has arrived, it is still necessary to wait for the previous task to be completed, that is, the serial scheduling engine receives the calculation result returned by the calculation engine indicating successful calculation.

[0068] In some embodiments, the use of the serial scheduling engine to schedule the next task in the first task to be scheduled includes: if the target serial scheduling queue is not empty, using the serial scheduling engine to schedule the next task in the first task to be scheduled; the method also includes: if the target serial scheduling queue is empty, releasing the first exclusive lock corresponding to the first task to be scheduled.

[0069] That is to say, when the target serial scheduling queue is not empty, it is necessary to continue scheduling the next task in the first task to be scheduled until the target serial scheduling queue is empty. When the target serial scheduling queue is empty, the scheduling is completed, and the first exclusive lock needs to be released.

[0070] In some embodiments, the method further includes: if the calculation result indicates a calculation failure, stopping the scheduling of the first task to be scheduled.

[0071] That is, when the calculation result indicates that the calculation has failed, it is necessary to stop scheduling the first task to be scheduled to avoid invalid scheduling of the first task to be scheduled.

[0072] In addition, it should be noted that the action of stopping the scheduling of the first task to be scheduled can be completed by the processor of the task scheduling device, or by notifying other scheduling engines to complete it.

[0073] In some embodiments, stopping scheduling of the task to be scheduled includes: using the serial scheduling engine to send a pause scheduling instruction to the parallel scheduling engine, so that the parallel scheduling engine responds to the pause scheduling instruction and stops scheduling the first task to be scheduled.

[0074] It should be noted that the above mentioned is to register the first task to be scheduled to the parallel scheduling engine first. When it is determined that the first task to be scheduled is a serial task, the parallel scheduling engine re-registers the first task to be scheduled to the serial scheduling engine. Therefore, when the calculation result indicates that the calculation fails, the serial scheduling engine is used to notify the parallel scheduling engine to stop scheduling the first task to be scheduled. In this way, invalid scheduling of the first task to be scheduled is avoided in a timely manner.

[0075] Here, the execution subject of step 101 to step 103 may be a processor of the task scheduling device.

[0076] By adopting the above technical solution, when it is determined that the first task to be scheduled is a serial task, the first task to be scheduled is registered to the serial scheduling engine; the target task in the first task to be scheduled is scheduled by using the serial scheduling engine to send the target task to the calculation engine for calculation processing to obtain the calculation result; if the calculation result indicates that the calculation is successful, the next task in the first task to be scheduled is scheduled by using the serial scheduling engine. In this way, a serial scheduling engine is added on the basis of the existing parallel scheduling engine, and the serial task can be sent to the serial scheduling engine for scheduling, that is, the serial scheduling engine is used to realize the scheduling of the serial task, and the dependence relationship between the serial tasks is solved.

[0077] In the above embodiment Figure 1 Based on this, the embodiment of the present application provides a task scheduling framework diagram, Figure 2 This is a task scheduling framework diagram in an embodiment of the present application.

[0078] like Figure 2 As shown, the task scheduling framework diagram includes: a distributed scheduling main engine 201 (ie, a parallel scheduling engine), a distributed scheduling co-engine 202 (ie, a serial scheduling engine) and a calculation engine 203.

[0079] The distribution mentioned here means that there are multiple scheduling nodes on the main engine or the co-engine, which can realize the scheduling of various different tasks.

[0080] Here, the distributed scheduling main engine is used to register serial tasks and / or parallel tasks, and schedule parallel tasks. The distributed scheduling co-engine is used to register serial tasks, and schedule serial tasks.

[0081] Specifically, when the distributed scheduling main engine determines that the first task to be scheduled is a parallel task, the distributed scheduling main engine sends the first task to be scheduled to the computing engine according to the set cycle time. If it is a serial task, the first serial task to be scheduled is first registered to the distributed scheduling co-engine, and then the distributed scheduling co-engine is used to send the first serial task to be scheduled to the computing engine.

[0082] Based on the above embodiments, this application provides a task scheduling method. Figure 3 FIG. 2 is a second flow chart of task scheduling in an embodiment of the present application. Figure 3 shown, specifically,

[0083] Step 301: Start;

[0084] Step 302: Obtain the task and configure scheduling information for the task;

[0085] Here, the task is the task to be scheduled. The scheduling information includes: the start scheduling time, end time and scheduling period of the task.

[0086] Here, the action of obtaining tasks is completed by the main scheduling engine.

[0087] Step 303: Register the task and scheduling information to the main scheduling engine;

[0088] Here, the main scheduling engine is the parallel scheduling engine.

[0089] Step 304: The main scheduling engine periodically retrieves tasks and adds data time information to the tasks;

[0090] Here, the main scheduling engine adds data time information to the task, so that it is easy to identify at which time point the task is not scheduled or fails to be scheduled.

[0091] Step 305: whether the task is a serial task; if not, execute step 310; if yes, execute step 306;

[0092] Step 306: registering the task and data time information to the serial scheduling queue of the coordination engine;

[0093] Here, the coordination engine is the serial scheduling engine.

[0094] Step 307: The serial scheduling thread acquires the exclusive lock of the task;

[0095] Step 308: Check whether the exclusive lock is acquired successfully; if successful, execute step 309; if failed, execute step 316;

[0096] Step 309: the serial scheduling thread sequentially retrieves tasks from the serial scheduling queue;

[0097] Step 310: Submit the task to the computing engine;

[0098] Step 311: The computing engine returns the task's Universally Unique Identifier (UUID) and subscribes to the event of the task's execution result;

[0099] Step 312: Whether the execution result indicates success; if successful, execute step 313; if failed, execute step 314;

[0100] Step 313: Is the serial dispatch queue empty? If not, execute step 309; if so, execute step 315;

[0101] Step 314: Notify the main scheduling engine to suspend scheduling, or notify the main scheduling engine to suspend scheduling and release the exclusive lock;

[0102] Step 315: Release the exclusive lock;

[0103] Step 316: End.

[0104] Since the computing engine crashes or has an abnormality during the computing process, some tasks fail to be scheduled or are not scheduled. Therefore, the embodiment of the present application provides another task scheduling method. Figure 4 This is a schematic diagram of the first process for scheduling unscheduled tasks and / or failed tasks in an embodiment of the present application.

[0105] like Figure 4 shown, specifically,

[0106] Step 401: determining unscheduled tasks and / or scheduling failed tasks, and taking the unscheduled tasks and / or scheduling failed tasks as second tasks to be scheduled;

[0107] It should be noted that unscheduled tasks are tasks that are not scheduled during the period from when the computing engine crashes or is abnormal to when it is restored to normal. Failed scheduling tasks are tasks that fail to be calculated due to crashes or abnormalities in the computing engine.

[0108] In some embodiments, determining unscheduled tasks and / or failed scheduling tasks includes: based on a preset period, executing a make-up period producer thread to periodically scan historical scheduled tasks; and determining unscheduled tasks and / or failed scheduling tasks from the historical scheduled tasks.

[0109] It should be noted that the period-filling producer thread is used to obtain unscheduled tasks and / or scheduling failed tasks.

[0110] Specifically, each task is sent to the computing engine after scheduling, and the computing engine returns an identifier representing the success or failure of the scheduling. The period-filling producer thread uses this identifier to determine the unscheduled tasks and / or the scheduling failed tasks from the historical scheduled tasks of the parallel scheduling engine.

[0111] Step 402: When it is determined that the second task to be scheduled is a serial task, register the second task to be scheduled with the serial scheduling engine.

[0112] It should be noted that the action of determining the task type of the second task to be scheduled may be completed by a processor of the task scheduling device, or by other scheduling engines.

[0113] In some embodiments, this step specifically includes: when it is determined that the second task to be scheduled is a serial task, writing the second task to be scheduled into a corresponding make-up cycle queue.

[0114] It should be noted that the second task to be scheduled here is a serial task, and the target task in the second task to be scheduled needs to be executed in chronological order. The embodiment of the present application utilizes the first-in-first-out feature of the queue to store the serial tasks in the queue in chronological order, which can ensure that the serial scheduling engine picks them up from the queue in chronological order. Since different types of serial tasks are included in the actual application, different types of second tasks to be scheduled here will correspond to different cycle-making queues.

[0115] In some embodiments, after writing the second task to be scheduled into the corresponding make-up cycle queue, the method further includes: executing a make-up cycle consumer thread to obtain a second exclusive lock corresponding to the second task to be scheduled; if the second exclusive lock is successfully acquired before the second task to be scheduled is scheduled, executing the make-up cycle consumer thread to sequentially retrieve the target tasks in the second task to be scheduled from the corresponding make-up cycle queue for scheduling.

[0116] It should be noted that the second exclusive lock means that the lock can only be held by one thread at a time. Here, a thread only needs to obtain the exclusive lock corresponding to a task for the first time, and then pick up the tasks in the queue one by one, and release the exclusive lock only when the queue is empty.

[0117] Specifically, before retrieving the target task from the second task to be scheduled, the make-up cycle consumer thread needs to be executed to acquire the second exclusive lock corresponding to the second task to be scheduled. After successfully acquiring the second exclusive lock, the make-up cycle consumer thread is executed to retrieve the target task from the second task to be scheduled from the make-up cycle queue in turn for scheduling, until the make-up cycle queue is empty and the scheduling ends.

[0118] In some embodiments, the method further includes: when determining that the second task to be scheduled is a parallel task, sending the second task to be scheduled to the computing engine.

[0119] It should be noted that parallel tasks are tasks that are executed in parallel.

[0120] Exemplarily, the second to-be-scheduled tasks at the first time point and the second time point are sent to the computing engine at the same time.

[0121] In the above embodiment Figure 4 Based on this, this application specifically provides another scheduling method: Figure 5 FIG. 2 is a schematic diagram of a second process for scheduling unscheduled tasks and / or scheduling failed tasks in an embodiment of the present application. Figure 5 shown, specifically,

[0122] Step 501: Start;

[0123] Step 502: Execute the replenishment cycle producer thread to determine the task of automatic replenishment cycle;

[0124] Here, the tasks that automatically make up the period include unscheduled tasks and scheduling failed tasks.

[0125] Specifically, the fill-in-cycle producer thread is executed to scan the historical scheduling tasks of the main scheduling engine (ie, the parallel scheduling engine) to obtain unscheduled tasks and / or scheduling failed tasks.

[0126] Step 503: writing the automatic period-filling task into the period-filling queue;

[0127] Step 504: Start the replenishment period consumer timing thread;

[0128] Step 505: Obtain the automatic replenishment cycle task from the replenishment cycle queue;

[0129] Step 506: Whether the task is a serial task; if so, execute step 509; if not, execute step 507;

[0130] Step 507: extracting automatic period-filling tasks from the period-filling queue in batches according to the remaining resources of the computing engine;

[0131] Here, batching means that at least one time point corresponding to a certain task fails to be scheduled or is not scheduled and is retrieved from the replenishment cycle queue at the same time.

[0132] Step 508: Send the automatic period-filling tasks in batches to the computing engine;

[0133] Here, batching means sending at least one time point corresponding to a certain task that fails to be scheduled or is not scheduled to the computing engine at the same time.

[0134] Here, the action of sending the automatic period-filling tasks in batches to the computing engine can be completed by the processor of the task scheduling device, or by the processor controlling other scheduling engines to complete the task.

[0135] Step 509: Acquire an exclusive lock;

[0136] Step 510: Check whether the exclusive lock is acquired successfully; if successful, execute step 511; if failed, execute step 518;

[0137] Step 511: sequentially extracting the automatic replenishment period tasks in the replenishment period queue;

[0138] Step 512: Send the automatic period-filling task to the computing engine;

[0139] Here, the action of sending the automatic cycle-filling task to the computing engine can be completed by a processor of the task scheduling device, or by the processor controlling a serial scheduling engine.

[0140] Step 513: The computing engine returns the task's Universally Unique Identifier (UUID) and subscribes to the event of the task's execution result;

[0141] Step 514: Whether the execution result indicates success; if successful, execute step 515; if failed, execute step 516;

[0142] Step 515: Check whether the cycle replenishment queue is empty; if not, execute step 511; if so, execute step 517;

[0143] Step 516: notify the main scheduling engine to suspend scheduling and release the exclusive lock;

[0144] Here, the main scheduling engine is the parallel scheduling engine.

[0145] Step 517: Release the exclusive lock;

[0146] Step 518: End.

[0147] Here, for Figure 5 Here is a specific example:

[0148] Assume that a task scheduling process is configured with a scheduler SchedulerA (i.e., the task to be scheduled). SchedulerA needs to be scheduled once every hour, starting from 2020-10-1 01:00 and ending at 2020-10-01 05:00. The complete scheduling data time set is 2020100101, 2020100102, 2020100103, 2020100104, and 2020100105. Each data time is dependent on the previous data time. The next data time can be submitted to the computing engine only after the execution of the previous data time is completed. Assume that the computing engine crashes or an exception occurs after the main scheduling engine ends at the 2020100101 data time. The entire process steps are as follows:

[0149] Step 1: When the computing engine crashes or an exception occurs, the coordination engine (i.e., serial scheduling engine) notifies the main scheduling engine through the message queue to suspend scheduling SchedulerA.

[0150] Step 2: At 2020-10-01 04:00, the coordination engine finds that the main scheduling engine's scheduling of SchedulerA is missing three scheduling cycles, so the consumer thread stores 2020100102, 2020100103, and 2020100104 in the automatic cycle replenishment queue.

[0151] Step 3: When the coordination engine finds that the computing engine has returned to normal, the complement cycle consumer thread will acquire the exclusive lock. After success, it will take out 2020100102 and send it to the computing engine. After the computing engine completes the calculation, it will notify the coordination engine through the message queue. The coordination engine will take out 2020100103 and send it to the computing engine until 2020100104 is calculated and the exclusive lock is released.

[0152] Step 4: After the data time 2020100104 is executed and no new automatic replenishment cycle is generated, the coordination engine will notify the main scheduling engine to resume scheduling SchedulerA. The main scheduling engine will submit the scheduling of the data time 2020100105 to the calculation engine on a regular basis.

[0153] The present application provides a task scheduling device, such as Figure 6 As shown, the task scheduling device includes: a parallel scheduling engine, a serial scheduling engine and a calculation scheduling engine; wherein,

[0154] The parallel scheduling engine 601 is used to register the first task to be scheduled to the serial scheduling engine when determining that the first task to be scheduled is a serial task;

[0155] The serial scheduling engine 602 is used to schedule the target task in the first task to be scheduled, so as to send the target task to the computing engine for computing processing;

[0156] The calculation scheduling engine 603 is used to perform calculation processing on the target task. If the calculation result indicates that the calculation is successful, the serial scheduling engine is used to schedule the next task in the first task to be scheduled.

[0157] In some embodiments, before registering the first task to be scheduled to the serial scheduling engine, a corresponding serial scheduling queue is created in the serial scheduling engine based on the serial task type of the first task to be scheduled; the device includes: a parallel scheduling engine 601, which is specifically used to write the first task to be scheduled into the corresponding serial scheduling queue based on the serial task type of the first task to be scheduled.

[0158] In some embodiments, when the serial scheduling engine is used to schedule the target task in the first task to be scheduled, the target serial scheduling queue corresponding to the first task to be scheduled is determined based on the serial task type of the first task to be scheduled; and the serial scheduling engine is used to retrieve the target task from the target serial scheduling queue for scheduling.

[0159] In some embodiments, before registering the first task to be scheduled with the serial scheduling engine, a corresponding serial scheduling thread is created in the serial scheduling engine based on the serial task type of the first task to be scheduled; when using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling, the target serial scheduling thread corresponding to the first task to be scheduled is determined based on the serial task type of the first task to be scheduled; the target serial scheduling thread is executed to obtain the first exclusive lock corresponding to the first task to be scheduled; if the first exclusive lock is successfully obtained before scheduling the first task to be scheduled, the target serial scheduling thread is executed to retrieve the target tasks from the corresponding target serial scheduling queue in turn for scheduling.

[0160] In some embodiments, the device includes: a computing scheduling engine 603, which is specifically used to schedule the next task in the first task to be scheduled using the serial scheduling engine if the target serial scheduling queue is not empty; if the target serial scheduling queue is empty, release the first exclusive lock corresponding to the first task to be scheduled.

[0161] In some embodiments, before determining that the first task to be scheduled is a serial task, the first task to be scheduled is acquired by using a parallel scheduling engine, and the first task to be scheduled is registered;

[0162] The parallel scheduling engine is used to determine the task type of the first task to be scheduled; the device includes: a parallel scheduling engine 601, which is specifically used to register the first task to be scheduled to the serial scheduling engine using the parallel scheduling engine when the parallel scheduling engine is used to determine that the first task to be scheduled is a serial task.

[0163] In some embodiments, the method further includes: when the parallel scheduling engine is used to determine that the first task to be scheduled is a parallel task, directly sending the first task to be scheduled to the computing engine.

[0164] In some embodiments, before sending the target task to the computing engine for computing processing, the format of the target task is converted from json format to yaml format.

[0165] In some embodiments, the method further includes: if the calculation result indicates a calculation failure, stopping the scheduling of the first task to be scheduled.

[0166] In some embodiments, stopping scheduling of the first task to be scheduled includes: using the serial scheduling engine to send a pause scheduling instruction to the parallel scheduling engine, so that the parallel scheduling engine responds to the pause scheduling instruction and stops scheduling of the first task to be scheduled.

[0167] In some embodiments, after stopping scheduling of the first task to be scheduled, unscheduled tasks and / or scheduling failed tasks are determined, and the unscheduled tasks and / or scheduling failed tasks are used as second tasks to be scheduled; when it is determined that the second task to be scheduled is a serial task, the second task to be scheduled is registered to the serial scheduling engine.

[0168] In some embodiments, determining unscheduled tasks and / or failed scheduling tasks includes: based on a preset period, executing a make-up period producer thread to periodically scan historical scheduled tasks; and determining unscheduled tasks and / or failed scheduling tasks from the historical scheduled tasks.

[0169] In some embodiments, when determining that the second task to be scheduled is a serial task, registering the second task to be scheduled to the serial scheduling engine includes: when determining that the second task to be scheduled is a serial task, writing the second task to be scheduled into a corresponding make-up cycle queue.

[0170] In some embodiments, after writing the second task to be scheduled into the corresponding make-up cycle queue, the make-up cycle consumer thread is executed to obtain the second exclusive lock corresponding to the second task to be scheduled; if the second exclusive lock is successfully acquired before the second task to be scheduled is scheduled, the make-up cycle consumer thread is executed to sequentially retrieve the target tasks in the second task to be scheduled from the corresponding make-up cycle queue for scheduling.

[0171] In some embodiments, the method further includes: when determining that the second task to be scheduled is a parallel task, sending the second task to be scheduled to the computing engine.

[0172] By adopting the above technical solution, when it is determined that the first task to be scheduled is a serial task, the first task to be scheduled is registered to the serial scheduling engine; the target task in the first task to be scheduled is scheduled by using the serial scheduling engine to send the target task to the calculation engine for calculation processing to obtain the calculation result; if the calculation result indicates that the calculation is successful, the next task in the first task to be scheduled is scheduled by using the serial scheduling engine. In this way, a serial scheduling engine is added on the basis of the existing parallel scheduling engine, and the serial task can be sent to the serial scheduling engine for scheduling, that is, the serial scheduling engine is used to realize the scheduling of the serial task, and the dependence relationship between the serial tasks is solved.

[0173] The present application embodiment provides a task scheduling device, such as Figure 7 As shown, the task scheduling device includes: a processor 701 and a memory 702 configured to store a computer program that can be run on the processor; wherein the first processor 701 is configured to execute the method steps in the aforementioned embodiment when running the computer program.

[0174] Of course, in practical applications, Figure 7 As shown, the various components in the terminal device are coupled together via a first bus system 703 .

[0175] Of course, in practical applications, Figure 7 As shown, the various components in the task scheduling device are coupled together through a bus system 703. It can be understood that the bus system 703 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 703 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 703.

[0176] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0177] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0178] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0179] Optionally, the computer-readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the processor in each method in the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0182] In addition, all functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art may understand that all or part of the steps of implementing the above method embodiments may be completed by hardware related to program instructions, and the above program may be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the above storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0183] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0184] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0185] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0186] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A task scheduling method, characterized in that: The method comprises: Obtaining a first task to be scheduled by using a parallel scheduling engine, and registering the first task to be scheduled; Determine the task type of the first task to be scheduled by using the parallel scheduling engine; When the parallel scheduling engine is used to determine that the first task to be scheduled is a serial task, the parallel scheduling engine is used to register the first task to be scheduled to the serial scheduling engine; Using the serial scheduling engine to schedule the target task in the first task to be scheduled, so as to send the target task to the calculation engine for calculation processing to obtain a calculation result; If the calculation result indicates that the calculation is successful, the serial scheduling engine is used to schedule the next task in the first task to be scheduled.

2. The method according to claim 1, characterized in that Before registering the first task to be scheduled to the serial scheduling engine, the method further includes: Based on the serial task type of the first task to be scheduled, creating a corresponding serial scheduling queue in the serial scheduling engine; The step of registering the first task to be scheduled to the serial scheduling engine includes: Based on the serial task type of the first task to be scheduled, the first task to be scheduled is written into a corresponding serial scheduling queue.

3. The method according to claim 2, characterized in that The using the serial scheduling engine to schedule the target task in the first to-be-scheduled task includes: Determine, based on the serial task type of the first task to be scheduled, a target serial scheduling queue corresponding to the first task to be scheduled; The serial scheduling engine is used to retrieve the target task from the target serial scheduling queue for scheduling.

4. The method according to claim 3, characterized in that Before registering the first task to be scheduled to the serial scheduling engine, the method further includes: Based on the serial task type of the first task to be scheduled, creating a corresponding serial scheduling thread in the serial scheduling engine; The using the serial scheduling engine to retrieve the target task from the target serial scheduling queue for scheduling includes: Determine, based on the serial task type of the first task to be scheduled, a target serial scheduling thread corresponding to the first task to be scheduled; Execute the target serial scheduling thread to acquire the first exclusive lock corresponding to the first task to be scheduled; If the first exclusive lock is successfully acquired before scheduling the first task to be scheduled, the target serial scheduling thread is executed to sequentially retrieve the target tasks from the corresponding target serial scheduling queue for scheduling.

5. The method according to claim 4, characterized in that The using the serial scheduling engine to schedule the next task in the first to-be-scheduled tasks includes: If the target serial scheduling queue is not empty, scheduling the next task in the first to-be-scheduled task using the serial scheduling engine; The method further comprises: If the target serial scheduling queue is empty, a first exclusive lock corresponding to the first task to be scheduled is released.

6. The method according to claim 1, characterized in that The method further comprises: When the parallel scheduling engine is used to determine that the first task to be scheduled is a parallel task, the first task to be scheduled is directly sent to the computing engine.

7. The method according to claim 1, characterized in that Before sending the target task to the computing engine for computing processing, the method further includes: Convert the target task format from JSON to YAML.

8. The method according to claim 1, characterized in that The method further comprises: If the calculation result indicates a calculation failure, the scheduling of the first task to be scheduled is stopped.

9. The method according to claim 8, characterized in that The stopping of scheduling of the first task to be scheduled includes: The serial scheduling engine is used to send a pause scheduling instruction to the parallel scheduling engine, so that the parallel scheduling engine responds to the pause scheduling instruction and stops scheduling the first task to be scheduled.

10. The method according to claim 8, characterized in that After stopping the scheduling of the first task to be scheduled, the method further includes: Determine an unscheduled task and / or a task that fails to be scheduled, and use the unscheduled task and / or the task that fails to be scheduled as a second task to be scheduled; When it is determined that the second task to be scheduled is a serial task, the second task to be scheduled is registered with the serial scheduling engine.

11. The method according to claim 10, characterized in that The determining of unscheduled tasks and / or scheduling failed tasks includes: Based on the preset period, the periodic producer thread is executed to periodically scan the historical scheduling tasks; Unscheduled tasks and / or scheduling failed tasks are determined from the historical scheduled tasks.

12. The method according to claim 10, characterized in that When determining that the second task to be scheduled is a serial task, registering the second task to be scheduled to the serial scheduling engine includes: When it is determined that the second task to be scheduled is a serial task, the second task to be scheduled is written into the corresponding make-up cycle queue.

13. The method according to claim 12, characterized in that After writing the second to-be-scheduled task into the corresponding make-up period queue, the method further includes: Execute the make-up cycle consumer thread to acquire the second exclusive lock corresponding to the second task to be scheduled; If the second exclusive lock is successfully acquired before scheduling the second task to be scheduled, the make-up cycle consumer thread is executed to sequentially retrieve the target task in the second task to be scheduled from the corresponding make-up cycle queue for scheduling.

14. The method according to claim 10, characterized in that The method further comprises: When it is determined that the second task to be scheduled is a parallel task, the second task to be scheduled is sent to the computing engine.

15. A task scheduling device, characterized in that: The device comprises: a parallel scheduling engine, a serial scheduling engine and a calculation scheduling engine; wherein, The parallel scheduling engine is used to obtain a first task to be scheduled and register the first task to be scheduled; determine the task type of the first task to be scheduled; and when determining that the first task to be scheduled is a serial task, register the first task to be scheduled to the serial scheduling engine; The serial scheduling engine is used to schedule the target task in the first task to be scheduled, so as to send the target task to the computing engine for computing processing; The calculation scheduling engine is used to perform calculation processing on the target task. If the calculation result indicates that the calculation is successful, the serial scheduling engine is used to schedule the next task in the first task to be scheduled.

16. A task scheduling device, characterized in that: The task scheduling device comprises: a processor and a memory configured to store a computer program that can be run on the processor. Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 14 when running the computer program.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.