Task processing method and task processing system

Abstract

The invention provides a task processing method and a task processing system, and relates to the technical field of computers. The task processing system combines a distributed scheduling framework and a distributed execution framework. The task scheduling module can send the fragmented tasks of the task to the target node, so that the target node performs batch processing on the fragmented tasks, and the task processing efficiency is improved. Moreover, the pre-processing function and the post-processing function determine the scheduling state of the fragmented task of the task, so that the scheduling state of the task is updated by utilizing the scheduling state of the fragmented task of the task, and the task execution module can update the execution state of the fragmented task of the task. Therefore, the execution state of the task is updated by utilizing the execution state of the fragmented task of the task, the updating of the fragmented task and the states of the task in different dimensions is realized, and the full-link tracking of the task is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task processing method and a task processing system. Background Art

[0002] With the development of technology, the business scale of enterprises is constantly expanding, which often involves task scheduling. At present, task scheduling can be mainly divided into two categories, namely single-machine task scheduling and distributed task scheduling. However, due to the limitation of single-machine processing capacity, the task processing efficiency is low, and when a single machine fails, it is easy to cause task loss. Therefore, there is an urgent need for a distributed task scheduling implementation solution. Summary of the invention

[0003] The embodiments of the present application provide a task processing method and a task processing system for implementing distributed scheduling and batch processing of tasks, thereby improving the processing efficiency of tasks.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, the present application provides a task processing method, which is applied to a task processing system combining a distributed scheduling framework and a distributed execution framework. The task processing method includes:

[0006] In response to the first task being scheduled, the first task is sliced ​​and processed through the distributed scheduling framework to obtain multiple first slice tasks of the first task. For each first slice task, the first slice task is assigned to the first target node, and the scheduling state corresponding to the first slice task is updated through the pre-processing function.

[0007] The execution status of the first shard task is updated through the first target node in combination with batch processing of the first shard task through the distributed execution framework. When the execution status of all first shard tasks indicates completion, the scheduling status of the first task is set to completion through the post-processing function of the distributed scheduling framework, where completion is success or failure.

[0008] In this application, the task processing method not only utilizes the scheduling capabilities of the distributed scheduling framework, but also combines the execution capabilities (such as batch processing capabilities) of the distributed execution framework. Since the distributed scheduling framework and the distributed execution framework themselves can improve performance, the combination of the two can effectively improve performance, thereby improving the efficiency of task processing. In addition, the distributed scheduling framework and the distributed execution framework can realize the timely update of the execution status and scheduling status of tasks and sharded tasks, thereby realizing the full-link tracking of tasks.

[0009] In a possible design, the process of batch processing the first shard task through the first target node includes:

[0010] The first sharding task is divided into a plurality of data sets, wherein at least one of the plurality of data sets includes the data in the first sharding task.

[0011] The target node processes each data set separately, thus achieving batch processing of tasks and improving system performance.

[0012] In a possible design, task cancellation (ie task termination) may also be performed. A task cancellation request is obtained, and the task cancellation request may include an identifier of the second task. Accordingly, the task cancellation request is used to trigger termination of the second task.

[0013] Afterwards, according to the execution status of the second task, the second task is terminated, and the execution status of the second task is updated to terminated. Through the post-processing function, the scheduling status of the second task is updated to success. Based on this, flexible termination of tasks can be achieved. And because task termination is also regarded as task scheduling completion, the scheduling status of the first task can be updated to success instead of terminated.

[0014] The above-mentioned process of terminating the second task according to the execution status of the second task may include: when the execution status of the second task is executing, for each second shard task of the second task, when the execution status of the second shard task is executing, continuing to execute the current data set corresponding to the second shard task through the second target node where the second shard task is located.

[0015] After the current data set is executed, stop executing the new data set corresponding to the second sharding task.

[0016] Based on this, when a sharding task is terminated, if the target node executing the sharding task is currently executing the data set of the sharding task, the current data set will continue to be executed through the target node. After the execution is completed, the execution of the new data set of the sharding task is stopped, that is, the execution of the unexecuted data set is stopped, and the task is terminated. In addition, data confusion can be avoided.

[0017] In a possible design, after stopping execution of the new data set corresponding to the second sharding task, the execution state of the second sharding task is set to terminated.

[0018] Accordingly, updating the execution status of the second task to be aborted includes:

[0019] When the execution status of each of the slice tasks of the second task is a final state, the execution status of the first task is updated to be terminated, wherein the final state includes one or more of success, failure, or termination.

[0020] Based on this, when the first task is terminated, when the execution status of each shard task of the second task (i.e., the above-mentioned second shard task) is in the terminal state, it indicates that the second task has been terminated successfully. Therefore, the execution status of the second task can be updated to terminated, which matches the actual execution status of the second task.

[0021] In a possible design, after the execution state of the second task is updated to be terminated, in response to the second task being rescheduled, when the execution state of the second shard task of the second task is terminated, business processing is performed on the data set of the second shard task that has not been business processed. Optionally, business processing is performed on the data set of the second shard task that has not been business processed through the second target node.

[0022] When the execution status of the second slice task of the second task is success or failure, the second slice task is not processed, thereby avoiding repeated execution of the second slice task.

[0023] In a possible design, abnormal tasks can also be identified. For each task whose execution status is in execution, the standard number of slices (or called the planned number of slices) and the actual number of slices of the task are obtained, as well as the planned execution time of the task. The actual number of slices indicates the number of slice tasks scheduled, and the tasks include the first task.

[0024] When the standard number of shards is different from the actual number of shards, and the first time difference between the current time and the planned execution time is greater than the first preset time threshold, the task is marked as a first abnormal task. The scheduling state of the first abnormal task is abnormal. Based on this, the task identification with abnormal scheduling is realized.

[0025] In a possible design, the tasks to be scheduled and the first abnormal task are sorted to obtain sorted tasks. After that, the sorted tasks can be scheduled in sequence. Based on this, the first abnormal task is rescheduled to ensure system reliability.

[0026] Among them, optionally, the tasks to be scheduled and the first abnormal task can be sorted based on the order of the planned execution time of each task in the tasks to be scheduled and the first abnormal task. Based on this, it is avoided that the tasks are not scheduled for a long time. Of course, other sorting rules can also be used, such as tasks have corresponding priorities, and they can be sorted according to the priorities.

[0027] In a possible design, the above-mentioned sequential scheduling of the sorted tasks may include:

[0028] When the execution status of the slice tasks of the scheduled task is success or failure, no business processing is performed on the slice tasks of the scheduled task to avoid repeated scheduling of the slice tasks.

[0029] In a second aspect, the present application provides a task processing system, which combines a distributed scheduling framework and a distributed execution framework to execute the task processing method as described above.

[0030] In a third aspect, the present application provides an electronic device, comprising a memory and one or more processors; the memory is used to store computer program code, and the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device performs the task processing method as described above.

[0031] In a fourth aspect, the present application provides a chip, the chip comprising a communication interface and at least one processor:

[0032] The communication interface is used to input and / or output signaling or data;

[0033] The at least one processor is used to execute a computer program to implement the task processing method as described above.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the task processing method as described above.

[0035] In a sixth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the task processing method as described above.

[0036] Optionally, the electronic device may be a server.

[0037] It can be understood that the beneficial effects that can be achieved by the task processing system described in the second aspect, the electronic device described in the third aspect, the chip described in the fourth aspect, the computer-readable storage medium described in the fifth aspect, and the computer program product described in the sixth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a distributed task scheduling system provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a task scheduling method provided in an embodiment of the present application Figure 1 ;

[0040] Figure 3 A schematic diagram of a task scheduling provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a task scheduling method provided in an embodiment of the present application Figure 2 ;

[0042] Figure 5 A schematic diagram of a task information provided in an embodiment of the present application Figure 1 ;

[0043] Figure 6 A schematic diagram of a task information provided in an embodiment of the present application Figure 2 ;

[0044] Figure 7 A schematic diagram of a task information provided in an embodiment of the present application Figure 3 ;

[0045] Figure 8 A schematic diagram of a task information provided in an embodiment of the present application Figure 4 ;

[0046] Fig. 9 A schematic diagram of a task scheduling method provided in an embodiment of the present application Figure 3 ;

[0047] Fig.10 A schematic diagram of a task information provided in an embodiment of the present application Figure 5 ;

[0048] Fig.11 A schematic diagram of a task information provided in an embodiment of the present application Figure 6 ;

[0049] Fig.12 A schematic diagram of a task scheduling method provided in an embodiment of the present application Figure 4 ;

[0050] Fig.13 A schematic diagram of a task information provided in an embodiment of the present application Figure 7 ;

[0051] Fig.14 A schematic diagram of a task information provided in an embodiment of the present application Figure 8 ;

[0052] Fig.15 A schematic diagram of a task information provided in an embodiment of the present application Figure 9 ;

[0053] Fig.16 A schematic diagram of a task information provided in an embodiment of the present application Figure 10 ;

[0054] Fig.17 A schematic diagram of a task scheduling method provided in an embodiment of the present application Figure 5 ;

[0055] Fig.18 A schematic diagram of a task information provided in an embodiment of the present application Figure 10 one. DETAILED DESCRIPTION

[0056] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In the embodiments of the present application, "first", "second", "1", "2" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more.

[0057] As the scale of enterprise business continues to expand, task scheduling faces challenges such as high concurrent processing requirements, consistency issues in task status, and recovery and management of abnormal tasks. Conventional distributed task scheduling systems can use frameworks such as Quartz, Elastic-job, or XXL-JOB, which are mainly responsible for task scheduling and simple task execution. Conventional distributed task scheduling systems usually have two major problems. First, the inefficiency of task distribution and execution. Conventional distributed task scheduling systems cannot fully optimize the scheduling strategy of tasks, resulting in inefficient utilization of resources during task scheduling and execution, especially in high-concurrency, large-scale task scenarios. The response speed and throughput of the system are limited. Second, there is a lack of an efficient batch processing framework. For tasks that need to process large amounts of data, conventional distributed task scheduling systems generally cannot support efficient task execution, resulting in long task execution time and affecting the overall performance of the system.

[0058] In addition, in conventional distributed task scheduling systems, the management of abnormal tasks usually relies on manual intervention, which is inefficient and can easily lead to tasks not being processed for a long time, affecting business continuity. In other words, conventional distributed task scheduling systems lack automated abnormality detection and recovery mechanisms, resulting in low reliability of task execution.

[0059] In conventional distributed task scheduling systems, the task cancellation operations at the task scheduling layer and the execution layer are often independent and lack an effective coordination mechanism. When users want to cancel a task, they often encounter the following problems: First, the interruption or omission of task cancellation. The task cancellation operation cannot guarantee that the executing task can be interrupted immediately, which may lead to waste of system resources and inconsistent task execution. Second, the task status synchronization problem. The inconsistent task status of different nodes causes confusion in task status monitoring, affecting the scheduling and status update of subsequent tasks.

[0060] Therefore, in response to the above problems, this application proposes a distributed task scheduling solution that combines Elastic-job with Spring batch framework. Elastic-job is responsible for scheduling tasks, and effectively splits tasks into multiple small tasks (i.e., shard tasks) for execution through sharding technology. Spring Batch provides standardized batch processing functions during task execution, and the execution of each task shard can be managed independently to improve the efficiency of task execution. In addition, from the two dimensions of scheduling and execution, the status of tasks and sharded tasks of tasks are divided into scheduling status and execution status. The status of tasks depends on the status of sharded tasks of tasks, and full-link tracking of tasks and sharded tasks of tasks is realized, so that flexible sharding management can be realized. In addition, when the distributed task solution is applied to large-scale distributed task processing scenarios, tasks can be executed in parallel through sharding, efficient management and reliable execution of tasks can be achieved, high-quality utilization of system resources can be ensured, system load can be reduced, processing efficiency can be improved, and the overall performance of the system can be improved.

[0061] In addition, based on the scheduling status and execution status of tasks and their fragments, automatic detection and recovery of abnormal tasks can be achieved, avoiding tasks from being unprocessed for a long time and ensuring the reliability of task execution.

[0062] In addition, it can realize flexible termination of tasks, ensure the timeliness of termination, and ensure the timely update of the status of tasks and sharded tasks.

[0063] Correspondingly, such as Figure 1 As shown, the distributed task scheduling system corresponding to the above distributed task scheduling solution may include a task scheduling creation module, a task to be scheduled scanning module, a task scheduling verification module, a task scheduling module, a task execution module, and a task cancellation module.

[0064] The above-mentioned task scheduling creation module is used to implement task name and configuration parameter definition, task scheduling strategy definition, task dependency definition, task configuration storage, task execution plan creation, etc.

[0065] The above-mentioned task to be scheduled scanning module is used to realize task scanning triggering, identification of tasks to be executed, identification of abnormal tasks, task priority sorting, etc.

[0066] The above-mentioned task scheduling verification module is used to implement the uniqueness verification of the tasks to be scheduled, the task execution time window verification, the uniqueness verification of the scheduled tasks, the task dependency verification, etc.

[0067] The above-mentioned task scheduling module is used to realize task triggering and scheduling, task distribution and sharding, registration and heartbeat monitoring, task status update, abnormal task scheduling, etc.

[0068] The above-mentioned task execution module is used to realize batch task initialization, task slice uniqueness verification, batch task execution triggering, batch task slice execution, batch task status update, etc.

[0069] The task cancellation module is used to implement task cancellation operations, update the status of tasks to be canceled, regularly scan tasks to be canceled, cancel batch task execution, cancel scheduled task scheduling, update task status, etc.

[0070] It should be noted that the above Figure 1 The modules included in the distributed task scheduling system shown are only examples. The distributed task scheduling system may also include other modules, such as a task report management module, which is used to implement task execution status tracking, execution status tracking of sharded tasks, task execution performance statistics, abnormal task identification and analysis, task dependency statistics, task execution parameter recording, historical execution trend analysis, etc. This application does not limit the functional modules included in the distributed task scheduling system.

[0071] The present application provides a distributed task scheduling system, which supports task scheduling and sharding task execution management functions based on Elastic-job and Spring batch. Based on the pre-listener and post-listener of Elastic-job, the scheduling status of the sharding tasks of the task is monitored and updated, thereby realizing the timely update of the scheduling status of the task. In addition, batch processing of sharding tasks is realized based on Spring batch, the task processing efficiency is improved, and the execution status of the sharding tasks is updated according to the execution status of the sharding tasks, thereby realizing the timely update of the execution status of the tasks.

[0072] The above process of updating the scheduling status of tasks and task slices based on Elastic-job's pre-listener and post-listener is mainly implemented through the task scheduling module. The operation process performed by the task scheduling module is mainly divided into four parts, namely task triggering and configuration registration, task slice management and scheduling, task status recording and task status update. For example, Figure 2 As shown, the operation process (i.e., task scheduling) may include:

[0073] S201 . In response to task scheduling processing, the task scheduling module determines whether to perform sharding on the task based on configuration information of the task to be processed.

[0074] The above-mentioned task to be processed (or referred to as the first task) may be a scheduled task. The configuration information of the task may include one or more of the information such as the task name, the task ID, the planned execution time of the task, the number of shards of the task, etc. Optionally, the number of shards of the task may be one or more. When the number of shards is one, it indicates that the task does not actually need to be sharded. In other words, the task can be considered as a single-shard task. When the number of shards is multiple, it indicates that the task is a multi-shard task, that is, there are multiple shard tasks in the task. For example, if the task is 10,000 data and the number of shards is 4, the task is divided into 4 shard tasks, each shard task includes 2,500 data. Of course, the number of data included in the shard task exemplified here is only an example, and the number of data included in each shard task may also be uneven. This application does not limit the specific number of shard tasks.

[0075] In an embodiment of the present application, when the planned execution time of a task is reached, it indicates that the task needs to be scheduled, and the task scheduling module can determine whether the task needs to be fragmented based on the configuration information of the task. If it is fragmented, it indicates that the task is a multi-fragmented task, and the task scheduling module can execute S202-S208. If it is not fragmented, it indicates that the task does not need to be fragmented. In other words, the task can be considered as a single-fragmented task, and the task scheduling module can execute S209-S214. It should be understood that the planned execution time of a task can be understood as the priority of the task. The earlier the planned execution time, the higher the priority, and the faster the task is scheduled, so as to avoid the related business being affected due to the failure to schedule the task in time.

[0076] Exemplarily, the configuration information includes the number of shards. When the number of shards is multiple, the task scheduling module can determine that the task to be processed needs to be sharded. When the number of shards is one, the task scheduling module can determine that the task to be processed does not need to be sharded.

[0077] S202: The task scheduling module determines the target node and the shard tasks corresponding to each target node from the node cluster based on the number of shards.

[0078] In the embodiment of the present application, the task scheduling module can determine multiple target nodes from the node cluster based on the number of shards and the shard tasks corresponding to each of the multiple target nodes in combination with the preset allocation rules. The target node can be a docker host.

[0079] For example, the preset allocation rule is an average allocation rule, the number of shards is 4, the number of nodes is 5, the number of target nodes can be 4, the shard task corresponding to target node 1 can be shard 0, the shard task corresponding to target node 2 can be shard 1, the shard task corresponding to target node 3 can be shard 2, and the shard task corresponding to target node 4 can be shard 3. In addition, the preset allocation rule can also be other rules, for example, shard tasks are preferentially allocated to nodes with smaller loads.

[0080] S203: For each target node, the task scheduling module sends the sharding task corresponding to the target node to the target node.

[0081] Following the above example, Figure 3 As shown, the task scheduling module sends the data of slice 0 to target node 1, sends the data of slice 1 to target node 2, sends the data of slice 2 to target node 3, and sends the data of slice 3 to target node 4. Of course, the example of sending the data of the slices to the corresponding target nodes in this application is only an example under an ideal state. It is also possible that the data of the slices is not sent to the corresponding node, but to other nodes, such as the data of slice 0 should be sent to target node 1, but is actually sent to target node 2. In general, the task scheduling module can send the slice task to the target node to perform business processing on the slice task through the target node. The target node may be corresponding to the slice task or not, and this application does not limit it.

[0082] In some embodiments, the task scheduling module can perform task slicing, obtain multiple slicing tasks of the task, and distribute the slicing tasks. Optionally, the task scheduling module can implement the slicing of tasks and the distribution of slicing tasks in a preset manner, such as through a Zookeeper cluster.

[0083] S204. The task scheduling module updates the scheduling status corresponding to the slice task to running through the front listener.

[0084] For example, for each shard task (or referred to as the first shard task) of the task, after the pre-listener listens to the assignment of the shard task to the corresponding target node, the scheduling state corresponding to the shard task can be updated to running (such as running), thereby implementing the pre-processing of the scheduling state (such as referring to the above Figure 3 The pre-processing shown in FIG. 1 ). Here, running can also be referred to as running or executing. The pre-listener is an example of a pre-processing function.

[0085] In addition, optionally, in actual applications, considering that the shard task may not be scheduled normally as required, the front listener may not listen to the shard task being assigned, and the scheduling state corresponding to the shard task may still be the initial state, such as waiting to be scheduled.

[0086] S205. The task scheduling module updates the scheduling status corresponding to the task to running according to the scheduling status corresponding to the sliced ​​task.

[0087] Exemplarily, when the scheduling state corresponding to the shard tasks of the task is running, the front listener can set the scheduling state corresponding to the task to running. In addition, optionally, if the scheduling states corresponding to the shard tasks of the task are all to be scheduled, the scheduling state corresponding to the task can still be the initial state, such as the to-be-scheduled state.

[0088] S206: The target node performs batch processing on the sharding tasks.

[0089] The implementation process of S206 can be referred to below. Here, it is only explained that after the shard task is executed, if the shard task is executed successfully, the execution status of the shard task is success. If the shard task fails to execute, the execution status of the shard task is failure, thereby realizing the update of the execution status (see the above Figure 3 Execution status shown).

[0090] S207: After all the slice tasks of the task to be processed are executed, the task scheduling module updates the scheduling status corresponding to the task to success or failure through the post-listener.

[0091] In the embodiment of the present application, after the shard task is executed, the task scheduling module can set the scheduling status indication of the shard task through the post-listener. For example, if the execution status of the shard task is success, the scheduling status of the shard task is also success. If the execution status of the shard task is failure, the scheduling status of the shard task is also failure. Among them, the post-listener is an example of a post-processing function.

[0092] like Figure 2 As shown, S207a, when the scheduling status of each shard task of the task is successful, it indicates that each shard task has been successfully processed, and the task scheduling module updates the scheduling status corresponding to the task to successful, thereby realizing the post-processing of the scheduling status (such as referring to the above Figure 3 post-processing shown).

[0093] In the case that the scheduling status of the existing task slice is failed, the task scheduling module updates the scheduling status corresponding to the task to failed (see S207b).

[0094] S208: The task scheduling module deletes the registration data of the task.

[0095] In the embodiment of the present application, when the scheduling status of the task is completed (such as the above-mentioned success or failure status), it indicates that all the shard tasks of the task have been executed and completed, indicating that the data of the task has been processed. The task scheduling module can determine that the task scheduling is completed and can clean up the data of the task, such as deleting the registration node data of the task through Zookeep. Of course, other data related to the task can also be deleted to avoid unnecessary task data occupying resources. Alternatively, the task scheduling module may not delete the registration data, that is, S208 is an optional step.

[0096] In some embodiments, the operation performed by the task scheduling module may be performed by node 1 (or device 1) that applies Elastic-job. The implementation process of S201-S208 may be specifically combined with the task triggering and configuration registration, task slicing management and scheduling, task status recording, and task status updating.

[0097] For task triggering and configuration registration, when the scheduled execution time of the task is reached, the task is scheduled and triggered, and device 1 determines multiple target nodes and the sharding tasks corresponding to each target node. After that, device 1 can start monitoring through the front listener. After monitoring that the target node is assigned a sharding task, device 1 can update the scheduling status of the sharding task to running.

[0098] For the above task slice management and scheduling, the device 1 can record the task, the ID of the task instance, the scheduling status of the task, the execution status, the scheduling status of the sliced ​​task, the execution status, etc. The task instance represents the task currently being executed, and the task and the task instance can be a one-to-N relationship.

[0099] Optionally, the device 1 can record the information through a preset storage medium, and the preset storage medium can be set according to the requirements. For example, the preset storage medium can be a node (such as a task management node, a sharding node, etc.), or a database, etc. This application does not limit the preset storage medium.

[0100] In addition, the target node starts batch processing of sharding tasks through Spring Batch to achieve detailed execution of assigned tasks and independent execution and management of each sharding task.

[0101] For the above task status record, after the sharding task is completed through Spring Batch execution, device 1 can monitor the execution status of the sharding task through the post-listener. After indicating completion, the scheduling status of the sharding task is also set to completion, which can be success or failure.

[0102] Furthermore, the device 1 can also update the scheduling status of the task in time through the post-listener. When the scheduling status of all the sliced ​​tasks of the task indicates completion, the scheduling status of the task is completion, which can be success or failure.

[0103] Among them, optionally, after determining the scheduling status of the shard task through the post-monitor, device 1 can update the information recorded by the relevant nodes, such as updating the scheduling status of the shard task in the above-mentioned task management node, to achieve timely dynamic update of the scheduling status of the shard task.

[0104] In addition, if the execution status or scheduling status of a task is failure, the task can be treated as an abnormal task, so that the abnormal task can be rescheduled. For details, please refer to the content related to abnormal tasks below.

[0105] For the above task status update, the device 1 determines that the scheduling status of each shard task of the task is a completion status or a failure status through the scheduling status of each shard of the task recorded by the relevant node, indicating that each shard task of the task has been executed, that is, the task has been executed, then the device 1 clears the registered node data related to the task, such as the configuration information of the task, etc. In addition, the device 1 can also delete the node running the shard task of the task in the task management node.

[0106] It should be understood that, as can be seen from the foregoing, storing information through nodes is only an example of storing information through a preset storage medium. Accordingly, device 1 clearing registered node data related to the task is also only an example, that is, device 1 can delete task-related data from the preset storage medium.

[0107] In the embodiment of the present application, the task scheduling module implements the beforeJobExecuted pre-listener and the afterJobExecuted post-listener based on the ElasticJobListener interface of the Elastic Job framework, thereby realizing the pre-listening and post-listening of the scheduling status of the task and the sharding task of the task, and ensuring the timely and accurate update of the scheduling status. At present, the above-mentioned Elastic-job is only an example of a distributed scheduling framework, and the distributed scheduling framework can also be other types of frameworks. Correspondingly, the update of the pre-scheduling status of the task and the sharding task of the task through the pre-listener of Elastic-job is only an example, and the update of the pre-scheduling status can also be realized by other means, such as by running the code related to the function of the pre-listener. In other words, the task scheduling module can update the pre-scheduling status of the task and the sharding task of the task through the pre-processing function, such as updating the scheduling status corresponding to the sharding task to running through the pre-processing function.

[0108] Similarly, using the post-listener of Elastic-job to update the post-scheduling status of a task or a task's fragment is only an example, and other methods may be used, such as running codes that can implement functions related to the post-listener. In other words, the task scheduling module may update the post-scheduling status of a task or a task's fragment through the post-processing function, such as updating the scheduling status corresponding to the task to success or failure through the post-processing function.

[0109] S209: The task scheduling module determines the target node.

[0110] In the embodiment of the present application, since sharding is not required, the task scheduling module can directly select a target node from the node cluster, and the target node is the node that executes the task to be processed.

[0111] S210: The task scheduling module allocates the task to the target node.

[0112] S211. The task scheduling module updates the scheduling status of the task to running through the front listener.

[0113] S212: The target node performs batch processing on the tasks.

[0114] S213: After the task is executed, the task scheduling module updates the scheduling status corresponding to the task to completed through the post-listener.

[0115] Among them, completion indicates success or failure.

[0116] S214: The task scheduling module deletes the registration data of the task.

[0117] Among them, the implementation process of S209-S214 is similar to the above S202-S208, and will not be repeated here.

[0118] The above describes the process of monitoring the scheduling status of the sharding task through the pre-listener and post-listener of Elastic-job. As mentioned above, the execution of the above sharding task can be realized through batch processing of the Spring Batch framework. The following will continue to introduce the execution process of the sharding task.

[0119] The execution process of the above-mentioned sharding tasks is mainly implemented by the task execution module of the target node. The operation process executed by the task execution module can be mainly divided into three parts, namely, sharding task execution, sharding status update, and task overall status update. For example, Figure 4 As shown, the operation process may include:

[0120] S301. For each shard task, after the target node receives the shard task, the task execution module of the target node updates the execution status of the shard task to being executed by calling the front listener.

[0121] Exemplarily, after being assigned to a shard task, the task execution module of the target node calls the pre-listener to initialize the execution state of the shard task, i.e., set it to executing. The target node deploys the Spring Batch framework. Accordingly, the task execution module can implement the call of the beforeJob pre-listener and the call of the afterJob post-listener by calling the JobExecutionListener interface of the Spring Batch framework.

[0122] Optionally, when the execution status of a shard task is in progress, the execution status of the pending task to which the shard task belongs can be set to in progress (e.g. Figure 5 Alternatively, after the shard task is assigned to the target node, the execution status of the task can be set to being executed. This application does not restrict the order of initialization of the shard task and the task status.

[0123] S302: The task execution module divides the slice task into multiple data blocks.

[0124] S303: For each data block, the task execution module performs business processing on the data in the data block.

[0125] In the embodiment of the present application, the task execution module of the target node divides the data of the sharding task into multiple data blocks (chunks) based on the Spring Batch framework. Then, the task execution module performs corresponding business processing on the data in each data block through the business processing code, thereby realizing batch processing of the task. Of course, similar to the above-mentioned Elastic-job, the Spring Batch framework is only an example of a distributed execution framework, and the distributed execution framework can also be other types of frameworks. Correspondingly, the above-mentioned division of the data of the sharding task into multiple chunks based on the Spring Batch framework is also only an example of batch processing of the sharding task. In general, the task execution module can divide the sharding task into multiple data sets, and there is at least one data set in the multiple data sets including the data in the sharding task. In addition, it should be noted that for a single sharding task, it can be understood as a multi-sharding task with a sharding number of 1, and the single sharding task can still be batched according to the batch processing process of the sharding task of the multi-sharding task.

[0126] S304: After all data blocks of the sharding task are processed, the task execution module updates the execution status of the sharding task to completed.

[0127] Where completion indicates failure or success.

[0128] In the embodiment of the present application, after all data blocks of the sharding task are processed, it indicates that the sharding task is executed, and the task execution module can record the execution result of the sharding task and update the execution status of the sharding task. In the case where the execution result is a failure result, the task execution module updates the execution status of the sharding task to failure. In the case where the execution result is a success result, the task execution module updates the execution status of the sharding task to success.

[0129] Exemplarily, the task execution module may monitor the execution status of the shard task by calling the post-listener, and update the execution status of the shard task.

[0130] S305: After each shard task is executed, if the execution status of all shard tasks is successful, the task scheduling module updates the execution status of the pending task to success through the post-listener.

[0131] For example, the task execution module can synchronize the execution status of the shard task to the task scheduling module through the post listener, so that the task scheduling module can update the execution status. For example, the task is a financial bill, which includes four shard tasks: shard 0, shard 1, shard 2, and shard 3. However, after the execution of shards 0 and 1 is successful, the execution status of shards 0 and 1 is T (see Figure 6 ). Where T represents success.

[0132] When the execution status of each shard task is completed, the task scheduling module can update the execution status of the task to completed through the post-listener, thereby updating the overall execution status of the task.

[0133] Specifically, when the execution status of each shard task of the task is successful, the post listener can set the execution status of the task to success (such as Figure 7 The execution status of the task shown is success). Success can also be referred to as execution success.

[0134] S306: If the execution status of any shard task is failure, the task scheduling module updates the execution status of the pending task to failure by calling the post-listener.

[0135] If the execution status of each shard task of the existing task is failed, the post listener can set the execution status of the task to fail (such as Figure 8 The execution status of the task shown is Failed, where Figure 8 The 2-F shown indicates that the execution status of the shard 2 is failure). The failure can also be referred to as execution failure.

[0136] In some embodiments, after the task is completed, whether it is successful or failed, it is the final state of the task. The task scheduling module and the task execution module can release the resources occupied by the task, such as cache resources or other resources, to ensure that the system has the resource environment required to execute the next task.

[0137] In the embodiment of the present application, the above-mentioned task scheduling module and task execution module can be on the same node, that is, on the same device, or they can be distributed on different devices. When distributed on different devices, it indicates that the logic code of the Elastic-job and Springbatch frameworks are distributed on different devices, and the business capabilities between the two are decoupled. Through interface calls, the reuse of business capabilities can be achieved, and the transplantation capability of enhanced functions can be enhanced. When distributed on the same device, it can save costs.

[0138] It should be noted that, similar to the above, the task execution module can also update the status of tasks and sharded tasks through the pre-processing function and post-processing function of the distributed execution framework. For example, the status before the task is executed can be updated by the pre-processing function of the distributed execution framework. And the status after the task is executed can be updated by the post-processing function of the distributed execution framework.

[0139] In some embodiments, the distributed task scheduling system also supports a task cancellation function (or task termination function). When a task needs to be canceled, the task cancellation module can terminate the execution of the task in a timely manner to ensure efficient use of resources. Exemplarily, the execution process of the task cancellation module mainly includes three parts, namely, task cancellation request triggering, task status update, task status check and resource cleanup. Fig. 9 , which describes the execution process in detail. Fig. 9 As shown, the execution process may include:

[0140] S601: The task cancellation module obtains a task cancellation request, wherein the task cancellation request includes an identifier of the task to be cancelled.

[0141] The identifier of the task to be canceled (or referred to as the second task) indicates information that can identify the task to be canceled, such as an ID, a name, and the like.

[0142] Exemplarily, the task cancellation request may be triggered by a user or a node. For example, the node is a target node. When a preset cancellation condition is met, the target node may determine that the task or shard task on the target node needs to be cancelled, and the target node may trigger a task cancellation request.

[0143] The preset cancellation conditions can be set as required. For example, the preset cancellation conditions include that the used resources of the target node are greater than the preset threshold, the current load is greater than the preset load, or the target node is abnormal (such as downtime, restart, etc.). This application does not limit the specific conditions included in the preset cancellation conditions. The used resources may include, but are not limited to, one or more of disk occupancy, CPU utilization, and memory utilization.

[0144] S602: The task cancellation module obtains the execution status of the task to be canceled.

[0145] In the embodiment of the present application, the task cancellation module can use the task indicated by the task cancellation request as a task to be cancelled, that is, a task that is no longer executed. Then, the task cancellation module can obtain the execution status of the task to be cancelled to terminate the execution of the task to be cancelled according to the execution status.

[0146] S603: When the execution status of the task to be canceled is in execution, the task canceling module sends a stop signal to the task executing module.

[0147] The stop signal is used to trigger the termination of the execution of the task to be cancelled. Exemplarily, the stop signal is used to trigger the termination of the execution of a new data block of the task to be cancelled.

[0148] Exemplarily, the task cancellation module can send a stop signal to the task execution module by calling the jobExecution.stop() or JobOperator.stop() method of Spring Batch. Of course, similar to the above, the SpringBatch framework is only an example of a distributed execution framework, and correspondingly, triggering the termination of a task by a stop signal is also only an example. The task cancellation module can trigger the termination of a task by the stop function under other frameworks. In general, the task cancellation module can send a stop request to the task execution module, and the stop request is used to trigger the termination of the task.

[0149] S604 . In response to the stop signal, for each shard task A of the task to be canceled whose execution status is "in execution", the task execution module updates the execution status of the shard task A to "in suspension".

[0150] In an embodiment of the present application, in response to a stop signal, for shard task B whose execution status is completed (successful or failed) for the task to be canceled, since shard task B has been executed and completed, shard task B does not need to be terminated. For shard task C whose execution status is to be executed for the task to be canceled, since shard task C has not yet started execution, the task execution module can directly set the execution status of shard task C to terminated. And for shard task A whose execution status is in execution for the task to be canceled, since the data in shard task A is currently being processed, the task execution module can set the execution status of shard task A to be stopped (i.e., stopping). Among them, shard task A, shard task B, and shard task C are all shard tasks of the task to be canceled, which can be called second shard tasks.

[0151] S605: After the data block processing of the currently executed shard task A is completed, the task execution module updates the execution status of the shard task A to aborted.

[0152] In an embodiment of the present application, in order to improve the reliability of business data execution, when it is necessary to terminate shard task A, the task execution module may not immediately terminate the execution of shard task A, but may terminate shard task A only after all data in the data block of the currently executed shard task A is executed, thereby stopping the execution of the remaining unexecuted data blocks. And the task execution module updates the execution status of shard task A from being terminated to terminated. Based on this, the smooth termination of shard task A can be achieved, and the problem of data confusion when the remaining unexecuted data blocks of shard task A are processed next time can be avoided if the data blocks of the executed shard task A are immediately terminated due to the dependence of the remaining unexecuted data blocks of shard task A on the currently executed data blocks, thereby ensuring data consistency and thus ensuring the reliability of business processing.

[0153] For example, shard task A includes shard 1, which includes 2000 data. Shard 1 is divided into 4 data blocks, namely chunk0, chunk1, chunk2, and chunk3, each of which includes 500 data. If the target node assigned with shard 1 receives a stop signal during the execution of chunk2, the task execution module of the target node can first update the execution status of shard 1 to aborted. Afterwards, after all 500 data in chunk2 are executed, the target node stops executing chunk2 and chunk3, thereby terminating the execution of shard task A, and the task execution module updates the execution status of shard 1 to aborted.

[0154] Optionally, similar to the above, the task execution module may update the execution status of the shard task A from being aborted to aborted through a post-listener.

[0155] In some embodiments, in response to the stop signal, the task execution module may also set the execution status of the task to be canceled to being suspended. Optionally, the task execution module may synchronize the execution status of the shard task A of the task to be canceled to the task scheduling module through a post-listener, and the task scheduling module updates the execution status of the task to be canceled to being suspended.

[0156] Optionally, during the suspension of the task to be cancelled, the scheduling state of the shard task A of the task to be cancelled can also be set to being suspended. In addition, the scheduling state of the task to be cancelled can also be set to being suspended.

[0157] It should be noted that, similar to the above, the above chunk is only an example of a data set. Accordingly, after the data set processing of the currently executed shard task A is completed, the task execution module updates the execution status of the shard task A to aborted.

[0158] S606: When the execution status of the fragmented task A of the task to be canceled is all aborted, the task execution module updates the execution status of the task to be canceled to aborted.

[0159] In an embodiment of the present application, when the scheduling status of each shard task of the task to be canceled has been aborted or completed, indicating that shard task A and shard task C of the task to be canceled have been terminated, the task execution module can update the execution status of the task to be canceled to aborted (or canceled, stopped, etc.) to achieve task status update.

[0160] For example, the task to be cancelled is as above Figure 5The task 1 shown is in execution. Task 1 includes four shard tasks, namely shard 0, shard 1, shard 2 and shard 3. After the task execution module successfully terminates shard 1 and shard 2, it updates the execution status of shard 1 and shard 2 to terminated (see Fig.10 0-S, 1-S, Fig.10 The S in the code indicates that the process has been terminated. Afterwards, when shard 2 and shard 3 are also terminated successfully, the execution status of shard 2 and shard 3 is also terminated (see Fig.11 2-S, 3-S), and accordingly, the execution status of task 1 is also updated to be aborted.

[0161] S607: The task scheduling module updates the scheduling status of the task to be canceled to success.

[0162] Among them, since termination is also the final state of the task, therefore, when it is determined that the execution state of the task is terminated, the scheduling state of the task to be canceled can be updated to success, thereby realizing the update of the task scheduling state. Exemplarily, the task scheduling module can determine that the execution state of the task is terminated through a post-processing function, such as a post-listener. In short, the task execution module can notify the task scheduling module that the execution state of the task to be canceled is updated to terminated.

[0163] In some embodiments, the above describes the process of terminating the task to be canceled when the execution status of the task to be canceled is in execution. In addition, when the execution status of the task to be canceled is pending, it indicates that the task to be canceled has not yet started execution, so the task cancellation module can directly cancel the task to be canceled, and the task cancellation module can directly update the execution status of the task to be canceled to terminated, and update the scheduling status of the task to be canceled to success.

[0164] In some embodiments, when the execution status of the task to be canceled is completed, it indicates that all the fragment tasks of the task to be canceled have been executed, and therefore, the task cancellation module determines that the task to be canceled cannot be canceled.

[0165] S608: The task scheduling module cleans up the resources occupied by the task to be canceled.

[0166] Exemplarily, when the execution status of the task to be canceled is aborted, it indicates that the task to be canceled is aborted successfully, that is, the fragmented tasks of the task to be canceled have all been aborted or completed, so the task scheduling module can release the resources occupied by the task to be canceled, such as cache or other resources, to complete the task abort operation, avoid unnecessary occupation of system resources, and ensure the success of scheduling the next task. Similar to the above, the cleanup of resources described in S608 is also an optional step.

[0167] In some embodiments, when terminating the shard task A of the task to be canceled, a termination failure may occur. Therefore, when the shard task A fails to terminate, the task cancellation module (such as the task cancellation module through the task execution module) can set the execution status of the task to be canceled to abort failure. Optionally, the task scheduling module monitors through the post-listener that the execution status of the task to be canceled is abort failure, and the task scheduling module can set the scheduling status of the task to be canceled to abort failure, thereby ensuring the timely and correct update of the task status. Alternatively, the task scheduling module does not change the scheduling status.

[0168] In some embodiments, for tasks whose status is aborted, the task scheduling module can reprocess the task according to user needs. If a request for continuing processing input by the user is received, the task execution module can continue to execute the corresponding slice task according to the status of each slice task of the task (such as scheduling status, execution status), thereby avoiding repeated execution of the completed status. This process is similar to the process of rescheduling abnormal tasks described below.

[0169] In addition, the task execution module can also record the execution status of each data block of the sharding task. When it is necessary to continue to execute the suspended sharding task, the task execution module can directly execute the unexecuted data block and avoid executing the executed data block. In other words, when it is necessary to terminate the sharding task, the task execution module can use the currently executed data block as the interruption point. When the sharding task is subsequently continued, the task execution module executes the data block after the interruption point.

[0170] In the embodiment of the present application, in essence, the above-mentioned task cancellation is actually implemented based on Elastic-job and Spring Batch. Spring Batch sends a stop signal through the jobExecution.stop() method, indicating that the job to be canceled is stopped and the execution of the relevant steps (StepExecution) is stopped. Afterwards, when the task to be canceled is a multi-slice task and the execution status of the task to be canceled is in execution, Spring Batch sets the execution status of the slicing task A of the task to be canceled to be suspended, and only executes the currently executed data block. After the current data block is executed, SpringBatch marks the slicing task A as suspended. When the execution status of all slicing tasks of the task to be canceled is completed or suspended, it indicates that the task to be canceled has been successfully suspended, so the execution status of the task to be canceled can be updated to suspended. Correspondingly, Elastic-job can update the scheduling status of the task to be canceled to success through the post-listener, thereby realizing flexible cancellation of the task. In addition, the task cancellation function solves the real-time and consistency problems of task cancellation and status update by effectively combining Elastic-job and Spring Batch. Of course, similar to the above, Elastic-job and SpringBatch are only exemplary frameworks, and the present application can also implement the task cancellation function described in the present application through other frameworks. In other words, the present application does not limit the implementation framework of the task cancellation function.

[0171] In some embodiments, the distributed task scheduling system also supports an abnormal task handling function. In order to implement the abnormal task handling function, the abnormal task can be first identified to reschedule the abnormal task. Among them, the identification of abnormal tasks can be implemented by an abnormal task identification module. The rescheduling of abnormal tasks can be implemented by an abnormal task scheduling module. The identification of abnormal tasks is first introduced below. Exemplarily, the abnormal task identification module can identify abnormal tasks during task execution. The operation process performed by the abnormal task identification module can be mainly divided into two parts, namely abnormal task detection and abnormal task marking. Fig.12 As shown, the operation flow (i.e., the abnormal task identification process) may include:

[0172] S401. The abnormal task identification module periodically queries the tasks in the execution state.

[0173] The execution status of the task being executed indicates the currently running task. The abnormal task identification module regularly detects all running tasks to achieve timely detection of the execution status of the tasks, so that the execution status of the tasks can be used to determine whether there are abnormal tasks.

[0174] In addition, the abnormal task identification module may detect the running tasks in real time instead of periodically. This application does not restrict the detection timing of the tasks.

[0175] S402: For each task whose execution status is "in progress", the abnormal task identification module obtains the planned number of slices and the actual number of slices of the task, and obtains the planned start time of the task.

[0176] In the embodiment of the present application, for each running task, the abnormal task identification module can obtain the planned number of shards and the actual number of shards of the task. In addition, optionally, the abnormal task identification module can determine the running task by the scheduling status of the task, such as the running task can be a task whose scheduling status is running (or called scheduling).

[0177] The planned number of slices of a task indicates the standard number of slices of the task, that is, the number of all slice tasks that need to be executed for the task. The actual number of slices of a task indicates the number of slice tasks that have been scheduled for the task, and the current state of the task is the number of slice tasks of target state 1. For example, if the state is the execution state, the target state 1 can be executing or completed (successful or failed). For another example, if the state is the scheduling state, the target state 1 can be running or completed (successful or failed), etc.

[0178] The planned start time of a task refers to the scheduling start time of the task, and may also refer to the execution start time of the task.

[0179] S403: When the actual number of shards is different from the planned number of shards and the time difference 1 between the current time and the planned start time is greater than a preset time threshold 1, the abnormal task identification module marks the task as an abnormal task.

[0180] Among them, the scheduling status of the abnormal task is abnormal.

[0181] In an embodiment of the present application, for each task whose execution status is in execution, the exception identification module can determine whether the actual number of shards of the task is different from the planned number of shards (that is, determine whether the actual number of shards of the task is less than the planned number of shards), and determine whether the time difference between the current time and the start time of the plan (that is, the planned execution time) is greater than the preset time threshold 1, so as to determine whether there is a timed-out and unscheduled shard task.

[0182] When the actual number of shards is different from the planned number of shards, and the time difference 1 (or called the first time difference) between the current time and the planned start time is greater than the preset time threshold 1 (or called the first preset time threshold), it indicates that there are timed-out and unprocessed shard tasks in the shard tasks of the task, and the timeouts are too many. Therefore, the abnormal task identification module determines that the task is an abnormal task, and marks the task as an abnormal task, thereby realizing the detection and marking of abnormal tasks.

[0183] When the actual number of shards is the same as the planned number of shards, or the time difference 1 between the current time and the planned start time is less than or equal to the preset time threshold 1, it indicates that there are no unprocessed shard tasks in the shard tasks of the task, or the shard tasks have not timed out too much. Therefore, the exception identification module can consider the task as a normal task and no marking is required.

[0184] Wherein, optionally, the marking process of the abnormal task may include that, when determining that the task is an abnormal task, the abnormal identification module may update the scheduling state of the task to an abnormal state. Optionally, the abnormal identification module may update the scheduling state of the task to an abnormal state by calling a post-listener to indicate that the abnormal task is a scheduling abnormality.

[0185] For example, the terminal device selects tasks whose execution status is running, and the tasks include: Fig.13 Task 1 and Fig.14 Task 2 shown. Fig.13 The planned execution time of Task 1 shown is 2024-09-23 08:00:00, the number of task slices (i.e., the planned number of slices, or standard slice data) is 4, and the execution status of the slice tasks is 0-T, 1-T, indicating that the execution status of slice 0 is successful, and the execution status of slice 1 is successful. The exception recognition module can determine that slices 2 and 3 of Task 1 are not executed, that is, slices 2 and 3 are not processed. Correspondingly, the actual number of slices of Task 1 is 2. Since the actual number of slices of Task 1 is 2, which is different from the number of task slices (i.e., 4), the exception recognition module can determine that the actual number of slices of Task 1 is different from the number of task slices.

[0186] And, the time difference 1 between the current time of the abnormal identification module and the planned execution time of task 1 (i.e., 2024-09-23 08:00:00). When the actual number of shards of task 1 is different from the number of task shards, and the time difference 1 is greater than the preset time threshold 1, it indicates that task 1 is an abnormal task. Therefore, the abnormal identification module can update the scheduling status of task 1 to an abnormal state (see Fig.15 Exception shown).

[0187] Similarly, Fig.14The scheduled execution time of Task 2 is 2024-09-23 08:00:00, the number of task shards is 1, and Task 2 does not actually need to be processed in shards. The task execution status is pending, indicating that Task 2 has not started processing, that is, the actual number of shards of Task 2 is 0.

[0188] And, the time difference 1 between the current time of the abnormal identification module and the planned execution time of task 2 (i.e., 2024-09-23 08:00:00). When the actual number of shards of task 2 is different from the number of task shards, and the time difference 1 is greater than the preset time threshold 1, it indicates that task 2 is an abnormal task. Therefore, the abnormal identification module can update the scheduling status of task 2 to an abnormal state (see Fig.16 Exception shown).

[0189] The abnormal tasks introduced above are mainly for tasks with scheduling exceptions. The scheduling exception may include not only the above-mentioned exception that there is a task that should be scheduled but has not been scheduled due to timeout, but also repeated scheduling, that is, the abnormality that the task's sharding tasks are repeatedly scheduled, but the repeated scheduling exception can be solved by checking the uniqueness of the task sharding. The repeated scheduling exception is not the focus of this application and is only briefly introduced here.

[0190] In some embodiments, after determining the abnormal task, a detailed log may be generated to facilitate subsequent analysis and debugging of related devices. The log records one or more of the abnormal cause of the abnormal task, the status of the abnormal task (i.e., scheduling status, execution status), and the status of the shard task of the abnormal task.

[0191] In some embodiments, after determining that a task is an abnormal task, the abnormal identification module may first add the task to the abnormal task list. Thereafter, for each abnormal task in the abnormal task list, the abnormal identification module may update the status (i.e., scheduling status, execution status) of the abnormal task.

[0192] The above describes the abnormal task identification process. After determining the abnormal task with scheduling abnormality, the abnormal task can be rescheduled through the abnormal task scheduling module to achieve abnormal task recovery, thereby improving the reliability of task scheduling. The operation process performed by the abnormal task scheduling module can be mainly divided into two parts, namely abnormal task scheduling and abnormal task status update. Fig.17 , which introduces the operation process in detail, that is, the scheduling process of abnormal tasks.

[0193] S501. The abnormal task scheduling module obtains the task to be scheduled.

[0194] Among them, the task to be scheduled means the task whose scheduling status is to be scheduled, that is, the task that has not started scheduling. The tasks to be scheduled may include tasks whose scheduled execution time is later than the current time. Of course, the tasks to be scheduled may also include tasks whose scheduled execution time is earlier than the current time but have not started scheduling. Optionally, the tasks to be scheduled may also be called tasks to be executed, that is, the tasks whose execution status is to be executed.

[0195] In some embodiments, the abnormal task scheduling module can search for the scheduled task from the task table, which records at least one task's information, including one or more of the planned execution time, the task's scheduling status, or the task's execution status. Of course, the information can also include other configuration information, such as the above Fig.13 The task identification number (identity document, ID) of the task shown, the execution status of the task, the execution status of the shard task, etc. In addition, it can be understood that Fig.13 The configuration parameters shown are only examples, and the configuration parameters can be set according to requirements. For example, the configuration parameters can also include the scheduling status of the task, the scheduling status of the task's fragmented tasks, etc.

[0196] S502: The abnormal task scheduling module obtains an abnormal task whose scheduling status is abnormal.

[0197] S503: Based on the sequence of the planned execution time of each task in the to-be-scheduled tasks and the abnormal tasks, the abnormal task scheduling module sorts the to-be-scheduled tasks and the abnormal tasks to obtain sorted tasks.

[0198] S504: The abnormal task scheduling module sends the sorted tasks to the task scheduling module.

[0199] In an embodiment of the present application, the abnormal task scheduling module can sort the abnormal tasks (or the first abnormal tasks) and the tasks to be scheduled (i.e., the tasks that need to be scheduled) in the order of the planned execution time to obtain the sorted tasks. Afterwards, the abnormal task scheduling module sends the sorted tasks (or the information of the sorted tasks (such as the task name, ID, etc.)) to the task scheduling module, so that the task scheduling module can schedule each task in the sorted tasks in sequence according to the order of priority, so that the tasks with earlier planned execution times can be scheduled faster, thereby ensuring the timeliness of task processing, thereby ensuring that the abnormal tasks can be scheduled in time, and improving the reliability of task scheduling. It should be understood that the abnormal task scheduling module can also sort the tasks in other ways, such as according to the priority of the task, the user's instructions, etc.

[0200] In some embodiments, after the above-mentioned abnormal task is rescheduled, the task scheduling module can update the status of the abnormal task (i.e., scheduling status, execution status) in a timely manner. Exemplarily, the task scheduling module can update the execution status of the abnormal task according to the execution status of the sliced ​​tasks of the abnormal task. When the execution status of each sliced ​​task of the abnormal task is successful, the task scheduling module updates the execution status of the abnormal task to success. When the execution status of the sliced ​​task of the abnormal task is failed, the task scheduling module updates the execution status of the abnormal task to failure. Similarly, the scheduling status update process of the task is similar to the execution status update process, which will not be repeated here. In addition, the specific update process of the status of the abnormal task can refer to the relevant content of the previous text.

[0201] Among them, optionally, since the execution status of some shard tasks of the abnormal task may be successful, there is no need to reschedule these shard tasks. Since the execution status of the shard tasks will be recorded when the shard tasks are executed, the task scheduling module can query the execution status of each shard task of the abnormal task. For each shard task, if the shard task is in a successful state, it indicates that the shard task has been successfully executed and does not need to be processed repeatedly. Therefore, the task scheduling module will not transmit the data of the shard task to the corresponding target node. For details, please refer to the task shard uniqueness check described above.

[0202] When the shard task is not in a successful state, it indicates that the shard task has not been successfully processed. Therefore, the task scheduling module can transparently transmit the data of the shard task to the corresponding target node, so that the target node can batch process the shard task based on the business logic code of the abnormal task to realize the execution of the shard task.

[0203] For example, as mentioned above Fig.15 The task 1 shown is an abnormal task. Afterwards, when the task 1 is rescheduled for execution, the task scheduling module determines that the execution status of the slice 0 and slice 1 of the task 1 are both successful. There is no need to reschedule the slice 0 and slice 1, but only to reschedule the slice 2 and slice 3. When the execution status of the slice 2 and slice 3 is successful, it indicates that all the slice tasks of the task 1 have been successfully executed. Accordingly, the task scheduling module can change the scheduling status of the task 1 from Fig.15 The exception shown is updated as Fig.18 Shown success.

[0204] In the embodiment of the present application, the status of the task and the sharded task of the task (such as the execution status and the scheduling status) can realize the accurate and rapid identification of abnormal tasks, so that the abnormal tasks can be scheduled in time to avoid the task being unprocessed for a long time, ensure the continuity of the business, and thus ensure the reliability of task execution. In addition, the abnormal task scheduling module reschedules the unfinished sharded tasks in a targeted manner based on the execution status of the sharded tasks of the abnormal task, avoids repeated scheduling of sharded tasks, and thus avoids unnecessary waste of resources.

[0205] In some embodiments, the abnormal task scheduling module or the abnormal task identification module can record the number of times a task is marked as an abnormal task, that is, the number of times the task is rescheduled. When the number is greater than a preset number threshold, it indicates that the task has been retried to schedule multiple times but still failed to schedule successfully. Therefore, in order to avoid wasting system resources, the abnormal task scheduling module can stop rescheduling the task, that is, suspend the execution of the task and release related resources.

[0206] It should be noted that the above-mentioned preset number threshold can be flexibly set according to requirements. In addition, other preset values ​​involved in this application can also be set according to requirements.

[0207] In the embodiment of the present application, the scheduling of abnormal tasks through the sorted tasks introduced above is only an example of a retry strategy, and the retry strategy can also be other strategies (such as node failover, compensation execution, etc.) to meet the scheduling requirements in different scenarios. The abnormal task scheduling module can trigger the retry mechanism for the abnormal task according to the retry strategy in the scenario corresponding to the abnormal task, realize the rescheduling of the task, and try to restore the abnormal task. In addition, when the abnormal task cannot be restored through the retry mechanism, the execution of the abnormal task can be terminated to release the resources occupied by the abnormal task and avoid unnecessary occupation of resources.

[0208] In some embodiments, not only may the above-mentioned scheduling anomaly occur in a task, but the scheduling may be normal but the execution may be abnormal. When the execution time of a task is greater than or equal to the preset execution time, it indicates that the task execution is too long and the task may have an execution anomaly. Then the task execution module can output an abnormal prompt message to enable relevant personnel to determine whether to terminate the task, so that the task is terminated after receiving the task cancellation request input by the user. Alternatively, other operations are performed according to the exception handling strategy corresponding to the preset execution anomaly. The exception handling strategy indicates that the task is terminated when the execution time of the task is greater than or equal to the preset execution time. Then the other operation can be a termination operation.

[0209] For example, for a multi-slice task (i.e., a task including multiple slice tasks), if the execution time of a slice task is greater than or equal to the preset execution time 1, the abnormality identification module can determine that the multi-slice task is executed too long. The execution time of the slice task can be the time from when the execution state of the slice task is switched to being executed to the current time.

[0210] For a single-slice task, when the execution time of the single-slice task is greater than or equal to the preset execution time 1, the abnormality identification module determines that the execution of the single-slice task is too long.

[0211] In some embodiments, the distributed task scheduling method (or task processing method) provided in the present application can be applied to data processing and analysis scenarios, such as large-scale data cleaning, transformation and loading (ETL) tasks, regular data statistics and analysis tasks, such as daily and monthly report generation, etc.

[0212] Alternatively, it can be applied to bill processing scenarios, such as bill generation and settlement tasks (e.g. regular generation of water, electricity, and gas bills), financial report generation and regular settlement, etc.

[0213] Alternatively, it can be applied to inventory management scenarios, such as inventory data synchronization.

[0214] Alternatively, it can be applied to automated operation and maintenance scenarios, such as regularly checking and cleaning log files, batch updating configuration files and patches, etc.

[0215] Alternatively, it can be applied to large-scale file processing scenarios, such as distributed file transfer tasks, file compression, decompression and archiving, etc.

[0216] Alternatively, it can be applied to Internet service scenarios, such as the collection and processing of user behavior logs, large-scale content review (such as image and video review), etc.

[0217] Alternatively, it can be applied to industrial IoT scenarios, such as regular collection and analysis of equipment data, monitoring of production line status and task allocation.

[0218] Alternatively, it can be applied to intelligent recommendation and advertising delivery scenarios, such as data calculation and model update of user recommendation algorithms, optimization of advertising delivery strategies, etc.

[0219] Alternatively, it can be applied to logistics and distribution management scenarios, such as dynamic optimization of distribution routes, order batch scheduling and distribution, etc.

[0220] In general, the distributed task scheduling method provided in this application can be widely used in various scenarios that require dynamic task allocation, efficient concurrent processing, anomaly detection and recovery, especially in data-intensive and task-complex businesses, and can significantly improve task scheduling and execution efficiency and enhance system reliability and stability.

[0221] In the embodiment of the present application, the distributed scheduling system not only records basic information about the task (such as task name, identification number, planned execution time, etc.), but also records detailed information such as the creator, update time, shard status, etc., to facilitate dynamic monitoring of tasks and troubleshooting.

[0222] In the embodiment of the present application, the present application provides a system (or task processing system) that combines a distributed scheduling framework and a distributed execution framework. The system not only utilizes the scheduling capability of the distributed scheduling framework, but also combines the execution capability (such as batch processing capability) of the distributed execution framework. Since the distributed scheduling framework and the distributed execution framework themselves can improve performance, the combination of the two can effectively improve the performance of the system. In addition, the reliability of the system can be guaranteed by identifying and scheduling abnormal tasks. The flexibility of the system can be guaranteed by terminating tasks.

[0223] The above-mentioned Elastic-job is only an example of a distributed scheduling framework, and Spring Batch is only an example of a distributed execution framework, which may also be other frameworks that can implement the corresponding functions. For example, Elastic-job can be replaced by frameworks such as Quartz, XXL-job, etc., and this application does not limit the specific framework. Accordingly, the solutions provided by this application, such as the implementation of the above-mentioned task scheduling, task execution, task cancellation, identification of abnormal tasks, and rescheduling solutions, rely on the related functions provided by Elastic-job and Spring Batch, such as the above-mentioned pre-listener, post-listener, stop signal, etc., are also only examples, and this application can implement this function through other codes that can implement this function.

[0224] It is understandable that the collection of business data involved in this application is authorized by the user. In addition, the operations performed by the above modules are only examples, and the above operations can also be performed by other modules. This application limits the specific modules of the above operations.

[0225] It should be noted that the execution order of the steps described above is only an example, and the present application does not limit the execution order of the above steps, and the execution order can be set according to the requirements. In addition, the modules involved in the present application can be deployed on the same node or on different nodes, and the present application does not limit them.

[0226] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of modules. It is understandable that the operations performed by the above modules are actually implemented by a distributed scheduling system, that is, a device (or electronic device, node, etc.). In order to implement the above functions, the electronic device includes a hardware structure and / or software module corresponding to each function. In combination with the units and algorithm steps of each example described in the embodiments disclosed in this application, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software.

[0227] Whether a function is executed in hardware or in a computer-driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present application.

[0228] Wherein, optionally, the electronic device may be a server. The number of the electronic devices may be one or more. The electronic device may execute the operations executed by some or all of the above modules.

[0229] In some embodiments, the present application provides an electronic device, which includes a memory and one or more processors; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device performs the task processing method as described above.

[0230] In some embodiments, the present application provides a chip, the chip comprising a communication interface and at least one processor:

[0231] The communication interface is used to input and / or output signaling or data;

[0232] The at least one processor is used to execute a computer program to implement the task processing method as described above.

[0233] In some embodiments, the present application provides a computer-readable storage medium, including computer instructions, which, when executed on an electronic device, enable the electronic device to perform the task processing method as described above.

[0234] In some embodiments, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the task processing method as described above.

[0235] The present application is an embodiment that can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0236] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0237] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0238] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0239] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0240] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0241] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A task processing method, characterized in that: A task processing system applied to a combination of a distributed scheduling framework and a distributed execution framework, the method comprising: In response to the first task being scheduled, the first task is sliced ​​through a distributed scheduling framework to obtain a plurality of first slice tasks of the first task; For each of the first shard tasks, assign the first shard task to a first target node, and update the scheduling state corresponding to the first shard task through a pre-processing function; By using the first target node, in combination with batch processing of the first shard task by using a distributed execution framework, an execution state of the first shard task is updated; When the execution status of all first shard tasks indicates completion, the scheduling status of the first task is set to completion through the post-processing function of the distributed scheduling framework, wherein the completion is success or failure.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a task cancellation request; wherein the task cancellation request is used to trigger the termination of the second task; According to the execution status of the second task, terminating the second task, and updating the execution status of the second task to terminated; Through the post-processing function, the scheduling status of the second task is updated to success.

3. The method according to claim 2, characterized in that The step of terminating the second task according to the execution status of the second task includes: In the case that the execution status of the second task is in execution, for each second shard task of the second task, if the execution status of the second shard task is in execution, the current data set of the second shard task is continued to be executed through the second target node where the second shard task is located; wherein the current data set belongs to a data set among multiple data sets obtained by dividing the second shard task when the second shard task is batch processed, and at least one data set among the multiple data sets includes data in the second shard task; After the execution of the current data set is completed, business processing of the new data set of the second sharding task is stopped.

4. The method according to claim 3, characterized in that: After stopping the business processing of the new data set of the second sharding task, the method further includes: Update the execution status of the second shard task to aborted; The updating the execution status of the second task to be terminated includes: When the execution status of each of the second-slice tasks is a terminal state, the execution status of the second task is updated to terminated; wherein the terminal state includes one or more of success, failure, or terminated.

5. The method according to claim 3 or 4, characterized in that: After updating the execution status of the second task to be suspended, the method further includes: In response to the second task being rescheduled, when the execution state of the second shard task of the second task is aborted, performing business processing on the data set of the second shard task that has not been business processed; When the execution status of the second fragment task of the second task is success or failure, no business processing is performed on the second fragment task.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: For each task whose execution status is "in execution", obtain the standard number of shards and the actual number of shards of the task, and obtain the planned execution time of the task; wherein the actual number of shards represents the number of shard tasks scheduled for the task; When the standard number of shards is different from the actual number of shards and a first time difference between the current time and the planned execution time is greater than a first preset time threshold, the task is marked as a first abnormal task; wherein the scheduling status of the first abnormal task is abnormal.

7. The method according to claim 6, characterized in that The method further comprises: Sort the tasks to be scheduled and the first abnormal tasks to obtain sorted tasks; The sorted tasks are scheduled in sequence.

8. The method according to claim 7, characterized in that The step of sorting the tasks to be scheduled and the first abnormal tasks to obtain sorted tasks includes: The tasks to be scheduled and the first abnormal task are sorted based on the sequence of the planned execution time of each task in the tasks to be scheduled and the first abnormal task.

9. The method according to claim 7 or 8, characterized in that: The sequentially scheduling the sorted tasks includes: When the execution status of the slice tasks of the scheduled task is success or failure, no business processing is performed on the slice tasks of the scheduled task.

10. A task processing system, characterized in that: The task processing system is a system that combines a distributed scheduling framework and a distributed execution framework, and the task processing system executes the task processing method according to any one of claims 1 to 9.

11. An electronic device, characterized in that: The electronic device includes a memory and one or more processors; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the task processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the task processing method as claimed in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task processing method according to any one of claims 1 to 9 is implemented.

Images