Timing task scheduling method and device, electronic equipment and storage medium

Abstract

The invention provides a timed task scheduling method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps of determining task data included in a target timed task in response to a task scheduling request for the target timed task; then, segmenting the task data based on a preset rule to obtain a plurality of segmented sub-task data corresponding to the sub-timed tasks; furthermore, the execution starting time of the sub timed tasks is determined, and the execution sequence of the sub timed tasks is determined based on the execution starting time. And when the execution starting time of the first executed sub-timed task arrives, starting a target scheduler to schedule each sub-timed task so as to distribute each sub-timed task to the corresponding task processing node. According to the method, the task processing accuracy is improved, and meanwhile system resource consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for scheduling a scheduled task. Background Art

[0002] The scheduling system of the big data platform is the key to ensuring the efficient execution of scheduled tasks. It uses technologies such as distributed computing models, resource management frameworks, and task scheduling algorithms to coordinate and manage the execution of large-scale scheduled tasks. With the development of big data technology, the scheduling system is also constantly being optimized and evolved to cope with the increasingly complex processing requirements of scheduled tasks.

[0003] The existing scheduling system continuously patrols the status of scheduled tasks and the information of scheduled tasks to be scheduled through the scheduler, and splits the scheduled tasks into corresponding sub-scheduled tasks according to predefined rules (such as the hash value of the task data), or simply splits the scheduled tasks according to the size or number of data blocks to realize the scheduling of scheduled tasks.

[0004] Therefore, the prior art has technical problems of low task processing accuracy and high system resource consumption. Summary of the invention

[0005] The present application provides a scheduling method, device, electronic device and storage medium for a timed task, which are used to solve the technical problems of low accuracy of task processing and high consumption of system resources in the prior art.

[0006] In a first aspect, the present application provides a method for scheduling a scheduled task, comprising:

[0007] In response to a task scheduling request for a target scheduled task, determining task data included in the target scheduled task;

[0008] The task data is segmented based on a preset rule to obtain a plurality of subtask data after segmentation; wherein the subtask data corresponds to the sub-scheduled tasks;

[0009] Determine the start execution time of the sub-scheduled tasks, and determine the execution order of each sub-scheduled task based on the start execution time;

[0010] When the start execution time of the first executed sub-scheduled task arrives, the target scheduler is started to schedule the sub-scheduled tasks so as to allocate the sub-scheduled tasks to the corresponding task processing nodes.

[0011] Optionally, determining the start execution time of the sub-scheduled task includes:

[0012] Obtain the task scheduling configuration corresponding to the sub-scheduled task;

[0013] According to the task scheduling configuration, the start execution time of the sub-scheduled task is determined based on a preset algorithm.

[0014] Optionally, the task data is stored in a first task database and a second task database; wherein the task data in the first task database is stored in a data table form, and the task data in the second task database is stored in a partition form;

[0015] The task data is segmented based on a preset rule to obtain a plurality of subtask data after segmentation, including:

[0016] Acquire a preset association field, and determine that the preset association field is a message key of the task data in the second task database; wherein the preset association field indicates an association relationship between the task data in the first task database and the task data in the second task database;

[0017] The task data in the first task database and the second task database are segmented based on the message key to obtain a plurality of segmented subtask data.

[0018] Optionally, the segmenting of the task data in the first task database and the second database based on the message key to obtain a plurality of segmented subtask data includes:

[0019] dividing the task data with the same message key in the second task database into the same partition of the second task database;

[0020] Based on the preset association field, determining the task data in the first task database that is associated with the task data in the second task database;

[0021] The task data in the first task database is sent to the partition where the task data in the associated second task database is located, to obtain subtask data of multiple partitions.

[0022] Optionally, allocating each of the scheduled sub-tasks to a corresponding task processing node includes:

[0023] Determine a task processing node for processing the sub-scheduled task, and computing resources included in the task processing node;

[0024] Determine the computing logic of the sub-scheduled tasks; and based on the computing resources and the computing logic, allocate each sub-scheduled task to a corresponding task processing node for processing.

[0025] Optionally, the method further includes:

[0026] Obtaining the subtask processing result fed back by the task processing node;

[0027] If the subtask processing result indicates that the sub-scheduled task processing has failed, then obtaining resource usage of other task processing nodes; based on the resource usage, selecting a target task processing node that can process the failed sub-scheduled task from the other task processing nodes;

[0028] The sub-scheduled tasks that failed to be processed are sent to the target task processing node for processing.

[0029] Optionally, the method further includes:

[0030] If the processing results of each subtask indicate that the sub-scheduled task is processed successfully, the subtask processing results of each sub-scheduled task are merged to obtain a final task processing result.

[0031] In a second aspect, the present application provides a scheduling device for a scheduled task, comprising:

[0032] A determination module, configured to determine, in response to a task scheduling request for a target scheduled task, task data included in the target scheduled task;

[0033] A segmentation module, used to segment the task data based on preset rules to obtain a plurality of subtask data after segmentation; wherein the subtask data corresponds to the sub-scheduled task;

[0034] The determination module is further used to determine the start execution time of the sub-scheduled tasks, and determine the execution order of each sub-scheduled task based on the start execution time;

[0035] The scheduling module is also used to start the target scheduler to schedule the sub-scheduled tasks when the start execution time of the first executed sub-scheduled task arrives, so as to allocate the sub-scheduled tasks to the corresponding task processing nodes.

[0036] Optionally, the determination module further includes: an acquisition module, configured to acquire a task scheduling configuration corresponding to the sub-scheduled task;

[0037] The determination module is further used to determine the start execution time of the sub-scheduled task based on a preset algorithm according to the task scheduling configuration.

[0038] Optionally, the task data is stored in a first task database and a second task database; wherein the task data in the first task database is stored in a data table form, and the task data in the second task database is stored in a partition form;

[0039] The acquisition module is also used to acquire preset associated fields;

[0040] The determination module is further used to determine that the preset association field is a message key of the task data in the second task database; wherein the preset association field indicates an association relationship between the task data in the first task database and the task data in the second task database;

[0041] The segmentation module is further used to segment the task data in the first task database and the second database based on the message key to obtain a plurality of subtask data after segmentation.

[0042] Optionally, the segmentation module is further used to divide the task data with the same message key in the second task database into the same partition of the second task database;

[0043] The determining module is further used to determine the task data in the first task database associated with the task data in the second task database based on the preset associated field;

[0044] The segmentation module also includes: a sending module, which is used to send the task data in the first task database to the partition where the task data in the associated second task database is located, to obtain subtask data of multiple partitions.

[0045] Optionally, the determination module is further used to determine a task processing node for processing the sub-scheduled task and computing resources included in the task processing node; determine the computing logic of the sub-scheduled task;

[0046] The scheduling module is further used to allocate each scheduled sub-task to a corresponding task processing node for processing based on the computing resources and the computing logic.

[0047] Optionally, the acquisition module is further used to acquire the subtask processing result fed back by the task processing node; if the subtask processing result indicates that the sub-scheduled task processing fails, then acquire the resource usage of other task processing nodes;

[0048] The determination module is further configured to select, based on the resource usage, a target task processing node that can process the failed scheduled sub-task from the other task processing nodes;

[0049] The sending module is further used to send the sub-scheduled task that fails to be processed to the target task processing node for processing.

[0050] Optionally, the scheduling device for the scheduled task further includes: a merging module, which is used to merge the subtask processing results of the sub-scheduled tasks to obtain a final task processing result if the sub-task processing results all indicate that the sub-scheduled task processing is successful.

[0051] In a third aspect, the present application provides an electronic device, including:

[0052] Memory and processor;

[0053] Wherein, the memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory to implement the scheduling method for timed tasks as described in the first aspect and various possible implementations.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the scheduling method of timed tasks as described in the first aspect and various possible implementation methods.

[0056] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the scheduling method of the timed task as described in the first aspect and various possible implementation methods.

[0057] The scheduling method, device, electronic device and storage medium of the timed task provided by the present application respond to the task scheduling request for the target timed task and determine the task data included in the target timed task. After that, the task data is segmented based on the preset rules to obtain multiple sub-task data corresponding to the sub-timed tasks after segmentation. Further, the start execution time of the sub-timed task is determined, and the execution order of each sub-timed task is determined based on the start execution time. By determining the start execution time of the multiple sub-timed tasks obtained by splitting, the dependency between the sub-tasks is determined, and then the correct execution order of the sub-timed tasks is obtained, avoiding the use of incomplete or incorrect data for processing, affecting the accuracy of task processing. When the start execution time of the first executed sub-timed task is reached, the target scheduler is started to schedule each sub-timed task to assign each sub-timed task to the corresponding task processing node. The target scheduler is dynamically awakened according to the start execution time of the first executed sub-timed task, and after all sub-timed tasks are assigned to the corresponding task processing nodes, the target scheduler is put into a dormant state, so that the target scheduler does not need to be in a working state all the time, saving system resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1Schematic diagram of the process of scheduling a scheduled task provided in the embodiment of the present application Figure 1 ;

[0060] Figure 2 A schematic diagram of a process for waking up a target scheduler provided in an embodiment of the present application;

[0061] Figure 3 A processing flow chart of a sub-scheduled task provided in an embodiment of the present application;

[0062] Figure 4 A system architecture diagram of a scheduling system for scheduled tasks provided in an embodiment of the present application;

[0063] Figure 5 A system architecture diagram of a scheduling system for timed tasks applicable to an embodiment of the present application;

[0064] Figure 6 An application scenario diagram of the timed task segmentation process provided in an embodiment of the present application;

[0065] Figure 7 Application scenarios of the scheduling method for scheduled tasks provided in the embodiments of the present application Figure 1 ;

[0066] Figure 8 Application scenarios of the scheduling method for scheduled tasks provided in the embodiments of the present application Figure 2 ;

[0067] Fig. 9 Schematic diagram of the process of scheduling a scheduled task provided in the embodiment of the present application Figure 2 ;

[0068] Fig.10 A schematic diagram of the structure of a scheduling device for scheduled tasks provided in an embodiment of the present application;

[0069] Fig.11 This is a hardware structure diagram of the electronic device provided for this application.

[0070] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0071] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0072] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0073] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0074] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0075] The scheduling system is one of the core components of the big data platform. It is responsible for ensuring the efficient execution of data processing. It plays a key role in the entire big data processing architecture and directly affects the efficiency and performance of data processing. The scheduling system relies on a variety of technologies, including distributed computing models, resource management frameworks, task scheduling algorithms, etc. These technologies work together to effectively allocate and manage computing resources in large-scale data processing environments.

[0076] With the continuous development of big data technology, data processing requirements are becoming more and more complex. In order to cope with the increasingly complex data processing requirements, the scheduling system is also constantly optimizing and evolving. At present, the task scheduling algorithm of the scheduling system includes both static scheduling that determines the resource allocation of all tasks before the task is executed, and dynamic scheduling that dynamically adjusts the task resource allocation according to the current scheduling system status.

[0077] Because business scenarios are usually more complex during big data development, a scheduled task can contain multiple sub-scheduled tasks, and there may be background tasks. That is, after the scheduled task is submitted to the scheduler through the scheduling system for task scheduling, the scheduled task will always run in the background.

[0078] Therefore, the existing scheduling system continuously polls the status of scheduled tasks and the information of scheduled tasks to be scheduled through the scheduler, and splits the scheduled tasks into corresponding sub-scheduled tasks according to predefined rules (such as the hash value of the task data), or simply splits the scheduled tasks according to the size or number of data blocks to realize the scheduling of scheduled tasks.

[0079] However, the existing scheduling methods for scheduled tasks are too simple in splitting scheduled tasks and do not fully consider the dependencies between scheduled sub-tasks or the complexity of task data, resulting in inaccurate task processing results. In addition, the scheduler needs to constantly poll the task status, and frequent status checks and task splitting operations will occupy a large amount of computing resources and network bandwidth, resulting in a high resource consumption of the scheduling system.

[0080] In response to the above technical problems, the inventors found that if the execution order of the sub-scheduled tasks is wrong, it may lead to the use of incomplete or incorrect data for processing, affecting the accuracy of task processing. Based on this, after the inventor splits the scheduled task into multiple sub-scheduled tasks, he determines the dependencies between the sub-tasks according to the start execution time of each sub-scheduled task, and then determines the correct execution order. Furthermore, the inventors took into account that the scheduler is used to schedule the sub-scheduled tasks, and the actual task processing process occurs at the task processing node. Therefore, when the task processing node processes the sub-scheduled tasks, the scheduler can be put into a dormant state, and when the start execution time of the first sub-scheduled task to be executed in the next scheduling is reached, the scheduler is woken up for scheduling, saving system resource consumption.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] An embodiment of the present application provides a method for scheduling a scheduled task. Figure 1 Schematic diagram of the process of scheduling a scheduled task provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the scheduling method of the scheduled task includes:

[0083] S101 : In response to a task scheduling request for a target scheduled task, determine task data included in the target scheduled task.

[0084] The task data is stored in the first task database and the second task database, and the task data in the first task database is stored in the form of a data table, and the task data in the second task database is stored in the form of partitions.

[0085] Exemplarily, the first task database may be a MySQL database, and the second task database may be a Kafka message queue. Furthermore, the MySQL database may be used as a main table to store the core task data of the target scheduled task; the Kafka message queue may be used as a dimension table to store dimension information related to the task data in the main table, so as to enrich and supplement the task data in the main table.

[0086] S102: Segment the task data based on preset rules to obtain a plurality of subtask data.

[0087] This step obtains a preset association field indicating the association relationship between the task data in the first task database and the task data in the second task database, and determines the preset association field as the message key of the task data in the second task database. Then, the task data in the first task database and the second task database are segmented according to the message key to obtain a plurality of segmented subtask data. Each subtask data corresponds to a sub-scheduled task.

[0088] It can be understood that the message key is an identifier used to identify the task data in the second task database. The role of the preset association field is to indicate the association relationship between the task data in the first task database and the task data in the second task database, which means that the preset association field has corresponding key values ​​in the first task database and the second task database, and these key values ​​are used to connect the related task data in the first task database and the second task database together. Exemplarily, the preset association field can be set to the task name.

[0089] Specifically, the task data in the second task database having the same message key are divided into the same partition in the second task database. Based on the preset association field, the task data in the first task database associated with the task data in the second task database is determined. Next, the task data in the first task database is sent to the partition where the task data in the associated second task database is located, and subtask data of multiple partitions is obtained.

[0090] In a specific example, task data with the same task name in the second task database are divided into the same partition, and task data in the first task database with the same task name corresponding to each partition of the second task database are determined. According to the task name, the task data in the first task database are sent to the corresponding partition of the second task database.

[0091] S103: Determine the start execution time of the sub-scheduled tasks, and determine the execution order of each sub-scheduled task based on the start execution time.

[0092] It is understandable that there may be dependencies between sub-scheduled tasks. For example, some sub-scheduled tasks can only start after other sub-scheduled tasks are completed, or the output of a sub-scheduled task may be the input of another sub-scheduled task. It can be seen that the dependency relationship will cause the sub-scheduled tasks to be executed in sequence, so the execution order of each sub-scheduled task needs to be determined by the start execution time of the sub-scheduled task.

[0093] Specifically, the task scheduling configuration of the sub-scheduled task may be obtained, and the start execution time of the sub-scheduled task may be determined based on the task scheduling configuration and a preset algorithm.

[0094] Among them, the task scheduling configuration usually contains parameters and constraints related to the execution of the sub-scheduled task, which may include the priority, dependency, resource requirements, time window, etc. of the sub-scheduled task. After obtaining the task scheduling configuration of the sub-scheduled task, you can understand the specific requirements and restrictions that need to be considered during the scheduling process of the sub-scheduled task.

[0095] The preset algorithm is used to calculate and schedule the execution time of sub-scheduled tasks, which may include earliest deadline priority, priority scheduling, dynamic scheduling, etc., which are not specifically limited here. Specifically, the preset algorithm will calculate the best start time for each sub-scheduled task, that is, the start execution time, based on the information in the task scheduling configuration of each sub-scheduled task.

[0096] In a possible implementation, the preset algorithm may be a built-in algorithm of the target scheduler, and after the target scheduled task is split into multiple sub-scheduled tasks, the start execution time of each sub-scheduled task is calculated by the built-in algorithm of the target scheduler. Then, the start time of the target scheduler is determined according to the start execution time of the sub-scheduled tasks, and then the target scheduler is put into a dormant state, and the target scheduler is awakened when the start time is reached.

[0097] Figure 2 A schematic diagram of a process for waking up a target scheduler provided in an embodiment of the present application, such as Figure 2 As shown, the specific steps to wake up the target scheduler are as follows:

[0098] Step a1, initializing the target scheduler;

[0099] Step a2: traverse all scheduled sub-tasks and obtain the task scheduling configuration of each scheduled sub-task;

[0100] Step a3: Calculate the start execution time of each scheduled sub-task and the start time of the target scheduler;

[0101] Step a4: encapsulate each scheduled sub-task and put the target scheduler into sleep mode.

[0102] Explanation: Encapsulation of sub-scheduled tasks is usually done to improve the modularity, manageability, and flexibility of the system. Encapsulation can hide complex task logic behind a simple interface, making the implementation details of the sub-scheduled task invisible to external systems or other tasks, which helps to simplify the design and understanding of the system.

[0103] Step a5: Determine whether the start time of the target scheduler has been reached. If so, wake up the target scheduler and return to step a2.

[0104] S104: When the start execution time of the first scheduled sub-task is reached, start the target scheduler to schedule each scheduled sub-task, so as to allocate each scheduled sub-task to a corresponding task processing node.

[0105] It should be noted that the start time of the target scheduler is usually slightly earlier than the start time of the first executed sub-scheduled task to ensure that the target scheduler has enough time to initialize. However, since the time interval between the start time of the target scheduler and the start time of the first executed sub-scheduled task is very short and can be ignored, the start time of the first executed sub-scheduled task can be regarded as the start time of the target scheduler.

[0106] In the process of scheduling each sub-scheduled task based on the target scheduler, it is necessary to determine the task processing node used to process the sub-scheduled task and the computing resources included in the task processing node. Further determine the computing logic of the sub-scheduled task, and based on the computing resources and computing logic, assign each sub-scheduled task to the corresponding task processing node for processing.

[0107] Furthermore, the subtask processing result fed back by the task processing node is obtained. If the subtask processing result indicates that the sub-scheduled task processing fails, the resource usage of other task processing nodes is obtained. Based on the resource usage, a target task processing node that can process the failed sub-scheduled task is selected from other task processing nodes, and the sub-scheduled task that failed to be processed is sent to the target task processing node for processing.

[0108] The reason for the failure of the sub-scheduled task processing may be an error in the processing process of the task processing node, or the processing time of the task processing node may be timed out. If an error occurs in the processing process of the task processing node, the sub-scheduled task is directly sent to other task processing nodes (target task nodes) for processing.

[0109] As for the failure of sub-scheduled task processing caused by the processing time timeout of the task processing node, data skew, insufficient resources, network delay, etc. will all cause the processing time timeout of the task processing node. Among them, data skew refers to the phenomenon of uneven data distribution. For example, the task data of the sub-scheduled task processed by the task processing node is huge, causing the processing time of the task processing node to time out. If the processing time of the task processing node times out due to data skew, other task processing nodes (target task nodes) can be requested to share the sub-task data that has not been processed.

[0110] In one possible implementation, Figure 3 The processing flow chart of the sub-timing task provided in the embodiment of the present application is as follows: Figure 3 As shown, processing a scheduled sub-task based on a task processing node includes the following steps:

[0111] Step b1, assigning the sub-scheduled tasks to the corresponding task processing nodes for processing;

[0112] Step b2, determine whether the sub-scheduled task has timed out;

[0113] If not, go to step b3; if so, go to step b4.

[0114] Step b3, determine whether the sub-scheduled task has been processed;

[0115] If not, continue to process the scheduled sub-task and return to step b2; if so, go to step b6.

[0116] Step b4: determine whether there is data skew;

[0117] If yes, go to step b5; if no, go to step b6.

[0118] Step b5, split the scheduled sub-task to obtain smaller scheduled sub-tasks, and go to step b7;

[0119] Step b6, determine whether the sub-scheduled task fails to process;

[0120] If not, the process ends; if so, go to step b7.

[0121] Step b7: Send the scheduled sub-task to other task processing nodes for processing, and return to step b2.

[0122] After all task processing nodes have processed all scheduled sub-tasks, if the sub-task processing results of each scheduled sub-task indicate that the sub-task processing is successful, the sub-task processing results of each scheduled sub-task are merged to obtain a final task processing result.

[0123] In summary, Figure 4 The system architecture diagram of the scheduling system for scheduled tasks provided in the embodiment of the present application is as follows: Figure 4 As shown, the scheduling system of the scheduled task includes a task configuration center, a target scheduler, a task splitting center, a task computing center and a task monitoring center. In a possible implementation, the task configuration center is responsible for sending the task scheduling configuration of multiple scheduled tasks to the target scheduler; the target scheduler is responsible for calculating the next start time and scheduling multiple scheduled tasks. After completing the above calculation and scheduling work, the target scheduler enters a dormant state. Before the scheduled task is scheduled to the specified task processing node, the scheduled task is split by the task splitting center to obtain the corresponding sub-scheduled tasks, and each sub-task is sent to the task processing node corresponding to the task computing center. After each task processing node completes the processing of the sub-scheduled task, the sub-task processing results of all sub-scheduled tasks are merged. In addition, the task monitoring center is responsible for monitoring the sub-task processing results, and rescheduling when it is detected that the sub-scheduled task processing fails. It should be noted that the messages between the task processing nodes of the task computing center are synchronized, and the task processing nodes can process sub-scheduled tasks that do not have dependencies in parallel.

[0124] The scheduling method of the timed task provided by the present application responds to the task scheduling request for the target timed task, determines the task data included in the target timed task, wherein the task data is stored in the first task database and the second task database. Afterwards, a preset association field indicating the association relationship between the first task database and the second task database is obtained, and the preset association field is determined as the message key of the task data in the second task database, and the task data in the first task database and the second task database are segmented based on the message key to obtain a plurality of subtask data after segmentation. Further, the task scheduling configuration of the sub-timed task is obtained, the task scheduling configuration is analyzed based on a preset algorithm, the start execution time of the sub-timed task is determined, and the execution order of each sub-timed task is determined based on the start execution time. By determining the start execution time of the multiple sub-timed tasks obtained by the split, the dependency relationship between the sub-tasks is determined, and then the correct execution order of the sub-timed tasks is obtained, thereby avoiding the use of incomplete or incorrect data for processing, which affects the accuracy of task processing. When the start execution time of the first executed sub-timed task is reached, the target scheduler is started to schedule each sub-timed task, and the task processing node for processing the sub-timed task is determined. According to the computing resources of the task processing node and the computing logic of the sub-scheduled tasks, each sub-scheduled task is assigned to the corresponding task processing node for processing. The target scheduler is dynamically awakened according to the start execution time of the first executed sub-scheduled task. After all sub-scheduled tasks are assigned to the corresponding task processing nodes, the target scheduler is put into a dormant state. Therefore, the target scheduler does not need to be in a working state all the time, saving system resource consumption.

[0125] Next, a specific embodiment is used to Figure 1 The scheduling method of the scheduled task is further explained.

[0126] Figure 5 is a system architecture diagram of a scheduling system for timed tasks applicable to an embodiment of the present application, such as Figure 5 As shown, the scheduling system of the scheduled task includes the scheduler job_launcher, the coordinator coordinator, the application monitoring component APP_monitor, the application protection component APP_guard, the distributed coordination service zookeeper, and the MySQL database and the kafka message queue.

[0127] Among them, the MySQL database and the kafka message queue are used to store the task data of scheduled tasks. The scheduler job_launcher is the scheduler process, and the coordinator is used for the initialization and load balancing management of the scheduler job_launcher. The application monitoring component APP_monitor is used to monitor the survival of each component, and the application protection component APP_guard is to ensure that the component can be restarted in time after a failure. The scheduler job_launcher reads all task data and task scheduling configurations from the MySQL database and the kafka message queue for scheduling, and stores the status information required for task scheduling, such as scheduling nodes, task names, checkpoints, etc. in zookeeper to ensure the high availability of the scheduling system.

[0128] It should be noted that the entire scheduling system is managed using a lightweight Python process, that is, the core logic and task scheduling functions of the scheduling system are written in Python, which makes the development and maintenance of the scheduling system easier by taking advantage of the simplicity and rich library support of Python. In addition, the scheduling system relies on Zookeeper for distributed services to help the scheduling system maintain consistency and high availability on multiple machines.

[0129] In addition, the scheduling system of the scheduled task of the present application abandons the traditional microservice architecture and uses a plug-in approach to provide scheduling services. The plug-in architecture allows the scheduling system to dynamically expand functions by loading and unloading plug-ins, and can flexibly add different scheduling algorithms or functional modules, thereby supporting different types of scheduled tasks, as well as customized complex scheduled tasks to meet different task requirements.

[0130] Through the plug-in architecture of the scheduling system, different types of scheduled tasks can be processed, and the task data can be divided into multiple sub-task data according to the processing logic of the scheduled task itself to obtain multiple corresponding sub-scheduled tasks. At the same time, an interface for custom splitting scheduled tasks is also provided for users to implement their own splitting logic. It should be noted that the scheduling system with a plug-in architecture is not only applicable to batch processing tasks, but also to stream processing tasks.

[0131] Since the task data of scheduled tasks are stored in the MySQL database and the Kafka message queue, where MySQL is the main table and the Kafka message queue is the dimension table, when splitting the scheduled tasks, the preset associated fields (such as task name) of the MySQL database and the Kafka message queue are used as the message keys of the Kafka message queue, and the task data with the same message key in the Kafka message queue are divided into the same partition of the Kafka message queue.

[0132] Next, the task data in the MySQL database is segmented according to the partitioning rules in the Kafka message queue, and the segmented task data is matched one by one with the corresponding partitions in the Kafka message queue. The subtask data corresponding to each partition is the content that the sub-scheduled task needs to actually process, and the processing logic is also executed by each sub-scheduled task.

[0133] For example, Figure 6 The application scenario diagram of the timed task segmentation process provided in the embodiment of the present application is as follows: Figure 6 As shown in the figure, the task data of the Kafka message queue will be partitioned according to the message key. Among them, the task data with message keys 1 and 3 will be assigned to the partitions corresponding to node 1 / node 3 / node 5, and the task data with message keys 2 and 4 will be assigned to the partitions corresponding to node 2 / node 4 / node 6. Task splitting and orchestration will pull the task data with task names 1 and 3 in the MySQL database to the partitions corresponding to node 1 / node 3 / node 5 for convergence calculation, and pull the task data with task names 2 and 4 to the partitions corresponding to node 2 / node 4 / node 6 for convergence calculation.

[0134] Explanatory, the task data stored in the three partitions corresponding to nodes 1, 3, and 5 are the same, and the task data stored in the three partitions corresponding to nodes 2, 4, and 6 are also the same. This is achieved through the replication mechanism of the Kafka message queue. Each partition can have multiple copies to ensure high availability and fault tolerance of the data. Therefore, the partitions corresponding to nodes 3 and 5 are copies of the partition corresponding to node 1, and the partitions corresponding to nodes 4 and 6 are copies of the partition corresponding to node 2. In the actual execution process, data aggregation calculations are performed preferentially on nodes 1 and 2.

[0135] It should be noted that after being split, the task data in the MySQL database can be sent to the corresponding partition of the Kafka message queue, and together with the task data of the Kafka message queue in the partition, form subtask data. In addition, according to the association between the MySQL database and the Kafka message queue, the task data in the MySQL database and the task data in the Kafka message queue can be sent to the data pool and split to obtain multiple subtask data, which is not specifically limited here. Among them, the data pool is a data aggregation platform or service used to collect, integrate and analyze data from different sources.

[0136] Furthermore, after obtaining multiple scheduled sub-tasks, the scheduled sub-tasks will be assigned to the task computing center of the server where the kafka message queue is located. The task computing center includes multiple task processing nodes. The task computing center will receive the computing resource information of the task processing nodes fed back by the task monitoring center every 5 seconds, and then schedule the scheduled sub-tasks according to the computing resource usage of the task processing nodes and the running status of the server. Explanatory, the task computing center can not only obtain the computing resource information of the task processing nodes on the server where it is located, but also obtain the computing resource information of the task processing nodes on other servers.

[0137] Specifically, if the task processing node that processes the sub-scheduled task and the Kafka message queue do not share a server, then the task computing center where the task processing node that processes the sub-scheduled task is located needs to pull the sub-task data across servers. Figure 7 Application scenarios of the scheduling method for scheduled tasks provided in the embodiments of the present application Figure 1 ,like Figure 7 As shown, the task computing center where the task processing node that processes the sub-scheduled task is located will remotely and randomly pull the sub-task data from server 1 or server 2. Among them, server 1 and server 2 are active-standby server architectures, and the sub-task data in server 1 and server 2 are the same.

[0138] If the task processing node that processes the scheduled subtasks shares a server with the Kafka message queue, it is only necessary to reasonably allocate the scheduled subtasks according to the computing resources of all the task processing nodes in the server to avoid data transmission across servers. When the server where the Kafka message queue is located is in normal state, it is more efficient to use the task processing node of the task computing center in the server to process the subtasks without allocating the subtasks to other servers that are farther away. In general, the computing logic should be moved first rather than the task data, and network transmission should be minimized to improve overall efficiency.

[0139] Figure 8 Application scenarios of the scheduling method for scheduled tasks provided in the embodiments of the present application Figure 2 ,like Figure 8 As shown, assume that server 1 is the primary server and server 2 is the backup server. Although the subtask data in server 1 and server 2 are the same, the task computing center in server 1 will preferentially pull the subtask data of server 1 instead of the subtask data of server 2.

[0140] Furthermore, after each task processing node receives the computing logic of the sub-scheduled task distributed by the task computing center, it starts to execute the task. At the same time, the task computing center maintains information communication with the task processing node. The purpose is that when a sub-scheduled task fails to be processed, the task processing node that processes the sub-scheduled task will actively send a message to other task processing nodes, requesting other task processing nodes to process the failed sub-scheduled task. In addition, if the processing time of a sub-scheduled task times out, the task processing node that processes the sub-scheduled task can request other task processing nodes to share the computing task, thereby reducing data skew.

[0141] In one possible implementation, Fig. 9 Schematic diagram of the process of scheduling a scheduled task provided in the embodiment of the present application Figure 2 ,like Fig. 9 As shown, the scheduling method of the scheduled task specifically includes the following steps:

[0142] Step c1, split the scheduled task to obtain multiple scheduled sub-tasks;

[0143] Step c2, configure the sub-scheduled task;

[0144] Step c3: assign the scheduled subtask to the task processing node for processing;

[0145] Step c4, determine whether the sub-scheduled task fails to process;

[0146] If yes, go to step c5; if no, go to step c7.

[0147] Step c5: perform fault-tolerance processing, that is, send the failed scheduled sub-task to other task processing nodes for processing, and execute step c6;

[0148] Step c6: determine whether the number of times other task processing nodes process the failed scheduled sub-task exceeds the maximum number of retries;

[0149] If yes, then end the process; if no, then return to step c4.

[0150] Step c7, merging the subtask processing results of the sub-scheduled tasks;

[0151] Step c8, determining whether the merge fails;

[0152] If yes, return to step c7; if no, end the process.

[0153] The scheduling system of this embodiment only uses the python resident process as a background service, and only relies on zookeeper for state storage, without relying on other complex frameworks, reducing the complexity of the scheduling system. At the same time, the plug-in architecture enables the scheduling system to support different types of scheduled tasks, as well as customized complex scheduled tasks, to meet different task requirements. In addition, the scheduling system splits the target scheduled task into sub-scheduled tasks to run, and automatically handles the dependencies between sub-scheduled tasks, improving the accuracy of task processing, and the target scheduler will also be dynamically awakened when the start execution time of the first executed sub-scheduled task is reached, which not only avoids the target scheduler's cyclic idling, but also saves the resource consumption of the scheduling system.

[0154] Fig.10 A schematic diagram of the structure of a scheduling device for a scheduled task provided in an embodiment of the present application, such as Fig.10 As shown, the scheduling device 1000 for the scheduled task includes: a determination module 1001, a segmentation module 1002, and a scheduling module 1003;

[0155] The determination module 1001 is used to determine the task data included in the target scheduled task in response to the task scheduling request for the target scheduled task;

[0156] The segmentation module 1002 is used to segment the task data based on a preset rule to obtain a plurality of subtask data after segmentation; wherein the subtask data corresponds to the sub-scheduled task;

[0157] The determination module 1001 is further used to determine the start execution time of the sub-scheduled tasks, and determine the execution order of each sub-scheduled task based on the start execution time;

[0158] The scheduling module 1003 is further used to start the target scheduler to schedule each sub-scheduled task when the start execution time of the first executed sub-scheduled task arrives, so as to allocate each sub-scheduled task to a corresponding task processing node.

[0159] Optionally, the determination module 1001 further includes: an acquisition module 1004, configured to acquire a task scheduling configuration corresponding to a scheduled sub-task;

[0160] The determination module 1001 is further configured to determine the start execution time of the sub-scheduled task based on a preset algorithm according to the task scheduling configuration.

[0161] Optionally, the task data is stored in a first task database and a second task database; wherein the task data in the first task database is stored in a data table form, and the task data in the second task database is stored in a partition form;

[0162] The acquisition module 1004 is also used to acquire the preset associated fields;

[0163] The determination module 1001 is further used to determine that the preset association field is a message key of the task data in the second task database; wherein the preset association field indicates the association relationship between the task data in the first task database and the task data in the second task database;

[0164] The segmentation module 1002 is further used to segment the task data in the first task database and the second database based on the message key to obtain a plurality of subtask data after segmentation.

[0165] Optionally, the segmentation module 1002 is further configured to divide the task data with the same message key in the second task database into the same partition of the second task database;

[0166] The determination module 1001 is further used to determine the task data in the first task database associated with the task data in the second task database based on the preset associated field;

[0167] The segmentation module 1002 further includes: a sending module 1005, which is used to send the task data in the first task database to the partition where the task data in the associated second task database is located, to obtain subtask data of multiple partitions.

[0168] Optionally, the determination module 1001 is further used to determine a task processing node for processing a sub-scheduled task and computing resources included in the task processing node; determine the computing logic of the sub-scheduled task;

[0169] The scheduling module 1003 is also used to allocate each scheduled sub-task to a corresponding task processing node for processing based on computing resources and computing logic.

[0170] Optionally, the acquisition module 1004 is further used to acquire the subtask processing result fed back by the task processing node; if the subtask processing result indicates that the sub-scheduled task processing fails, then acquire the resource usage of other task processing nodes;

[0171] The determination module 1001 is further used to select a target task processing node that can process the failed scheduled sub-task from other task processing nodes based on resource usage;

[0172] The sending module 1005 is further used to send the sub-scheduled task that failed to be processed to the target task processing node for processing.

[0173] Optionally, the scheduled task scheduling device 1000 further includes: a merging module 1006, which is used to merge the subtask processing results of each sub-scheduled task if the sub-task processing results all indicate that the sub-scheduled task is processed successfully, to obtain a final task processing result.

[0174] The scheduling device for scheduled tasks provided in the embodiments of the present application can be used to execute the scheduling method for scheduled tasks in any of the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0175] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0176] Fig.11 The hardware structure diagram of the electronic device provided in this application. Fig.11 As shown, the electronic device 1100 includes:

[0177] Processor 1101 and memory 1102;

[0178] Memory stores computer-executable instructions;

[0179] The processor executes the computer execution instructions stored in the memory 1102, so that the electronic device executes the scheduling method of the timed task as described above.

[0180] It should be understood that the processor 1101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The memory 1102 may include a high-speed random access memory (RAM), or may also include a non-volatile memory (NVM), such as at least one disk storage, or may be a USB flash drive, a mobile hard disk, a read-only memory, a disk, or an optical disk.

[0181] The electronic device provided in the embodiment of the present application can be used to execute the scheduling method of the timed task provided in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0182] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the scheduling method of the timed task as described above.

[0183] The embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to implement the scheduling method of the timed task as mentioned above.

[0184] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0185] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0186] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0187] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0188] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0189] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0190] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0192] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for scheduling a scheduled task, characterized in that: include: In response to a task scheduling request for a target scheduled task, determining task data included in the target scheduled task; The task data is segmented based on a preset rule to obtain a plurality of subtask data after segmentation; wherein the subtask data corresponds to the sub-scheduled tasks; Determine the start execution time of the sub-scheduled tasks, and determine the execution order of each sub-scheduled task based on the start execution time; When the start execution time of the first executed sub-scheduled task arrives, the target scheduler is started to schedule the sub-scheduled tasks so as to allocate the sub-scheduled tasks to the corresponding task processing nodes.

2. The method according to claim 1, characterized in that: Determining the start execution time of the sub-scheduled task includes: Obtain the task scheduling configuration corresponding to the sub-scheduled task; According to the task scheduling configuration, the start execution time of the sub-scheduled task is determined based on a preset algorithm.

3. The method according to claim 1, characterized in that The data of the task data is stored in a first task database and a second task database; wherein the task data in the first task database is stored in the form of a data table, and the task data in the second task database is stored in the form of a partition; The task data is segmented based on a preset rule to obtain a plurality of subtask data after segmentation, including: Acquire a preset association field, and determine that the preset association field is a message key of the task data in the second task database; wherein the preset association field indicates an association relationship between the task data in the first task database and the task data in the second task database; The task data in the first task database and the second task database are segmented based on the message key to obtain a plurality of segmented subtask data.

4. The method according to claim 3, characterized in that: The step of segmenting the task data in the first task database and the second task database based on the message key to obtain a plurality of segmented subtask data includes: dividing the task data with the same message key in the second task database into the same partition of the second task database; Based on the preset association field, determining the task data in the first task database that is associated with the task data in the second task database; The task data in the first task database is sent to the partition where the task data in the associated second task database is located, to obtain subtask data of multiple partitions.

5. The method according to claim 1, characterized in that The method of allocating each of the scheduled sub-tasks to a corresponding task processing node includes: Determine a task processing node for processing the sub-scheduled task, and computing resources included in the task processing node; Determine the computing logic of the sub-scheduled tasks; and based on the computing resources and the computing logic, allocate each sub-scheduled task to a corresponding task processing node for processing.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining the subtask processing result fed back by the task processing node; If the subtask processing result indicates that the sub-scheduled task processing has failed, then obtaining resource usage of other task processing nodes; based on the resource usage, selecting a target task processing node that can process the failed sub-scheduled task from the other task processing nodes; The sub-scheduled tasks that failed to be processed are sent to the target task processing node for processing.

7. The method according to claim 6, characterized in that The method further comprises: If the processing results of each subtask indicate that the sub-scheduled task is processed successfully, the subtask processing results of each sub-scheduled task are merged to obtain a final task processing result.

8. A scheduling device for a scheduled task, characterized in that: include: A determination module, configured to determine, in response to a task scheduling request for a target scheduled task, task data included in the target scheduled task; A segmentation module, used to segment the task data based on preset rules to obtain a plurality of subtask data after segmentation; wherein the subtask data corresponds to the sub-scheduled task; The determination module is further used to determine the start execution time of the sub-scheduled tasks, and determine the execution order of each sub-scheduled task based on the start execution time; The scheduling module is also used to start the target scheduler to schedule the sub-scheduled tasks when the start execution time of the first executed sub-scheduled task arrives, so as to allocate the sub-scheduled tasks to the corresponding task processing nodes.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the scheduling method for a timed task as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the scheduling method for a timed task according to any one of claims 1 to 7 when executed by a processor.

Images