Data processing task scheduling method, device, computer equipment and storage medium

Abstract

The present application provides a method, device, computer equipment and storage medium for scheduling data processing tasks. The method includes: in response to a scheduling start instruction, generating an initial scheduling log, and taking the data processing task that meets the startup conditions as the target task of this scheduling; calling the target task for execution according to the preset execution mechanism, generating an initial task log for each called target task, and updating the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed; in response to a log check instruction, checking the scheduling log and task log to determine whether there is an interrupted target task; the log check instruction is performed periodically according to the first interval after the scheduling system is started; if so, starting from the interrupted target task, continue to execute the calling target task. This method can greatly reduce the pressure on system operation and maintenance personnel and ensure that the system can effectively and stably operate for a long time.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for scheduling data processing tasks. Background Art

[0002] In the current business software environment, business systems basically need to process data in the system every day, such as data cleaning, format conversion, report making, etc. To complete this work, a series of repeatable operations are required. In order to complete this work with maximum efficiency and automation, the scheduling system was invented to handle this heavy work in a planned and automatic manner.

[0003] Since daily data processing is the prerequisite for ensuring the accurate operation of the system, ensuring the smooth operation of the scheduling system is a key factor. When the scheduling system crashes for some reason, the daemon process can be used at the system level to restart the scheduling system process, but after the system restarts, the original running tasks cannot be automatically restarted. At the same time, the scheduling system and the business system need to be connected through the network. During the scheduling execution process, the communication channel between the scheduling system and the business system may be lost. After the channel is lost, the originally executed program will also be interrupted and unable to return information to the scheduling system. Then the scheduling system always thinks that the program is being executed but has actually been interrupted. In the above situation, traditional technology can only rely on manual inspection of the execution status and then restart the interrupted task. If the operation and maintenance personnel cannot handle it in time, the daily data processing tasks will not be completed in time, thus affecting the normal operation of the system. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical deficiencies, especially the technical defect that the prior art cannot automatically restart the interrupted task.

[0005] In a first aspect, the present application provides a method for scheduling a data processing task, which is applied to a scheduling system, where the scheduling system is used to retrieve data from a business system to perform a data processing task. The method for scheduling a data processing task includes:

[0006] In response to the scheduling start instruction, an initial scheduling log is generated, and the data processing tasks that meet the start conditions are used as the target tasks of this scheduling;

[0007] Call the target task for execution according to the preset execution mechanism, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed;

[0008] In response to the log check instruction, check the scheduling log and the task log to determine whether there is an interrupted target task; the log check instruction is performed periodically according to the first interval after the scheduling system is started;

[0009] If so, start from the interrupted target task, return to call the target task execution according to the preset execution mechanism, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed.

[0010] In one embodiment, the scheduling log and the task log include the execution status;

[0011] During the execution of each target task, the corresponding task log and scheduling log are updated until all target tasks are completed, including:

[0012] After the first target task is called, the execution status is updated to "Executing" in the scheduling log;

[0013] For any called target task, in the corresponding task log, the execution status is updated to "in progress", and after the target task is completed, the execution status is updated to "completed";

[0014] When all target tasks are called and the execution status of the corresponding task logs are all completed, the execution status is updated to completed in the scheduling log.

[0015] In one of the embodiments, the scheduling log and the task log include the heartbeat time;

[0016] During the execution of each target task, the corresponding task log and scheduling log are updated until all target tasks are completed, including:

[0017] For the task log whose execution status is in execution, the heartbeat time is updated to the current time according to the second interval;

[0018] For the scheduling log whose execution status is "Executing", the heartbeat time is updated to the current time according to the third interval.

[0019] In one embodiment, checking the scheduling log and the task log to determine whether there is an interrupted target task includes:

[0020] Determine whether there is a scheduling log whose execution status is in execution, and whether the difference between the heartbeat time and the current time is greater than the third interval. If so, determine it as the target scheduling log;

[0021] In the task log corresponding to the target scheduling log, determine whether there is a task log with an execution status of being executed, and whether the difference between the heartbeat time and the current time is greater than the second interval. If so, determine the corresponding target task as an interrupted target task.

[0022] In one embodiment, the scheduling log and the task log include a start time and an end time;

[0023] During the execution of each target task, the corresponding task log and scheduling log are updated until all target tasks are completed, including:

[0024] After the first target task is called, the start time is updated to the current time in the scheduling log;

[0025] For any called target task, in the corresponding task log, the start time is updated to the current time, and after the target task is completed, the end time is updated to the current time;

[0026] When all target tasks are called and the execution status of the corresponding task logs are all completed, the end time is updated to the current time in the scheduling log.

[0027] In one embodiment, the preset execution mechanism includes:

[0028] Obtain the resource utilization of the current scheduling system and determine the maximum data processing capacity based on the resource utilization;

[0029] Based on the maximum data processing volume, the target task with the largest data volume is preferentially called from the target tasks.

[0030] In one embodiment, each target task carries a tag related to the priority, and the preset execution mechanism includes:

[0031] The target tasks are called and executed in descending order of priority.

[0032] In a second aspect, the present application provides a scheduling device for a data processing task, which is applied to a scheduling system, and the scheduling system is used to retrieve data from a business system to perform a data processing task. The scheduling device for the data processing task includes:

[0033] The first response module is used to generate an initial scheduling log in response to the scheduling start instruction, and take the data processing task that meets the start condition as the target task of this scheduling;

[0034] The first processing module is used to call the target task for execution according to the preset execution mechanism, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed;

[0035] A second response module is used to check the scheduling log and the task log in response to the log check instruction to determine whether there is an interrupted target task; the log check instruction is performed periodically according to the first interval after the scheduling system is started;

[0036] The second processing module is used to start from the interrupted target task and return to call the target task for execution according to the preset execution mechanism when there is an interrupted target task, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks have been completed.

[0037] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the steps of the scheduling method for data processing tasks in any of the above-mentioned embodiments are performed.

[0038] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the scheduling method for data processing tasks in any of the above-mentioned embodiments.

[0039] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0040] By generating a scheduling log at the beginning of scheduling, recording the relevant information of this scheduling, and determining the target tasks included in this scheduling process, when the scheduling system calls each target task according to the preset mechanism, a task log is created for each target task to record the execution status, and the task log and scheduling log are updated in real time during the execution process to track the progress of task execution. At the same time, a periodic log check mechanism is set up to check whether there are any task interruptions. If an interrupted task is found, the scheduling process is restarted from the task, and the log record and update are repeated until all tasks are successfully completed. The scheme uses a log recording mechanism to clearly and completely track a scheduling process and the execution of each task. The set log check mechanism can automatically discover and reschedule interrupted tasks. Whether the scheduling system crashes suddenly or the communication channel is lost due to network fluctuations, etc., any of which triggers the interruption of the target task, the interrupted task can be automatically restored. It can greatly reduce the pressure on system operation and maintenance personnel and ensure that the system can effectively and stably operate for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A flowchart of a method for scheduling data processing tasks provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a module of a scheduling device for data processing tasks provided in an embodiment of the present application;

[0044] Figure 3 This is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The present application provides a method for scheduling data processing tasks, which is applied to a scheduling system, and the scheduling system is used to retrieve data from a business system to perform data processing tasks. It can be understood that the business system is a system that produces data, and the scheduling system is a system that pulls data from the business system and executes set data processing tasks on the pulled data. The connection between the business system and the scheduling system can adopt the method of TCP long connection, which is convenient for the scheduling system to retrieve data from the business system. The data processing tasks here can be designed according to the needs of the business part, and may include data cleaning, calculating indicators according to preset formulas, generating reports, etc. In order to ensure that interrupted tasks can be automatically restored when accidents such as network fluctuations and scheduling system crashes occur, this embodiment provides a method for scheduling data processing tasks, please refer to Figure 1 , the method includes steps S102 to S108.

[0047] S102, in response to the scheduling start instruction, generating an initial scheduling log, and taking the data processing tasks that meet the start conditions as the target tasks of this scheduling.

[0048] It can be understood that the scheduling start instruction is a command to start a scheduling process. The scheduling start instruction can be manually issued by the staff or triggered according to the business process. The scheduling log is a log that records the execution status of this scheduling process. The execution status here can include the execution status, such as completed and executing, start time, end time, etc. In order to facilitate the distinction between each scheduling, the identifier of this scheduling, such as the scheduling ID number, can also be included. After starting the scheduling process, an initial scheduling log will be formed first, which is still in a blank state and will gradually write information as the scheduling process proceeds. Each scheduling process is composed of multiple data processing tasks. The scheduling system will determine which data processing tasks need to be executed for this scheduling based on the pre-configured start conditions. The configuration of the start condition can be set by the business personnel. It can be time-driven, data-driven, event-driven, dependency-driven, etc. Time-driven refers to the arrival of the time when the task needs to be executed. Data-driven refers to the triggering when the data that the task depends on reaches a certain threshold or meets a certain condition, such as triggering the processing when the data volume accumulates to 100G. Event-driven refers to driving the execution of tasks when external events occur, such as triggering the transaction processing task when a transaction request is received. Dependency-driven means that the task is driven to execute after the predecessor task is completed.

[0049] S104, calling the target task for execution according to the preset execution mechanism, generating an initial task log for each called target task, and updating the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed.

[0050] It can be understood that the preset execution mechanism is a mechanism for determining the order of calling each target task in the scheduling system. The task log is a log used to record the execution status of a single data processing task. After a single data processing task is called, it is necessary to record the relevant information of the target task before actually starting to execute the data processing task. The execution status here may include execution status, such as completed and executing, start time, end time, etc. In order to facilitate the distinction between each target task, the identifier of the target task, such as the task id number, may also be included. In order to determine the scheduling process to which the target task belongs, the identifier of the corresponding scheduling process, such as the scheduling id number, may also be included. When any target task is executed, it is necessary to keep updating the task log corresponding to the task according to the execution progress of the task. At the same time, since the scheduling process is composed of multiple target tasks, the execution status of each target task will determine the execution status of the entire scheduling process. Therefore, it is also necessary to keep updating the scheduling log according to the execution progress of each target task. The update of a single task log can be stopped when the corresponding target task is completed, and the update of the scheduling log needs to be maintained until all target tasks are completed. Before starting to schedule the first target task, the scheduling system can first create a task pool and put the task IDs of all target tasks into the task pool. Whenever a target task is called, the corresponding task ID in the task pool is cleared. When the task pool is cleared, it can be determined that all target tasks have been called. If the task logs of all target tasks reflect that the tasks have been completed, the scheduling system can determine that all target tasks have been completed.

[0051] S106, in response to the log check instruction, check the scheduling log and the task log to determine whether there is an interrupted target task. The log check instruction is performed periodically at a first interval after the scheduling system is started.

[0052] It can be understood that what step S102 and step S104 illustrate is the log recording mechanism in the normal execution process of the system, and in order to ensure that the interrupted task can be automatically restored, a log check command that is performed periodically according to the first interval is set in the scheduling system. After each scheduling system is started, the log check will be triggered, and the records in the scheduling log and the task log will be used to determine whether there is an interrupted target task. If there is, it will proceed to step S108, and the interrupted target task will be used as the first target task to be called, and the scheduling will be continued according to the process of step S106. In this way, when the scheduling system is started and runs normally, the inspection of the scheduling log and the task log in each cycle will not affect the entire scheduling process. If the scheduling system is restarted, it will be reflected in the task log and / or the scheduling log, and the interrupted target task will be re-executed at this time, a new task log will be created for it, and it will continue to be executed according to the execution progress of the scheduling process.

[0053] S108, if yes, start from the interrupted target task, return to call the target task for execution according to the preset execution mechanism, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed.

[0054] This solution generates a scheduling log at the beginning of scheduling, records the relevant information of this scheduling, and determines the target tasks included in this scheduling process. When the scheduling system calls each target task according to the preset mechanism, it creates a task log for each target task to record the execution status, and updates the task log and scheduling log in real time during the execution process to track the progress of task execution. At the same time, a periodic log check mechanism is set up to check whether there are any task interruptions. If an interrupted task is found, the scheduling process is restarted from the task, and the log record and update are repeated until all tasks are successfully completed. This solution uses a log recording mechanism to clearly and completely track a scheduling process and the execution of each task. The set log check mechanism can automatically discover and reschedule interrupted tasks. Whether the scheduling system crashes suddenly or the communication channel is lost due to network fluctuations, etc., any of which triggers the interruption of the target task can automatically restore the interrupted task. It can greatly reduce the pressure on system operation and maintenance personnel and ensure that the system can effectively and stably operate for a long time.

[0055] In one embodiment, the scheduling log and the task log include the execution status. During the execution of each target task, the corresponding task log and scheduling log are updated until all target tasks are completed, including:

[0056] (1) After the first target task is called, the execution status is updated to "Executing" in the scheduling log.

[0057] (2) For any called target task, the execution status is updated to "in progress" in the corresponding task log, and after the target task is completed, the execution status is updated to "completed";

[0058] (3) When all target tasks are called and the execution status of the corresponding task logs are all completed, the execution status is updated to completed in the scheduling log.

[0059] It can be understood that both the scheduling log and the task log contain the execution status information. The execution status of the scheduling log is used to reflect the stage of the entire scheduling process. The execution status in the task log only reflects the stage of the corresponding target task. Specifically, the first target task is called, marking the beginning of the entire scheduling process. At this time, the execution status in the scheduling log can be updated to executing. At this time, the target task will be continuously called according to the preset execution mechanism. For each called target task, an initial task log will be created for it after it is called. At this time, the task status can be updated to executing. Until the target task feedbacks to the scheduling system that it has been processed, the task status can be updated to completed. During this period, the execution status in the scheduling log will remain in execution until the task pool is cleared and the task status of all task logs under the same scheduling id is completed, which marks the end of the entire scheduling process. At this time, the execution status in the scheduling log can be updated to completed.

[0060] In one embodiment, the scheduling log and the task log include the heartbeat time. During the execution of each target task, the corresponding task log and the scheduling log are updated until all target tasks are completed, and further include:

[0061] (1) For a task log whose execution status is "in progress", the heartbeat time is updated to the current time at a second interval.

[0062] (2) For the scheduling log whose execution status is "Executing", the heartbeat time is updated to the current time according to the third interval.

[0063] It can be understood that the heartbeat time is a record item set to facilitate determining whether the process is alive. When any target task is called, the task status in the corresponding task log will change to "in execution". At this time, the heartbeat time will be updated according to the second interval, and the heartbeat time will be updated to the current time each time. Similarly, when the first target task is called, the execution status in the scheduling log will change to "in execution". At this time, the heartbeat time will be updated according to the third interval, and the heartbeat time will be updated to the current time each time. The second interval and the third interval here can be selected to be the same or different.

[0064] In one embodiment, checking the scheduling log and the task log to determine whether there is an interrupted target task includes:

[0065] (1) Determine whether there is a scheduling log in the execution state of being executed, in which the difference between the heartbeat time and the current time is greater than the third interval. If so, determine it as the target scheduling log.

[0066] It can be understood that only the scheduling process that is still being executed may be interrupted, and it is determined one by one whether the time difference between the latest heartbeat time of the scheduling log corresponding to the scheduling process being executed and the current time is greater than the third interval. If the difference is greater than the third interval, it means that the heartbeat of this scheduling log has expired, and the scheduling process corresponding to this scheduling log has been interrupted, so this scheduling log is marked as the target scheduling log.

[0067] (2) In the task log corresponding to the target scheduling log, determine whether there is a task log in the execution status of being executed, and whether the difference between the heartbeat time and the current time is greater than the third interval. If so, determine the corresponding target task as an interrupted target task.

[0068] It can be understood that in order to continue the scheduling from the interrupted target task, it is necessary to first determine which target task is interrupted, and the scheduling system further checks all the executing task logs corresponding to the scheduling log. It is determined whether the difference between the latest heartbeat time of each task log and the current time is greater than the second interval. If it is found that there is a task log greater than the second interval, it can be determined that the corresponding target is interrupted. At this time, the execution will continue according to the steps of step S108, and the scheduling process will continue to be executed from the target task.

[0069] In one embodiment, the scheduling log and the task log include a start time and an end time.

[0070] During the execution of each target task, the corresponding task log and scheduling log are updated until all target tasks are completed, including:

[0071] (1) After the first target task is called, the start time is updated to the current time in the scheduling log.

[0072] (2) For any target task that is called, in the corresponding task log, the start time is updated to the current time, and after the target task is completed, the end time is updated to the current time.

[0073] (3) When all target tasks are called and the execution status of the corresponding task logs is completed, the end time is updated to the current time in the scheduling log.

[0074] It can be understood that both the scheduling log and the task log contain the start time and the end time. The start time and the end time of the scheduling log are used to reflect the start and end time points of the entire scheduling process. The start time and the end time in the task log reflect the start and end time points of the corresponding target task. Specifically, the first target task is called, marking the beginning of the entire scheduling process. At this time, the start time in the scheduling log can be updated to the current time, and the end time can be temporarily left blank. At this time, the target task will be continuously called according to the preset execution mechanism. For each called target task, an initial task log will be created for it after it is called. At this time, the start time can be updated to the current time, and the end time can be temporarily left blank. Until the target task feedbacks to the scheduling system that it has been processed, the end time can be updated to the current time. During this period, until the task pool is cleared and the task status of all task logs under the same scheduling id is completed, this marks the end of the entire scheduling process. At this time, the end time can be updated to the current time.

[0075] In one embodiment, the preset execution mechanism includes:

[0076] (1) Obtain the resource utilization of the current scheduling system and determine the maximum data processing capacity based on the resource utilization.

[0077] (2) Based on the maximum data processing volume, the target task with the largest data volume is preferentially retrieved from the target tasks.

[0078] It can be understood that resource utilization reflects the usage of computing resources in the scheduling system, which can be CPU utilization, memory occupancy, etc. When the target task needs to be called, the maximum data processing scale that the scheduling system can currently bear will be evaluated according to the current resource utilization, as the maximum data processing volume. Among all the target tasks to be scheduled, they are sorted according to the amount of data that each task needs to process. Among them, the target tasks with the largest data volume and close to but not exceeding the maximum data processing volume are preferentially selected for execution. Repeat the above steps until all target tasks are scheduled. At each call, if there is a parallel scheduling mechanism in the scheduling system, multiple target tasks can be called at a time, otherwise the target tasks will be called one by one. This execution mechanism maximizes the currently available resources and avoids resource fragmentation caused by the use of small data volume tasks. At the same time, tasks that exceed the system processing capacity will not be selected to cause failure. It can improve resource utilization efficiency and achieve stable and efficient scheduling.

[0079] In one embodiment, each target task carries a tag related to the priority, and the preset execution mechanism includes: calling the target tasks for execution in descending order of priority.

[0080] The present application provides a data processing task scheduling device, which is applied to a scheduling system. The scheduling system is used to retrieve data from a business system to execute data processing tasks. Figure 2 The scheduling device for data processing tasks includes a first response module 210, a first processing module 220, a second response module 230 and a second processing module 240.

[0081] The first response module 210 is used to generate an initial scheduling log in response to a scheduling start instruction, and to use the data processing task that meets the start condition as the target task of this scheduling.

[0082] The first processing module 220 is used to call the target task for execution according to the preset execution mechanism, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed.

[0083] The second response module 230 is used to respond to the log check instruction and check the scheduling log and the task log to determine whether there is an interrupted target task. The log check instruction is executed periodically at a first interval after the scheduling system is started.

[0084] The second processing module 240 is used to start from the interrupted target task and return to call the target task for execution according to the preset execution mechanism when there is an interrupted target task, generate an initial task log for each called target task, and update the corresponding task log and scheduling log during the execution of each target task until all target tasks are completed.

[0085] For the specific definition of the scheduling device of the data processing task, please refer to the definition of the scheduling method of the data processing task above, which will not be repeated here. The various modules in the above-mentioned scheduling device of the data processing task can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0086] The present application provides a computer device, including one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the scheduling method for data processing tasks in any of the above embodiments are executed.

[0087] Indicatively, Figure 3 As shown, Figure 3This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 3 , the computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions that can be executed by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the scheduling method of the data processing task of any of the above embodiments.

[0088] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0089] The present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for scheduling data processing tasks in any of the above-mentioned embodiments.

[0090] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0091] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0092] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for scheduling data processing tasks, characterized in that: Applied to a scheduling system, the scheduling system is used to retrieve data from a business system to perform a data processing task, and the scheduling method of the data processing task includes: In response to the scheduling start instruction, an initial scheduling log is generated, and the data processing task that meets the start condition is used as the target task of this scheduling; the scheduling log and task log include the execution status and heartbeat time; The target task is called for execution according to the preset execution mechanism, an initial task log is generated for each of the called target tasks, and during the execution of each of the target tasks, the corresponding task log and the scheduling log are updated until all the target tasks are completed; specifically comprising: after the first target task is called, the execution status is updated to being executed in the scheduling log; for any of the called target tasks, the execution status is updated to being executed in the corresponding task log, and after the target task is completed, the execution status is updated to being completed; for the task log whose execution status is being executed, the heartbeat time is updated to the current time according to the second interval; for the scheduling log whose execution status is being executed, the heartbeat time is updated to the current time according to the third interval; when all the target tasks are called and the execution status of the corresponding task logs are all completed, the execution status is updated to being completed in the scheduling log; In response to a log check instruction that is performed periodically at a first interval after the scheduling system is started, the scheduling log and the task log are checked to determine whether there is an interrupted target task; specifically comprising: judging whether there is a scheduling log in which the difference between the heartbeat time and the current time is greater than the third interval in the scheduling log in which the execution status is being executed, and if so, determining it as the target scheduling log; judging whether there is a task log in which the difference between the heartbeat time and the current time is greater than the second interval in the task log corresponding to the target scheduling log in which the execution status is being executed, and if so, determining the corresponding target task as the interrupted target task; If so, starting from the interrupted target task, return to the target task execution according to the preset execution mechanism, generate an initial task log for each called target task, and during the execution of each target task, update the corresponding task log and scheduling log until all target tasks are completed.

2. The method for scheduling data processing tasks according to claim 1, characterized in that: The scheduling log and the task log include a start time and an end time; During the execution of each target task, updating the corresponding task log and the scheduling log until all the target tasks are completed, including: After the first target task is called, in the scheduling log, the start time is updated to the current time; For any of the called target tasks, in the corresponding task log, the start time is updated to the current time, and after the target task is completed, the end time is updated to the current time; When all the target tasks are called and the execution statuses of the corresponding task logs are all completed, the end time is updated to the current time in the scheduling log.

3. The method for scheduling data processing tasks according to claim 1, characterized in that: The preset execution mechanism includes: Obtaining the current resource utilization of the scheduling system, and determining the maximum data processing capacity according to the resource utilization; According to the maximum data processing amount, the target task with the largest data amount is preferentially retrieved from the target tasks.

4. The method for scheduling data processing tasks according to claim 1, characterized in that: Each of the target tasks carries a tag related to the priority, and the preset execution mechanism includes: The target tasks are retrieved and executed in descending order of priority.

5. A data processing task scheduling device, characterized in that: Applied to a scheduling system, the scheduling system is used to retrieve data from a business system to perform a data processing task, and the scheduling device for the data processing task includes: A first response module is used to generate an initial scheduling log in response to a scheduling start instruction, and use the data processing task that meets the start condition as the target task of this scheduling; the scheduling log and task log include the execution status and heartbeat time; The first processing module is used to call the target task for execution according to the preset execution mechanism, generate an initial task log for each of the called target tasks, and update the corresponding task log and the scheduling log during the execution of each of the target tasks until all the target tasks are completed; specifically including: after the first target task is called, updating the execution status to being executed in the scheduling log; for any of the called target tasks, updating the execution status to being executed in the corresponding task log, and after the target task is completed, updating the execution status to being completed; for the task log with the execution status being being executed, updating the heartbeat time to the current time according to the second interval; for the scheduling log with the execution status being being executed, updating the heartbeat time to the current time according to the third interval; when all the target tasks are called and the execution status of the corresponding task logs are all completed, updating the execution status to being completed in the scheduling log; The second response module is used to respond to the log check instruction that is periodically performed at the first interval after the scheduling system is started, check the scheduling log and the task log to determine whether there is the interrupted target task; the log check instruction is periodically performed at the first interval after the scheduling system is started; specifically comprising: judging whether there is a scheduling log in the execution state of being executed, in which the difference between the heartbeat time and the current time is greater than the third interval, and if so, determining it as the target scheduling log; judging whether there is a task log in the execution state of being executed, in the task log corresponding to the target scheduling log, in which the difference between the heartbeat time and the current time is greater than the second interval, and if so, determining the corresponding target task as the interrupted target task; The second processing module is used to, in the case where there is an interrupted target task, start from the interrupted target task, return to the process of calling the target task for execution according to the preset execution mechanism, generate an initial task log for each called target task, and update the corresponding task log and the scheduling log during the execution of each target task until all the target tasks are completed.

6. A computer device, characterized in that: The method comprises one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the data processing task scheduling method as described in any one of claims 1 to 4 are executed.

7. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data processing task scheduling method as described in any one of claims 1 to 4.

Images