Task scheduling method and system for big data development

Abstract

The invention relates to the technical field of ETL task batch processing, and discloses a task scheduling method and system oriented to big data development, the method is applied to a task scheduling architecture comprising task dispatchers, task queues and task executors, and the task dispatchers are used for scheduling tasks according to scheduling configuration and resource analysis conditions. Obtaining a target task and sequentially sending the target task to different task queues; the task executors continuously consume and execute the target tasks from respective task queues; any task dispatcher carries out heartbeat information maintenance and task execution state monitoring on other task dispatchers and other task actuators in real time, and the task execution states comprise failure, suspension, execution waiting and success; and when any task distributor or any task actuator is found to have a fault, starting task takeover, and after the target task is interrupted, recovering execution of the interrupted target task from the interruption point. The method has the effects of improving the task scheduling efficiency and ensuring the system stability.

Description

Technical Field

[0001] The present application relates to the technical field of ETL task batch processing, and in particular to a task scheduling method and system for big data development. Background Art

[0002] With the rapid advancement of information technology and the rapid expansion of data volume, the traditional ETL (Extract-Transform-Load) data processing model is facing unprecedented challenges. Traditional ETL task scheduling systems usually rely on a single script to perform batch data processing tasks, that is, encapsulating the entire ETL process, including data extraction, cleaning, transformation, loading and other links, in one script. This model is prone to frequent single point failures, scalability limitations, insufficient concurrency and other problems when processing large amounts of data and highly complex data processing scenarios. Once an error occurs during task execution, it is usually necessary to execute the entire script from scratch, which not only wastes time and energy, but also seriously wastes system resources due to repeated processing of completed parts, thereby affecting the overall business efficiency. In addition, the script has high logical complexity, the code is difficult to maintain and understand, and requires developers to have a solid programming foundation.

[0003] In the prior art, the complexity of the script is reduced by splitting a large script into multiple small scripts and executing them in steps. However, this method requires precise definition of the boundaries and dependencies of each step. At the same time, the management of the split small scripts becomes more complicated, increasing maintenance costs and error risks, making it difficult to split a single script and complex to manage.

[0004] With respect to the above-mentioned related technologies, the inventors have found that the existing ETL task scheduling methods have the problems of low efficiency, great difficulty in script splitting and complex management. Summary of the invention

[0005] In order to improve scheduling efficiency while reducing the difficulty of script maintenance and the risk of errors, the present application provides a task scheduling method and system for big data development.

[0006] In a first aspect, the present application provides a task scheduling method for big data development.

[0007] This application is achieved through the following technical solutions:

[0008] A task scheduling method for big data development is applied to a task scheduling architecture including a task distributor, a task queue and a task executor, and includes the following steps:

[0009] The multiple task dispatchers obtain target tasks and send them to different task queues in sequence according to the scheduling configuration and resource analysis;

[0010] The task executors continuously consume and execute the target tasks from the respective task queues;

[0011] Any task dispatcher maintains heartbeat information and monitors task execution status of other task dispatchers and other task executors in real time, where the task execution status includes failure, suspension, pending execution and success;

[0012] When any task distributor or any task executor is found to have a fault, task takeover is initiated, and after the target task is interrupted, the interrupted target task is resumed from the interruption point.

[0013] In a preferred example, the present application can be further configured as follows: when any task dispatcher or any task executor is found to have a fault, the task takeover is started, and after the target task is interrupted, the steps of resuming execution of the interrupted target task from the interruption point include:

[0014] Enable a healthy task dispatcher or a healthy task executor, and synchronously reset the status of all interrupted target tasks to pending execution;

[0015] Start the task monitoring thread and poll all the target tasks to be executed to see if they meet the execution conditions;

[0016] If the target task to be executed meets the execution condition, the healthy task dispatcher sends the target task to be executed to the task queue in sequence according to the preset order;

[0017] Healthy task executors consume and execute the target tasks from the task queue.

[0018] In a preferred example, the present application can be further configured as follows: it also includes the following steps:

[0019] When the execution status of any target task is successful and is sent to the task queue, it is determined that the target task has a re-run operation, and the target task is deleted from the task queue and the task monitoring thread for the target task is terminated.

[0020] In a preferred example, the present application can be further configured as follows: the steps of the multiple task dispatchers acquiring target tasks and sending them to different task queues in sequence according to the scheduling configuration and resource analysis include:

[0021] Start the batch processing thread and monitor the execution status of the target task;

[0022] If the task is executed successfully, the next target task to be executed is obtained;

[0023] According to a preset queue threshold, determining the queue resource status of the task queue;

[0024] When the queue resource state of the task queue is idle, the acquired target task to be executed is sent to the task queue.

[0025] In a preferred example, the present application may be further configured as follows: after the step of starting the batch processing thread and monitoring the execution status of the target task, it also includes:

[0026] If the task execution fails, the execution status of the target task is updated to failed, and the execution status of the next-level target task corresponding to the target task is updated to suspended;

[0027] Stop the batch thread.

[0028] In a preferred example, the present application can be further configured as follows: before the step of starting the batch processing thread and monitoring the execution status of the target task, the following steps are also included:

[0029] Perform timeliness check until the result of timeliness check meets the preset conditions.

[0030] In a second aspect, the present application provides a task scheduling system for big data development.

[0031] This application is achieved through the following technical solutions:

[0032] A task scheduling system for big data development, including a task dispatcher, a task queue and a task executor task scheduling architecture;

[0033] The multiple task dispatchers are used to obtain target tasks and send them to different task queues in sequence according to the scheduling configuration and resource analysis situation;

[0034] The task executor is used to continuously consume and execute the target task from the respective task queue;

[0035] A thread module for any task dispatcher to perform heartbeat information maintenance and task execution status monitoring on other task dispatchers and other task executors in real time, where the task execution status includes failure, suspension, pending execution and success;

[0036] Also includes,

[0037] The thread module is used to start task takeover when any task distributor or any task executor fails, and resume execution of the interrupted target task from the interruption point after the target task is interrupted.

[0038] In a third aspect, the present application provides a computer device.

[0039] This application is achieved through the following technical solutions:

[0040] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned task scheduling methods for big data development when executing the computer program.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium.

[0042] This application is achieved through the following technical solutions:

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned task scheduling methods for big data development.

[0044] In a fifth aspect, the present application provides a computer program product.

[0045] This application is achieved through the following technical solutions:

[0046] A computer program product includes a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned task scheduling methods for big data development.

[0047] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0048] The task dispatcher obtains the target task and sends it to the task queue in sequence according to the scheduling configuration and resource analysis; the task executor continuously consumes and executes the target task from the task queue; any task dispatcher performs heartbeat information maintenance and task execution status monitoring on other task dispatchers and other task executors in real time, so as to record the execution status and running status of each link in the ETL process, which is conducive to the subsequent implementation of the breakpoint resumption function of the ETL process; when any task dispatcher or any task executor is found to have a failure, task takeover is initiated to reschedule the task, and after the target task is interrupted, the interrupted target task is resumed from the interruption point, thereby ensuring the continuity and integrity of ETL data processing and improving the scheduling efficiency of tasks; there is no need to split the script, reducing the maintenance difficulty and error risk of the script. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A task distribution flow chart of a task scheduling method for big data development is provided as an exemplary embodiment of the present application.

[0050] Figure 2A schematic diagram of a task takeover process when a task distributor fails in a task scheduling method for big data development provided by yet another exemplary embodiment of the present application.

[0051] Figure 3 A schematic diagram of a task takeover process when a task executor fails in a task scheduling method for big data development provided by another exemplary embodiment of the present application.

[0052] Figure 4 A structural block diagram of a task scheduling system for big data development provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0053] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0056] In the context of enterprise-level applications, how to efficiently and accurately process and analyze huge data sets and build an efficient, stable, and highly scalable task scheduling system has become the key to supporting business decision optimization and improving operational efficiency.

[0057] This application builds a distributed task scheduling and fault recovery system based on a microservice architecture. The system has high availability, strong concurrency capabilities and excellent performance, and is specifically used for task scheduling and management.

[0058] The present application embodiment provides a task scheduling method for big data development, which is applied to a task scheduling architecture including a task dispatcher, a task queue, and a task executor. The main steps of the method are described as follows:

[0059] The multiple task dispatchers obtain target tasks and send them to different task queues in sequence according to the scheduling configuration and resource analysis;

[0060] The task executors continuously consume and execute the target tasks from the respective task queues;

[0061] Any task dispatcher maintains heartbeat information and monitors task execution status of other task dispatchers and other task executors in real time, where the task execution status includes failure, suspension, pending execution and success;

[0062] When any task distributor or any task executor is found to have a fault, task takeover is initiated, and after the target task is interrupted, the interrupted target task is resumed from the interruption point.

[0063] Specifically, the scheduling configuration includes the task priority, scheduling strategy, dependency, resource requirements, etc. Among them, the task scheduling strategy includes: First Come First Served (FCFS) strategy: assign tasks in the order they arrive; Priority strategy: prioritize high priority tasks; Round Robin strategy: assign a fixed time slice to each task and execute them in turns; Dependency strategy: schedule related tasks after the dependent tasks are completed according to the task dependencies.

[0064] Resource analysis includes computing the resources required to execute analysis tasks, such as CPU, GPU, memory, storage, network bandwidth, etc. It analyzes the currently available resources by monitoring the system's resource usage.

[0065] The task dispatcher determines how to sort and allocate tasks based on the scheduling configuration and the current resource analysis of the system, decides which tasks can be assigned to which nodes, and sends the tasks to the task queue.

[0066] The task executor continuously consumes and executes the target task from the task queue.

[0067] By using a shared storage system, the heartbeat information of the task distributor and the task executor is maintained and the task execution status is monitored in real time.

[0068] Among them, the task distributor node starts a heartbeat update task (HeartBeatService) based on the heartbeat mechanism, and periodically updates the status information of the current node to the shared storage system, such as every 10 seconds / time, as an indicator of the node's health status. If the node fails to write back its status information within the predetermined time, that is, it times out, the node is determined to be a faulty node and the task distributor fails. Similarly, the task executor periodically updates the status information of the current node to the shared storage system based on the heartbeat mechanism. If the node fails to write back its status information within the predetermined time, the task executor is determined to have failed. Through the heartbeat mechanism, the system can grasp the health status of each task distributor node and each task executor node in real time and accurately, ensuring the stability and reliability of the overall system.

[0069] After confirming the existence of a faulty node, the target healthy node will collect the distribution or execution tasks that are currently being processed and to be processed by the faulty node, and the target healthy node will take over the task scheduling and continue to perform task distribution or execution work.

[0070] While starting the heartbeat update task, based on the task fault detection mechanism, the task dispatcher will start an independent fault detection timer task (CheckTransferJob Service) in parallel. The fault detection timer task regularly scans and analyzes the heartbeat timestamps of all task dispatcher nodes in the system except itself and the heartbeat timestamps of all task executor nodes except itself, and compares the heartbeat timestamp with the preset safety threshold. When the heartbeat timestamp exceeds the safety threshold, it identifies potential abnormal tasks and marks the node as "failed". Once the node fails, the system will immediately trigger the corresponding fault handling mechanism.

[0071] The task fault detection mechanism enables mutual monitoring between nodes, which enhances the robustness and scalability of the system. There is no need to rely on a single central node to monitor the task execution status, thus achieving decentralization. It ensures that the system can instantly perceive the abnormal status of any node and respond in real time, reducing the response time of faults. It also improves the accuracy of fault detection and reduces false alarms and missed alarms. It supports flexible configuration of thresholds, which can be adjusted and adapted to different application scenarios and load changes according to the actual operation of the system, making it easy to deploy and maintain.

[0072] When any task dispatcher or any task executor is found to have a fault, task takeover is initiated, and after the target task is interrupted, the interrupted target task is resumed from the interruption point to implement the breakpoint resume function.

[0073] In one embodiment, the steps of the multiple task dispatchers acquiring target tasks and sending them to different task queues in sequence according to the scheduling configuration and resource analysis include:

[0074] Start the batch processing thread and monitor the execution status of the target task;

[0075] If the task is executed successfully, the next target task to be executed is obtained;

[0076] According to a preset queue threshold, determining the queue resource status of the task queue;

[0077] When the queue resource state of the task queue is idle, the acquired target task to be executed is sent to the task queue.

[0078] In one embodiment, after the step of starting the batch processing thread and monitoring the execution status of the target task, the process further includes:

[0079] If the task execution fails, the execution status of the target task is updated to failed, and the execution status of the next-level target task corresponding to the target task is updated to suspended;

[0080] Stop the batch thread.

[0081] In one embodiment, before the step of starting the batch processing thread and monitoring the execution status of the target task, the following steps are also included:

[0082] Perform timeliness check until the result of timeliness check meets the preset conditions.

[0083] Reference Figure 1 ,Specifically, before starting the batch processing thread, a timeliness check is performed to ,pre-check whether the task is still valid and needs to be executed, such as ,whether the task is within the validity period and whether the task meets a ,specific time range, etc., to ensure that the batch tasks are meaningful when ,executed and comply with the current time or business rules, and avoid processing ,expired or invalid tasks.

[0084] If the timeliness check meets the preset conditions, the task execution status is analyzed, including failure, suspension, pending and success, until all "failed" and "suspended" task node states are reset to "pending" to complete the task execution status analysis. If the timeliness check does not meet the preset conditions or the system is in a dormant state, the timeliness check is repeated until the timeliness check meets the preset conditions.

[0085] Next, the batch processing thread is called with the help of the DAG scheduler. By starting the batch processing thread, the execution status of the target task is monitored.

[0086] If the task is executed successfully, the next target task to be executed is obtained.

[0087] According to the preset queue threshold, the queue resource status of the target task queue is determined.

[0088] When the queue resource state of the target task queue is idle, the target task to be executed is sent to the task queue. When the queue resource state of the target task queue is busy or the system is in a dormant state, the queue resource state of the target task queue is continuously waited and repeatedly judged until the tasks in the queue are consumed and space is vacated, at which time the queue resource state of the target task queue is restored to idle. By detecting the queue resources, it is ensured that the queue is not full to ensure the orderly and efficient execution of the tasks.

[0089] If the task execution fails, the execution status of the target task is updated to failure, and the execution status of the target task at the next level corresponding to the target task is updated to suspended, and then the batch processing thread is stopped.

[0090] Batch scheduling of tasks in batch mode can reduce the overhead of task scheduling.

[0091] In one embodiment, when any task dispatcher or any task executor is found to have a fault, the task takeover is initiated, and after the target task is interrupted, the steps of resuming execution of the interrupted target task from the interruption point include:

[0092] Enable a healthy task dispatcher or a healthy task executor, and synchronously reset the status of all interrupted target tasks to pending execution;

[0093] Start the task monitoring thread and poll all pending target tasks to see if they meet the execution conditions;

[0094] If the target task to be executed meets the execution condition, the healthy task dispatcher sends the target task to be executed to the task queue in sequence according to the preset order;

[0095] Healthy task executors consume and execute the target tasks from the task queue.

[0096] Specifically, in a batch processing thread, if a task dispatcher or task executor is found to have a fault, or a task execution fails, the faulty task dispatcher or task executor is located, the failed task and failure log are recorded, and all subtasks that have not yet been executed are suspended. The subtask is the target task of the next level corresponding to the failed target task.

[0097] Then, once the task failure is eliminated, enable the healthy task dispatcher or healthy task executor, and synchronously reset the status of all interrupted target tasks to pending execution; start the task monitoring thread (DagBatchinvoke), such as Figure 1As shown, all target tasks to be executed are polled to see if they meet the execution conditions; if the target tasks to be executed meet the execution conditions, the healthy task dispatcher sends the target tasks to be executed to the task queue in sequence according to the preset order; the healthy task executor consumes and executes the target tasks from the task queue. The preset order can be a preconfigured scheduling strategy, and the execution condition can be that the predecessor task of the task is executed "successfully".

[0098] Furthermore, after the task failure is eliminated, the administrator can manually start the breakpoint-resume process from the failed task of the process, including:

[0099] When the process resumes, the process status of the task dispatcher is reset to "to be run", and the status of all interrupted target tasks, including failed tasks and their unexecuted subtasks, is reset to "to be run" synchronously. For successfully completed tasks, their original status is kept unchanged to avoid repeated processing.

[0100] The system will then start the monitoring thread to poll all "to be run" tasks under the process to see if they meet the execution conditions. If so, it will determine the resource status of the target task queue.

[0101] If the task queue is idle, the target tasks to be run will be sent to the task execution queue one by one according to the logical relationship of the flowchart.

[0102] If the task queue is not idle, it will continue to wait until the tasks in the task queue are consumed and space is freed up before sending the target task to be run.

[0103] Each task queue is equipped with a dedicated task thread pool, which continuously takes out target tasks from the task queue and executes them, ensuring the execution efficiency of tasks and efficient use of system resources.

[0104] The task thread pool continuously takes out the target task from the task queue and executes it. At the same time, it starts the monitoring thread synchronously to monitor in real time whether the task execution thread is executing normally. The task executor starts a dedicated monitoring thread in parallel, which is responsible for monitoring the execution status of the task in real time and collecting the execution log of the task, so as to grasp the execution status of the task in real time. In this embodiment, the monitoring thread can be a timed polling task, for example, a timer with a default of 10 seconds.

[0105] When the monitoring thread finds that the task execution is completed, it automatically terminates the current monitoring thread.

[0106] When the monitoring thread finds that the task is manually terminated, it updates the task status and information, and stops the task thread pool and monitoring thread.

[0107] When the task execution exceeds the component setting time, the component log is updated, and after the task component execution timeout continues for a preset period of time, such as after the task component execution timeout continues for 24 hours, the task thread pool and monitoring thread are forcibly terminated.

[0108] The above-mentioned breakpoint-resume mechanism not only improves the system's response speed and troubleshooting capabilities, but also ensures the transparency and traceability of task execution, providing strong support for subsequent process optimization and fault analysis.

[0109] Reference Figure 2 During the task distribution process, if the task distributor encounters a node failure, the healthy task distributor node will take over the failed node and continue to execute the distribution tasks currently being processed and pending by the failed node to complete the task scheduling work. The healthy node will also start and resume the execution monitoring mechanism to ensure that the execution status of all distribution tasks can be tracked and recorded in real time. In addition, the healthy nodes will continue to monitor the health of the remaining distributor nodes and continue to trigger the task takeover logic when necessary.

[0110] Reference Figure 3 ,During the task execution process, if the task executor encounters a node failure, the ,healthy task executor node will be responsible for taking over the ,faulty node, resetting the task status in the faulty node, and the task ,dispatcher will re-assign a new task executor to the interrupted task to continue ,task processing, and complete the task scheduling.

[0111] For task executor nodes that are confirmed to be "failed", the healthy node of the task executor first collects and analyzes the "executing" tasks before the node fails, isolates the failed node, and notifies the task distribution node to stop using it to avoid further distribution of tasks to the "failed" node. At the same time, all tasks in the task queue that have been assigned to the node are reallocated, and the status of the "running" tasks in the original node is updated to "pending execution", waiting to be reallocated to the healthy node for re-execution.

[0112] After task isolation and takeover are completed, the task distributor immediately intervenes and is responsible for redistributing the task to the queue of the available task executor node to ensure that the task can resume operation as soon as possible.

[0113] Once the tasks are redistributed to the new task executor node, the node will immediately start executing these tasks. At the same time, the node will also start a monitoring thread to track the execution of the tasks to ensure that they can be successfully completed. In addition, each node will continue to monitor the health of the remaining executor nodes and continue to trigger the task takeover logic when necessary.

[0114] At the same time, after the "failed" node of the faulty task executor resumes operation and passes the fault detection, the heartbeat information can be written back normally through the heartbeat mechanism. At this time, the heartbeat information of the executor will be updated and maintained, and the task executor will also recognize and re-join the list of healthy nodes, and allow the node to re-participate in task execution. According to the current task load, the "pending" accumulated tasks will be redistributed. When the task distribution node recognizes the addition of the new executor, it will distribute and schedule the tasks normally, and adjust and allocate the tasks according to the resource conditions of each executor node, thereby improving the effective utilization of resources and the efficiency of task execution.

[0115] In one embodiment, the following steps are also included:

[0116] When the execution status of any target task is successful and is sent to the task queue, it is determined that the target task has a re-run operation, and the target task is deleted from the task queue and the task monitoring thread for the target task is terminated.

[0117] The monitoring thread started by the task executor can be used to check whether the current task has been rerun. If there is a task rerun operation, an exception is triggered and the monitoring thread ends.

[0118] In summary, a task scheduling method for big data development utilizes microservices, based on a task dispatcher, obtains target tasks and sends them to the task queue in sequence according to scheduling configuration and resource analysis; based on a task executor, continuously consumes and executes target tasks from the task queue; and performs heartbeat information maintenance and task execution status monitoring on the task dispatcher and the task executor in real time, so as to record the execution status and running status of each link in the ETL process, which is conducive to the subsequent implementation of the breakpoint resumption function of the ETL process; when it is found that the task dispatcher or the task executor fails, task takeover is started to reschedule the task, and after the target task is interrupted, the interrupted target task is resumed from the interruption point, thereby ensuring the continuity and integrity of ETL data processing and improving the scheduling efficiency of ETL tasks; there is no need to split the script, reducing the maintenance difficulty and error risk of the script.

[0119] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] The embodiment of the present application also provides a task scheduling system for big data development, which includes a task dispatcher, a task queue, and a task scheduling architecture of a task executor;

[0121] The multiple task dispatchers are used to obtain target tasks and send them to different task queues in sequence according to the scheduling configuration and resource analysis situation;

[0122] The task executor is used to continuously consume and execute the target task from the respective task queue;

[0123] A thread module for any task dispatcher to perform heartbeat information maintenance and task execution status monitoring on other task dispatchers and other task executors in real time, where the task execution status includes failure, suspension, pending execution and success;

[0124] Also includes,

[0125] The thread module is used to start task takeover when any task distributor or any task executor fails, and resume execution of the interrupted target task from the interruption point after the target task is interrupted.

[0126] Reference Figure 4 The task scheduling system includes a task dispatcher (Task Dispatcher), a task queue (Queue) and a task executor (Task Executor) task scheduling architecture.

[0127] The task dispatcher has functions such as scheduling interface, timeliness check, job monitoring, thread pool, resource analysis, task priority strategy design, heartbeat detection, distribution takeover, heartbeat information maintenance, task takeover, etc. The task dispatcher determines whether to send the task to the task queue and completes the task distribution based on the scheduling configuration and resource analysis.

[0128] The task executor has functions such as queue monitoring, timeliness verification, task monitoring, thread pool, node isolation / suspension, task reloading, heartbeat information maintenance, heartbeat detection, etc. The task executor continuously consumes and executes tasks from the task queue.

[0129] In this embodiment, by using a shared storage system, the status of each task distributor and each task executor is actively monitored to maintain the heartbeat information of each node of the task distributor and the task executor. At the same time, the execution status of the task is monitored to monitor the entire ETL task running status.

[0130] When a related service is monitored, such as a task dispatcher or task executor that has a failure, task takeover is initiated. After the target task is interrupted, the interrupted target task is resumed from the interruption point to achieve task rescheduling, that is, the breakpoint resumption function of the ETL process, thereby ensuring the continuity and integrity of data processing. When an error occurs during task execution, there is no need to execute the entire script from the beginning, thereby improving efficiency.

[0131] When a task is monitored to have an exception and cannot be advanced for a long time, an exception will be triggered and the task will be prompted to be abnormal. After the operation and maintenance personnel successfully eliminate the exception, the scheduling system will reset the task execution status and implement breakpoint resumption, and resume from the failed task node. There is no need to execute the entire script process from the beginning, which reduces the tedious work of re-sorting the task process and avoids the problem of repeated processing of completed tasks, significantly shortening the task completion time and effectively utilizing system resources.

[0132] The system monitors the status of related services and task execution in real time through the heartbeat mechanism. It can quickly identify faulty nodes and, through task takeover logic, have healthy nodes take over the tasks of faulty nodes to ensure stable operation of tasks.

[0133] In one embodiment, the execution status and running state of each link are recorded in the ETL process, and recorded in the database for display through a visual method such as canvas. By recording the status and log information of each task node during the task execution process, the breakpoint resume function of the ETL process is realized on this basis.

[0134] A task scheduling system for big data development adopts a microservice architecture design, which subdivides the task scheduling function into multiple independent service modules, and decomposes the huge application into a series of small services focusing on specific business functions, so that the failure of a single service will not affect the operation of the entire system, realize service independence and decoupling, and improve the stability and availability of the system; and by defining API interfaces, efficient interaction between services is achieved based on lightweight communication protocols, thereby ensuring the flexibility and efficient operation of the system, reducing the frequency of manual intervention in the task scheduling process, and effectively reducing operation and maintenance costs.

[0135] A microservice architecture for a task scheduling system developed for big data achieves independent deployment, flexible expansion, and rapid iteration of services, and promotes seamless communication between services.

[0136] A task scheduling system developed for big data can monitor the operating status of each microservice in real time. Once a service node failure / abnormality is detected, the healthy node will take over the task of the abnormal node and resume the execution of the interrupted target task from the interruption point. It has the functions of automatic fault takeover, breakpoint resumption and task reallocation after fault recovery. It has the advantages of high availability, high concurrency and high performance, and also significantly enhances the flexibility and maintainability of the system.

[0137] A task scheduling system for big data development fully considers the impact of node failures on the overall system stability, and is particularly suitable for scenarios with high requirements on the system's fault tolerance and maintainability.

[0138] A task scheduling system for big data development can realize task rescheduling based on the task breakpoint resumption mechanism in complex data processing processes, especially in scenarios involving large-scale data batch running. It does not require manual intervention for investigation and processing, improves the recovery efficiency after task anomalies, improves the scheduling efficiency of the system and reduces the waste of system resources.

[0139] For the specific definition of a task scheduling system for big data development, please refer to the definition of a task scheduling method for big data development in the above text, which will not be repeated here.

[0140] Each module in the above-mentioned task scheduling system for big data development can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0141] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, any of the above-mentioned task scheduling methods for big data development is implemented.

[0142] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any one of the above-mentioned task scheduling methods for big data development is implemented.

[0143] In one embodiment, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above-mentioned task scheduling methods for big data development.

[0144] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0145] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A task scheduling method for big data development, characterized in that: The task scheduling architecture applied to the task dispatcher, task queue and task executor includes the following steps: The multiple task dispatchers obtain target tasks and send them to different task queues in sequence according to the scheduling configuration and resource analysis; The task executors continuously consume and execute the target tasks from the respective task queues; Any task dispatcher maintains heartbeat information and monitors task execution status of other task dispatchers and other task executors in real time, where the task execution status includes failure, suspension, pending execution and success; When any task distributor or any task executor is found to have a fault, task takeover is initiated, and after the target task is interrupted, the interrupted target task is resumed from the interruption point.

2. The task scheduling method for big data development according to claim 1, characterized in that: When any task dispatcher or any task executor is found to have a fault, the task takeover is started, and after the target task is interrupted, the steps of resuming execution of the interrupted target task from the interruption point include: Enable a healthy task dispatcher or a healthy task executor, and synchronously reset the status of all interrupted target tasks to pending execution; Start the task monitoring thread and poll all the target tasks to be executed to see if they meet the execution conditions; If the target task to be executed meets the execution condition, the healthy task dispatcher sends the target task to be executed to the task queue in sequence according to the preset order; Healthy task executors consume and execute the target tasks from the task queue.

3. The task scheduling method for big data development according to claim 2 is characterized in that: The following steps are also included: When the execution status of any target task is successful and is sent to the task queue, it is determined that the target task has a re-run operation, and the target task is deleted from the task queue and the task monitoring thread for the target task is terminated.

4. The task scheduling method for big data development according to any one of claims 1 to 3, characterized in that: The steps of the multiple task distributors acquiring target tasks and sending them to different task queues in sequence according to the scheduling configuration and resource analysis include: Start the batch processing thread and monitor the execution status of the target task; If the task is executed successfully, the next target task to be executed is obtained; According to a preset queue threshold, determining the queue resource status of the task queue; When the queue resource state of the task queue is idle, the acquired target task to be executed is sent to the task queue.

5. The task scheduling method for big data development according to claim 4 is characterized in that: After the step of starting the batch processing thread and monitoring the execution status of the target task, the method further includes: If the task execution fails, the execution status of the target task is updated to failed, and the execution status of the next-level target task corresponding to the target task is updated to suspended; Stop the batch thread.

6. The task scheduling method for big data development according to claim 4, characterized in that: Before the step of starting the batch processing thread and monitoring the execution status of the target task, The following steps are included: Perform timeliness check until the result of timeliness check meets the preset conditions.

7. A task scheduling system for big data development, characterized in that: The task scheduling architecture includes task dispatcher, task queue and task executor; The multiple task dispatchers are used to obtain target tasks and send them to different task queues in sequence according to the scheduling configuration and resource analysis situation; The task executor is used to continuously consume and execute the target task from the respective task queue; A thread module for any task dispatcher to perform heartbeat information maintenance and task execution status monitoring on other task dispatchers and other task executors in real time, where the task execution status includes failure, suspension, pending execution and success; Also includes, The thread module is used to start task takeover when any task distributor or any task executor fails, and resume execution of the interrupted target task from the interruption point after the target task is interrupted.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 6 when being executed by a processor.

Images