Distributed scheduling method and distributed scheduling system

Abstract

The invention provides a distributed scheduling method and a distributed scheduling system, the method is applied to an execution unit in the distributed scheduling system, and execution task information of a to-be-processed task is acquired from a coordination service unit in the distributed scheduling system; when it is determined that the to-be-processed task can be executed based on cluster resource information of the distributed scheduling system and / or a task upper limit threshold value, generating an executable command corresponding to the execution task information by executing a python script corresponding to a task type in the task information; and calling a corresponding cluster in the distributed scheduling system to execute the to-be-processed task according to the executable command. According to the technical scheme, automatic execution of the tasks is achieved, manual intervention is reduced, execution efficiency is improved, the distributed scheduling system can run more efficiently and orderly, and therefore the requirement of services for task processing is better met.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a distributed scheduling method and a distributed scheduling system. Background Technology

[0002] In the process of big data generation and development, it is necessary to rely on a task scheduling system to manage and schedule tasks in a unified manner, ensure the upstream and downstream dependencies between tasks, monitor the task running status, issue timely alarms in case of abnormalities, handle anomalies as automatically as possible, ensure the high availability of the scheduling system, and ensure that the daily data statistics tasks are completed on time.

[0003] Therefore, there is an urgent need for a solution that can more efficiently schedule and execute tasks. Summary of the Invention

[0004] This application provides a distributed scheduling method and a distributed scheduling system to achieve the goal of efficiently scheduling and executing tasks.

[0005] In a first aspect, embodiments of this application provide a distributed scheduling method, applied to an execution unit in a distributed scheduling system, comprising:

[0006] Obtain the execution task information of the task to be processed from the coordination service unit in the distributed scheduling system;

[0007] When it is determined that the task to be processed can be executed based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, the executable command corresponding to the task information is generated by the Python script corresponding to the task type in the task information.

[0008] According to the executable command, the corresponding cluster in the distributed scheduling system is invoked to execute the task to be processed.

[0009] In one possible implementation, executing the task to be processed includes:

[0010] Enable a thread to execute the executable command;

[0011] Accordingly, the method further includes:

[0012] Obtain the output information and / or error information of the thread during execution;

[0013] Based on the output information and / or the error information, update the execution status of the task to be processed to the execution task information table in the storage unit of the distributed scheduling system.

[0014] Secondly, embodiments of this application provide a distributed scheduling system, which includes: an execution unit, a scheduling unit, a storage unit, and a coordination service unit;

[0015] The scheduling unit generates execution task information based on the upstream and downstream information and detailed execution information of the task to be processed obtained from the storage unit, and stores it in the storage unit.

[0016] When the scheduling unit detects a ZooKeeper node containing the task to be processed in the coordination service unit based on the task execution information, the scheduling unit stores the task execution information in the ZooKeeper node.

[0017] The execution unit executes the method described in any one of rights 1 to 3.

[0018] In one possible implementation, the execution unit includes: a Python interpreter;

[0019] The Python interpreter executes:

[0020] Based on the task type in the execution task information, the Python parsing script corresponding to the task type is determined;

[0021] Based on the Python parsing script, an executable command corresponding to the task information is generated.

[0022] In one possible implementation, the scheduling unit includes: a scheduler and a Quartz tasker;

[0023] The scheduler retrieves the task description information of the task to be processed from the task configuration table of the storage unit at a first preset frequency; and adds the task to be processed to the corresponding task list based on the task description information.

[0024] When the Quartz task generator detects that a pending task in the corresponding task list has reached its trigger execution time, it generates execution task information for the pending task based on the upstream and downstream information and detailed execution information of the pending task obtained from the task configuration table, and stores it in the execution task information table of the storage unit.

[0025] In one possible implementation, the scheduling unit further includes: a task monitor;

[0026] The task monitor executes:

[0027] Retrieve execution task information from the execution task information table;

[0028] If no ZooKeeper node for the task to be processed is detected in the coordination service unit, the ZooKeeper node is created in the coordination service unit, and the task execution information is stored in the ZooKeeper node.

[0029] Alternatively, if a ZooKeeper node for the task to be processed is detected in the coordination service unit, the task execution information is stored in the ZooKeeper node.

[0030] In one possible implementation, the system further includes: a resource coordinator;

[0031] In response to the resource status acquisition request from the execution unit, the resource coordinator sends the cluster resource information of the distributed scheduling system to the execution unit.

[0032] In one possible implementation, the scheduling unit registers its online status with the coordination service unit based on a heartbeat mechanism.

[0033] When the coordination service unit detects that the online status of the scheduling unit has changed to offline status, it switches the event for executing the pending task to the other scheduling unit based on the online status of other scheduling units.

[0034] Thirdly, embodiments of this application provide a distributed scheduling apparatus, comprising: an execution unit applied in a distributed scheduling system, the apparatus comprising:

[0035] The acquisition module is used to acquire the execution task information of the task to be processed from the coordination service unit in the distributed scheduling system.

[0036] The determination module is used to generate an executable command corresponding to the task information by using a Python script corresponding to the task type in the task information when it is determined that the task to be processed can be executed based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system.

[0037] The calling module is used to invoke the corresponding cluster in the distributed scheduling system to execute the task to be processed based on the executable command.

[0038] In one possible implementation, the calling module executes the task to be processed, specifically for:

[0039] Enable a thread to execute the executable command;

[0040] Accordingly, the determining module is also used for:

[0041] Obtain the output information and / or error information of the thread during execution;

[0042] Based on the output information and / or the error information, update the execution status of the task to be processed to the execution task information table in the storage unit of the distributed scheduling system.

[0043] In one possible implementation, before obtaining the execution task information of the task to be processed from the coordination service unit of the distributed scheduling system, the calling module is further configured to:

[0044] Based on the heartbeat mechanism, the online status of the execution unit is registered to the coordination service unit;

[0045] When the coordination service unit detects that the online status of the execution unit has changed to offline status, it switches the event for executing the task to be processed to the other execution units based on the online status of the other execution units.

[0046] Fourthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0047] The memory stores computer-executed instructions;

[0048] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0049] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0050] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0051] The distributed scheduling method and distributed scheduling system provided in this application are applied to the execution unit of the distributed scheduling system. The method obtains the execution task information of the task to be processed from the coordination service unit of the distributed scheduling system. When it is determined that the task to be processed can be executed based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, the method generates an executable command corresponding to the task information by using the Python script corresponding to the task type in the task information. According to the executable command, the corresponding cluster in the distributed scheduling system is called to execute the task to be processed. In this technical solution, firstly, the execution task information of the tasks to be processed is obtained from the coordination service unit of the distributed scheduling system. This ensures that the execution unit can accurately understand the specific requirements and details of the tasks, laying the foundation for the correct execution of subsequent tasks. Next, based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, it is determined whether the tasks to be processed can be executed. This resource-task matching mechanism avoids system chaos and task execution failures caused by insufficient resources or excessive tasks, improving the stability and reliability of the system, while also achieving reasonable allocation and efficient utilization of cluster resources. Then, executable commands are generated by using Python scripts corresponding to the task type in the task information. Leveraging the flexibility and powerful functionality of Python scripts, task information can be quickly and accurately transformed into operable commands, adapting to the needs of different types of tasks and enhancing the system's versatility and scalability. Finally, the corresponding cluster in the distributed scheduling system is called to execute the tasks to be processed based on the generated executable commands, achieving automated task execution, reducing manual intervention, improving execution efficiency, and enabling the distributed scheduling system to operate more efficiently and orderly, thereby better meeting the business's needs for task processing. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figure 1 A schematic diagram of an example structure of a distributed scheduling system provided in an embodiment of this application;

[0054] Figure 2 A flowchart illustrating the distributed scheduling method provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram of the structure of the distributed scheduling system provided in the embodiments of this application. Figure 1 ;

[0056] Figure 4 A schematic diagram of the structure of the distributed scheduling system provided in the embodiments of this application. Figure 2 ;

[0057] Figure 5 This is another example structural diagram of the distributed scheduling system provided in the embodiments of this application;

[0058] Figure 6 This is a schematic diagram of the structure of the distributed scheduling device provided in the embodiments of this application;

[0059] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0060] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] In the process of big data generation and development, it is necessary to rely on a task scheduling system to manage and schedule tasks in a unified manner, ensure the upstream and downstream dependencies between tasks, monitor the task running status, issue timely alarms in case of abnormalities, handle anomalies as automatically as possible, ensure the high availability of the scheduling system, and ensure that the daily data statistics tasks are completed on time.

[0063] However, in existing solutions, task management is chaotic and cannot provide real-time feedback on task execution. Therefore, there is an urgent need for a more efficient solution to schedule and execute tasks.

[0064] Based on the aforementioned technical problems, the inventor's technical concept is as follows: A distributed scheduling system can be designed. After the network end obtains the task to be processed, the storage unit stores the relevant information of the task to be processed. Then, the scheduling unit triggers scheduling from the storage unit. When it can be triggered for execution, the execution unit obtains the task to be processed and generates an executable command corresponding to the task to be processed based on the Python script corresponding to the task type of the task to be processed. Then, the corresponding cluster is called to execute the executable command, thereby achieving efficient processing of the task to be processed.

[0065] Furthermore, in order to detect tasks to be processed in real time, output information and / or error information of threads during execution can be obtained. Based on the obtained information, the execution status of tasks to be processed is fed back to the storage unit in real time so that the network end can view it in real time.

[0066] Figure 1 This is a schematic diagram of an example structure of a distributed scheduling system provided in an embodiment of this application, such as... Figure 1 As shown in the diagram, the structure includes: a network terminal (Web), a scheduling unit (e.g., a master-active scheduler node and a master-standby scheduler node), a coordination service unit (which may be ZooKeeper), an execution unit (e.g., task executor #1-master node, task executor #2-master node, task executor #3-master node), and a storage unit (e.g., MySQL).

[0067] The devices mentioned above will be briefly explained, and their specific implementation is given in the following embodiments, which will not be repeated here.

[0068] 1) Web interface: Responsible for the configuration and maintenance of various task information, including basic information management, dependency management, input / output management, scheduling information configuration, alarm information configuration, etc.

[0069] 2) MySQL: Responsible for storing task description information, execution task information, upstream and downstream information of pending tasks, and detailed execution information, etc.

[0070] 3) master-active: The scheduler master node, mainly responsible for tasks such as triggering, dependency checking, task publishing to ZooKeeper, and task alerting;

[0071] 4) master-standby: The scheduler's master node and standby node, which uses ZooKeeper to identify the master and standby nodes and perform automatic failover and disaster recovery.

[0072] 5) ZooKeeper: Responsible for master-slave failover and managing executable task information;

[0073] 6) executor: Task executor, including modules such as task parser, executor, YARN resource awareness, task information management, and task alarm.

[0074] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0075] Figure 2 This is a flowchart illustrating the distributed scheduling method provided in an embodiment of this application, as shown below. Figure 2 As shown, this method is applied to the execution unit in the aforementioned distributed scheduling system, and includes the following steps:

[0076] Step 21: Obtain the execution task information of the task to be processed from the coordination service unit in the distributed scheduling system;

[0077] In this step, in order to process the task to be processed, the execution unit first obtains the execution task information of the task to be processed from the coordination service unit, such as the ZooKeeper node.

[0078] Among them, the execution task information is based on the upstream and downstream information and detailed execution information of the task to be processed. It is obtained from the execution task information table in the storage unit of the distributed scheduling system. After being determined by the scheduling unit of the coordination service unit, it is stored in the execution task information table.

[0079] In one possible implementation, upstream and downstream information is used to indicate the dependencies of the task to be processed during execution. That is, other tasks or data sources that the task depends on before execution, and subsequent tasks that will be affected after the task is completed. For example, if the execution of the task depends on the execution result of task A, then in actual execution, the task to be processed will only begin after the execution result of task A is obtained. Detailed execution information includes the specific steps of task execution, the required resources, execution parameters, and execution environment requirements.

[0080] Step 22: When it is determined that the task to be processed can be executed based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, generate the executable command corresponding to the task information by executing the Python script corresponding to the task type in the task information.

[0081] In this step, the cluster resource information of the distributed scheduling system can indicate whether the remaining available cluster resources can be used to execute pending tasks, and / or whether there is a pre-set upper limit threshold for the number of tasks to be executed, and is also used to determine whether pending tasks can be executed.

[0082] Furthermore, when all the above requirements are met, the task type of the task to be processed is parsed from the task execution information. In this embodiment, different task types usually have different Python scripts pre-stored for processing. The scripts are programs designed for specific task types and encapsulate the logic and algorithms required to complete the task.

[0083] For example, for a data cleaning task, there might be a script called data_cleaning.py, which contains logic for removing duplicate data and handling missing values.

[0084] Furthermore, based on the task type in the task execution information, after finding the corresponding Python script, other content in the task execution information (such as parameters and specific operations) is integrated together to generate a command that can be directly executed in the operating system, that is, the executable command corresponding to the task execution information.

[0085] In the above implementation, the corresponding Python script can be flexibly called according to different task types, and the specific requirements of the task to be processed can be passed to the script to realize the automated execution of the task to be processed. At the same time, the task information is converted into executable commands, which makes it convenient to run the task in different environments (such as local development environment, production server environment), improving the repeatability and manageability of task execution.

[0086] Optionally, the task type can be: MapReduce task, Hive SQL, Spark task, Spark SQL, shell, Spark-streaming, etc.

[0087] Step 23: Based on the executable command, invoke the corresponding cluster in the distributed scheduling system to execute the task to be processed.

[0088] In this step, by invoking the corresponding cluster to execute the task to be processed according to the executable command, the distributed scheduling system can make full use of the advantages of different clusters to achieve reasonable allocation of resources and efficient execution of tasks.

[0089] Optionally, executing the pending task includes: enabling a thread to execute an executable command; correspondingly, when executing the pending task, it is also possible to: obtain the output information and / or error information of the thread during execution; and update the execution status of the pending task to the execution task information table in the storage unit of the distributed scheduling system based on the output information and / or error information.

[0090] In this implementation, enabling threads to execute executable commands means placing the operation of executing pending tasks into the execution stream, allowing other operations to be processed while the pending task is being executed, thus achieving concurrent execution.

[0091] The output information can include: During the execution of executable commands, the thread may generate some normal output content, such as intermediate results of task execution, progress information, and log records. For example, in a data processing task, the script may output information such as the number of data rows processed and the percentage of processing progress.

[0092] Error messages can be generated by the thread when an exception or error occurs during execution. These error messages can help developers or operations personnel quickly locate the problem. For example, when an input file does not exist, a script syntax error occurs, or there is insufficient memory, the thread will output the corresponding error message.

[0093] Furthermore, the execution status of a task to be processed can have multiple states, such as "running," "successful," and "failed." Based on the obtained output and error information, the execution status of the task can be determined and updated accordingly. For example, if no error information is obtained and the output information shows that the task has been completed normally, the execution status of the task can be updated to "successful"; if an error information is obtained, the execution status can be updated to "failed."

[0094] Finally, the storage unit in the distributed scheduling system is used to persistently store relevant task information. The execution task information table is a data structure within the storage unit, used to record detailed information about each task, including basic task information, execution status, execution time, etc. By updating the task execution status to the execution task information table, the system can easily track and manage tasks (especially since the network end can directly access and view them), and it also provides basic data for subsequent data analysis, monitoring, and reporting.

[0095] The distributed scheduling method provided in this application is applied to the execution unit in a distributed scheduling system. It obtains the execution task information of the task to be processed from the coordination service unit in the distributed scheduling system. When it is determined that the task to be processed can be executed based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, it generates an executable command corresponding to the task information by using the Python script corresponding to the task type in the task information. According to the executable command, it calls the corresponding cluster in the distributed scheduling system to execute the task to be processed. In this technical solution, firstly, the execution task information of the tasks to be processed is obtained from the coordination service unit of the distributed scheduling system. This ensures that the execution unit can accurately understand the specific requirements and details of the tasks, laying the foundation for the correct execution of subsequent tasks. Next, based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, it is determined whether the tasks to be processed can be executed. This resource-task matching mechanism avoids system chaos and task execution failures caused by insufficient resources or excessive tasks, improving the stability and reliability of the system, while also achieving reasonable allocation and efficient utilization of cluster resources. Then, executable commands are generated by using Python scripts corresponding to the task type in the task information. Leveraging the flexibility and powerful functionality of Python scripts, task information can be quickly and accurately transformed into operable commands, adapting to the needs of different types of tasks and enhancing the system's versatility and scalability. Finally, the corresponding cluster in the distributed scheduling system is called to execute the tasks to be processed based on the generated executable commands, achieving automated task execution, reducing manual intervention, improving execution efficiency, and enabling the distributed scheduling system to operate more efficiently and orderly, thereby better meeting the business's needs for task processing.

[0096] Based on the above embodiments, Figure 3 A schematic diagram of the structure of the distributed scheduling system provided in the embodiments of this application. Figure 1 ,like Figure 3 As shown, the distributed scheduling system includes: an execution unit, a scheduling unit, a storage unit, and a coordination service unit;

[0097] Optionally, the scheduling unit generates execution task information based on the upstream and downstream information and detailed execution information of the task to be processed obtained from the storage unit, and stores it in the storage unit.

[0098] In this implementation, the storage unit, such as MySQL, contains the upstream and downstream information and detailed execution information of the task to be processed in the task configuration table. After the scheduling unit obtains the upstream and downstream information and detailed execution information, it generates the execution task information and stores it in the execution task information table in MySQL.

[0099] Optionally, when the scheduling unit detects a ZooKeeper node with pending tasks in the coordination service unit based on the task execution information, it stores the task execution information in the ZooKeeper node.

[0100] In this implementation, the scheduling unit groups tasks by date and retrieves task information waiting to be executed from the task information table, such as the execution task information of tasks to be processed.

[0101] In a distributed system, the coordination service unit (i.e., ZooKeeper) maintains various data nodes (ZooKeeper nodes) in a tree structure. Each ZooKeeper node can store data and can have child nodes. For tasks to be processed, corresponding nodes are created in ZooKeeper to identify and manage these tasks. For example, each task may correspond to a unique ZooKeeper node, and the node's path and name can be determined based on relevant task information.

[0102] The scheduling unit queries ZooKeeper to check if any ZooKeeper nodes are associated with the task to be processed. This is to determine if the task has already been identified and managed in ZooKeeper. If such a node exists, it means that the task is within the management scope of ZooKeeper, possibly because it was created previously or has already been operated on by other components.

[0103] Furthermore, when the scheduling unit detects a ZooKeeper node with pending tasks, it stores the task execution information in that node. At this point, other components (such as the execution unit) can access this ZooKeeper node to obtain the task execution information and perform subsequent task execution operations.

[0104] For the implementation of the execution unit, please refer to the above method embodiments.

[0105] In addition, the scheduling unit registers its online status with the coordination service unit based on a heartbeat mechanism. Accordingly, in subsequent implementations, when the coordination service unit detects that the online status of a scheduling unit has changed to offline status, it switches the execution of the pending task to other scheduling units based on the online status of other scheduling units.

[0106] In this implementation, the coordination service unit continuously monitors the heartbeat messages sent by each scheduling unit to determine the online status of the scheduling units. When the coordination service unit does not receive a heartbeat message from a scheduling unit within a specified time, it will detect that the online status of that scheduling unit has changed to offline status.

[0107] Next, the coordination service unit checks the online status information of other scheduling units. This information helps the coordination service unit determine which scheduling units are available and capable of taking over tasks originally handled by offline scheduling units. The coordination service unit then reassigns the events for executing pending tasks originally allocated to the offline scheduling unit to other online scheduling units. This ensures that pending tasks are not rendered unavailable due to the failure of a scheduling unit, guaranteeing high system availability and continuous task processing.

[0108] In addition, the system also includes a resource coordinator; the resource coordinator, in response to the execution unit's request to obtain resource status, sends the cluster resource information of the distributed scheduling system to the execution unit.

[0109] In one possible implementation, the resource coordinator could be yarn, which can obtain cluster resource information of the distributed scheduling system detected by yarn based on the API interface, in order to determine whether the task to be processed can be executed, and how to allocate the task to the corresponding cluster when it is executed.

[0110] The distributed scheduling system provided in this application includes an execution unit, a scheduling unit, a storage unit, and a coordination service unit. The scheduling unit generates execution task information based on upstream and downstream information and detailed execution information of the task to be processed obtained from the storage unit, and stores this information in the storage unit. When the scheduling unit detects a ZooKeeper node with a task to be processed in the coordination service unit based on the execution task information, it stores the execution task information in the ZooKeeper node. The execution unit executes the above method. In this distributed scheduling system, the scheduling unit obtains upstream and downstream information and detailed execution information of the task to be processed from the storage unit to generate and store execution task information, ensuring accurate task execution planning. By detecting ZooKeeper nodes through the coordination service unit and storing the execution task information when a corresponding node exists, accurate task distribution and management are achieved. The execution unit executes tasks based on the relevant information. The entire process ensures orderly task processing, improves the reliability and accuracy of scheduling, enhances the system's ability to handle large-scale tasks, effectively improves the overall operating efficiency and stability of the system, and guarantees efficient task execution in a distributed environment.

[0111] Based on the above embodiments, Figure 4 A schematic diagram of the structure of the distributed scheduling system provided in the embodiments of this application. Figure 2 ,like Figure 4 As shown, the execution unit includes a Python interpreter; the scheduling unit includes a scheduler, a Quartz task manager, and a task monitor.

[0112] Optionally, the Python parser executes: based on the task type in the task information, it determines the corresponding Python parsing script for the task type; based on the Python parsing script, it generates the executable command corresponding to the task information.

[0113] In this implementation, see the above. Figure 2 The detailed description of the illustrated embodiment.

[0114] Optionally, the scheduler retrieves task description information of the tasks to be processed from the task configuration table of the storage unit at a first preset frequency; and adds the tasks to be processed to the corresponding task list based on the task description information.

[0115] In this implementation, the first preset frequency can be once every 60 seconds, that is, the scheduler monitors the tasks in the task configuration table of the storage unit in real time, obtains the task description information of the tasks to be processed, and stores it in the task list for triggering.

[0116] The task list records basic information about all pending tasks and their respective trigger times. The trigger time can be a specific date and time, or it can be a rule set according to a certain time interval (such as 2:00 AM every day, the 15th minute of every hour, etc.).

[0117] In one possible implementation, the task description information is a summary of the basic information of the task, including the task name, storage address, business type, general content of the task, and the triggering method of the task (such as real-time or timed).

[0118] Optionally, when the Quartz tasker detects that a pending task in the corresponding task list has reached its trigger execution time, it generates execution task information for the pending task based on the upstream and downstream information and detailed execution information of the pending task obtained from the task configuration table, and stores it in the execution task information table in the storage unit.

[0119] In this implementation, the Quartz task generator generates execution task information for the tasks to be processed based on upstream and downstream information and detailed execution information obtained from the task configuration table, combined with the characteristics of the task itself. Execution task information is a comprehensive data set that integrates all the key information required for task execution and serves as an important basis for subsequent task execution. The generated execution task information is stored in the execution task information table of the storage unit.

[0120] The task information table records detailed execution information for each task, facilitating task management, monitoring, and traceability within the system. Subsequent scheduling and execution units can retrieve task information from this table to complete task scheduling and execution operations.

[0121] Optionally, the task monitor performs the following: retrieves task information from the task information table; if no ZooKeeper node with pending tasks is detected in the coordination service unit, creates a ZooKeeper node in the coordination service unit and stores the task information in the ZooKeeper node; or, if a ZooKeeper node with pending tasks is detected in the coordination service unit, stores the task information in the ZooKeeper node.

[0122] In this implementation, see the above. Figure 3 The detailed description of the illustrated embodiment.

[0123] The distributed scheduling system provided in this application embodiment includes an execution unit comprising a Python parser; a scheduling unit comprising a scheduler, a Quartz task generator, and a task monitor; the Python parser executes a Python parsing script corresponding to the task type based on the task type in the task information, and generates an executable command corresponding to the task information based on the Python parsing script; when the Quartz task generator detects that a pending task in the corresponding task list has reached its trigger execution time, it generates execution task information for the pending task based on the upstream and downstream information and detailed execution information of the pending task obtained from the task configuration table, and stores it in the execution task information table of the storage unit; the task monitor executes: retrieves the execution task information from the execution task information table; if no ZooKeeper node for the pending task is detected in the coordination service unit, a ZooKeeper node is created in the coordination service unit, and the execution task information is stored in the ZooKeeper node; or, if a ZooKeeper node for the pending task is detected in the coordination service unit, the execution task information is stored in the ZooKeeper node. In this solution, the Python parser accurately matches the parsing script based on the task type and generates executable commands, ensuring the professionalism and accuracy of task execution. The Quartz task manager, upon the task trigger time, utilizes upstream and downstream information and other details to generate and properly store execution task information, achieving rational task planning and orderly storage. The task monitor continuously monitors the execution task information table and the ZooKeeper node of the coordination service unit. Regardless of whether the node exists, it ensures that the execution task information is correctly stored in the ZooKeeper node, effectively guaranteeing the timeliness and integrity of task scheduling.

[0124] Figure 5 This is another example structural diagram of the distributed scheduling system provided in the embodiments of this application, such as... Figure 5 As shown, this method is applied to the aforementioned distributed scheduling system, and depending on the specific execution device, it may include, for example:

[0125] MySQL includes: a task configuration table and a task execution information table; the Master (scheduler master node) includes: Scheduler, QuartzJob, and TaskWatcher; yarn; zookeeper; the Executor includes: CheckRM, ReadyWatcher, JobExecutor, and a Python interpreter.

[0126] The specific execution steps may include the following:

[0127] Step 1: The Scheduler retrieves task description information from the task configuration table every 60 seconds.

[0128] Step 2: The Scheduler retrieves the list of online tasks and tasks already in the Quartz queue;

[0129] Step 3: Scheduled online / offline tasks in the Scheduler's cyclic task list;

[0130] Step 4: The Scheduler triggers a scheduled task for QuartzJob;

[0131] Step 5: QuartzJob retrieves upstream and downstream information and detailed execution information of the current task (task to be processed) from the task configuration table;

[0132] Step 6: QuartzJob generates execution task information and updates the execution task information table based on the upstream and downstream information and detailed execution information of the current task.

[0133] Step 7: TaskWatcher retrieves task execution information from the task execution information table;

[0134] Step 8: TaskWatcher retrieves information on tasks to be executed by grouping them by date;

[0135] Step 9: TaskWatcher checks if the ZooKeeper node exists and creates a node ready for execution, such as / schedule / ready_{date}.

[0136] Step 10: TaskWatcher checks whether concurrency is supported and whether dependent tasks have been completed, and creates a task node / schedule / ready_{date} / {job_id} on the ZooKeeper node;

[0137] Step 11: ReadyWatcher retrieves node information under the task node from ZooKeeper every 30 seconds;

[0138] Step 12: ReadyWatcher loops through the information of the ready nodes to create the corresponding running node / schedule / running_{date};

[0139] Step 13: ReadyWatcher checks whether the number of pending tasks in execution exceeds the maximum number of tasks that can be executed on a single node;

[0140] Step 14: ReadyWatcher obtains information on ready tasks below the number of executable tasks, and iteratively checks whether resources (cluster resource information) allow execution; if execution is allowed, it generates executable commands using a Python script based on the task type.

[0141] Step 15: ReadyWatcher updates the task status and starts a thread to execute the task;

[0142] Step 16: JobExecutor starts a thread to execute executable commands;

[0143] Step 17: JobExecutor obtains the execution thread info and error input streams, determines whether the execution was successful based on the results, whether retrying is necessary, and updates the execution status.

[0144] The Python interpreter includes: executor-python interpreter and executor-yarn resource control.

[0145] executor-python parser: Based on the task type, it calls different Python parsing scripts to generate corresponding task executable command files for the scheduler to use during actual runtime. Since this part frequently modifies various parameters, it uses Python scripts to generate the files.

[0146] executor-yarn resource control: It obtains the current resource status of the cluster through the Yarn API interface and controls whether the executor can continue to submit tasks to the current cluster;

[0147] In addition, the ZooKeeper part of the overall operation uses Curator distributed locks.

[0148] The distributed scheduling method provided in this application is similar in principle and technical effect to the embodiments described above.

[0149] Figure 6This is a schematic diagram of the structure of the distributed scheduling device provided in the embodiments of this application, as shown below. Figure 6 As shown, the distributed scheduling device provided in this embodiment is applied to the execution unit of a distributed scheduling system, including:

[0150] The acquisition module 61 is used to obtain the execution task information of the task to be processed from the coordination service unit in the distributed scheduling system;

[0151] The determination module 62 is used to generate an executable command corresponding to the task information by executing the Python script corresponding to the task type in the task information when the cluster resource information and / or task upper limit threshold of the distributed scheduling system determine that the task to be processed can be executed.

[0152] Module 63 is used to invoke the corresponding cluster in the distributed scheduling system to execute the pending task based on the executable command.

[0153] In one possible implementation, module 63 is invoked to execute the task to be processed, specifically for:

[0154] Enable thread execution of executable commands;

[0155] Accordingly, module 62 is also used for:

[0156] Obtain the output and / or error information of the thread during execution;

[0157] Based on the output information and / or error information, update the execution status of the tasks to be processed in the execution task information table of the storage unit in the distributed scheduling system.

[0158] The distributed scheduling device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0159] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device provided in this embodiment can be an execution unit in the aforementioned distributed scheduling system, including at least one processor 71 and a memory 72. Optionally, the electronic device further includes a communication component 73. The processor 71, memory 72, and communication component 73 are connected via a bus 74.

[0160] In a specific implementation, at least one processor 71 executes computer execution instructions stored in memory 72, causing at least one processor 71 to perform the above-described method.

[0161] The specific implementation process of processor 71 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0162] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0163] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0164] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0165] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0166] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0167] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0168] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0169] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0170] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0172] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0174] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A distributed scheduling method, characterized in that, An execution unit applied in a distributed scheduling system, the method comprising: Obtain the execution task information of the task to be processed from the coordination service unit in the distributed scheduling system; When it is determined that the task to be processed can be executed based on the cluster resource information and / or task upper limit threshold of the distributed scheduling system, the executable command corresponding to the task information is generated by the Python script corresponding to the task type in the task information. According to the executable command, the corresponding cluster in the distributed scheduling system is invoked to execute the task to be processed.

2. The method according to claim 1, characterized in that, Executing the task to be processed includes: Enable a thread to execute the executable command; Accordingly, the method further includes: Obtain the output information and / or error information of the thread during execution; Based on the output information and / or the error information, update the execution status of the task to be processed to the execution task information table in the storage unit of the distributed scheduling system.

3. A distributed scheduling system, characterized in that, The distributed scheduling system includes: an execution unit, a scheduling unit, a storage unit, and a coordination service unit; The scheduling unit generates execution task information based on the upstream and downstream information and detailed execution information of the task to be processed obtained from the storage unit, and stores it in the storage unit. When the scheduling unit detects a ZooKeeper node containing the task to be processed in the coordination service unit based on the task execution information, the scheduling unit stores the task execution information in the ZooKeeper node. The execution unit executes the method described by right 1 or right 3.

4. The system according to claim 3, characterized in that, The execution unit includes: a Python interpreter; The Python interpreter executes: Based on the task type in the execution task information, the Python parsing script corresponding to the task type is determined; Based on the Python parsing script, an executable command corresponding to the task information is generated.

5. The system according to claim 3, characterized in that, The scheduling unit includes: a scheduler and a Quartz task generator; The scheduler retrieves the task description information of the task to be processed from the task configuration table of the storage unit at a first preset frequency; and adds the task to be processed to the corresponding task list based on the task description information. When the Quartz task generator detects that a pending task in the corresponding task list has reached its trigger execution time, it generates execution task information for the pending task based on the upstream and downstream information and detailed execution information of the pending task obtained from the task configuration table, and stores it in the execution task information table of the storage unit.

6. The system according to claim 5, characterized in that, The scheduling unit further includes: a task monitor; The task monitor executes: Retrieve execution task information from the execution task information table; If no ZooKeeper node for the task to be processed is detected in the coordination service unit, the ZooKeeper node is created in the coordination service unit, and the task execution information is stored in the ZooKeeper node. Alternatively, if a ZooKeeper node for the task to be processed is detected in the coordination service unit, the task execution information is stored in the ZooKeeper node.

7. The system according to claim 3, characterized in that, The scheduling unit registers its online status with the coordination service unit based on a heartbeat mechanism. When the coordination service unit detects that the online status of the scheduling unit has changed to offline status, it switches the event for executing the pending task to the other scheduling unit based on the online status of other scheduling units.

8. The system according to any one of claims 4-7, characterized in that, The system also includes: a resource coordinator; In response to the resource status acquisition request from the execution unit, the resource coordinator sends the cluster resource information of the distributed scheduling system to the execution unit.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in claim 1 or 2.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in claim 1 or 2.

Images