A distributed system and its coordinated scheduling method for job tasks

Abstract

The present invention discloses a distributed system and a coordinated scheduling method for its job tasks. The management end writes the job tasks and operation commands into a message queue. Each distributed consumer end obtains the job tasks and operation commands by subscribing to the message queue, and competes for the task lock from the zookeeper scheduling server. The first consumer end that obtains the task lock implements the operation. After each distributed consumer end monitors that the job task is in the startup state, it queries the status of each process node of the job task in turn. When the status of the process node is found to be "unfinished", the process node is started as the currently executed process node, and the unexecuted shard nodes under the process node are obtained and executed. The present invention realizes the coordinated scheduling of job tasks. The execution states on multiple consumer ends are mutually perceived. The consumer ends can perceive the task execution status of other consumer ends in real time. The consumer end with stronger capabilities will obtain more shards, thereby achieving the effect of dynamic load balancing.

Description

Technical Field

[0001] The present application belongs to the field of distributed real-time computing technology, and in particular relates to a method for coordinating and scheduling distributed systems and job tasks thereof. Background Art

[0002] With the advent of the Internet of Everything and the era of big data, massive data processing and computing tasks rely on increasingly advanced server hardware resources, placing increasing demands on server performance. While high-end servers offer superior performance, they also impose high costs on businesses. Therefore, utilizing affordable PCs to complete massive data processing tasks has become a key industry focus.

[0003] With the deepening application of distributed analytics, an increasing number of distributed products have emerged. However, current distributed products primarily focus on distributed data storage, distributed computing, and distributed services (microservices), while products for distributed task scheduling are relatively scarce. To utilize fewer server resources to process massive data processing and computing tasks, the following issues need to be addressed:

[0004] In a distributed environment, there are differences in machine system time. How can we achieve simultaneous or scheduled concurrency of tasks on multiple machines?

[0005] In a distributed environment, how can we achieve consistency in task execution status across multiple machines while ensuring data independence?

[0006] When an exception occurs (such as a system crash), how can we ensure that the task can continue to be completed normally?

[0007] Most load balancing products use services and tasks as atoms (indivisible) for balanced distribution, and are unable to resolve load imbalances caused by differences in the complexity of services and tasks. How can we achieve true load balancing and maximize the utilization of server resources?

[0008] Only by solving the above problems can we solve the efficient processing of massive data processing tasks. Summary of the Invention

[0009] The purpose of this application is to provide a distributed system and a coordinated scheduling method for its job tasks, so as to achieve consistency in task execution status and high efficiency of task execution in a distributed environment.

[0010] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0011] A method for coordinating and scheduling job tasks in a distributed system, wherein the distributed system includes a management terminal, a distributed consumer terminal, and a Zookeeper scheduling server. The method for coordinating and scheduling job tasks in the distributed system includes:

[0012] The management end writes the job tasks and operation commands into the message queue according to the job task process;

[0013] After each distributed consumer obtains the job task by subscribing to the message queue, it competes for the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock registers the task information with the Zookeeper scheduling server and then releases the task lock.

[0014] Each distributed consumer obtains the operation command of the job task by monitoring the message queue, and obtains the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock;

[0015] After each distributed consumer detects that the job task is in the startup state, it queries the zookeeper scheduling server to query the status of each process node of the job task in turn. When the status of the process node is found to be "incomplete", it starts the process node as the currently executed process node, obtains the unexecuted shard nodes under the process node, and executes them.

[0016] Furthermore, the process of querying the status of each process node of the job task from the zookeeper scheduling server in sequence, and starting the process node as the currently executed process node when the status of the process node is found to be "unfinished", further includes:

[0017] According to the order of each process node of the job task, the next process node will be started only when the status of the previous process node is "Complete".

[0018] Furthermore, the coordinated scheduling method for job tasks in the distributed system further includes:

[0019] After completing the sharding node, modify the sharding status of the sharding node to "completed";

[0020] After completing all shard nodes of the process node, change the status of the process node to "Complete";

[0021] When the status of all process nodes of a job task is "Complete", change the status of the job task to "Complete".

[0022] Furthermore, the coordinated scheduling method for job tasks in the distributed system further includes:

[0023] When each distributed consumer obtains the job task by subscribing to the message queue, if the job task is a scheduled task, the scheduled trigger is started on each consumer;

[0024] When the timing trigger is triggered, the task lock is competed for from the Zookeeper scheduling server, and the task lock is obtained from the Zookeeper scheduling server. The first consumer end that obtains the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock.

[0025] Furthermore, the coordinated scheduling method for job tasks in the distributed system further includes:

[0026] When a consumer fails to execute a shard node, the job status will be changed to "failed";

[0027] After other consumer ends detect the "failed" status of a job task, they stop all shard nodes that are executing the "failed" job task.

[0028] This application also proposes a distributed system, which includes a management end, a distributed consumption end, and a Zookeeper scheduling server, wherein:

[0029] The management end writes the job tasks and operation commands into the message queue according to the job task process;

[0030] After each distributed consumer obtains the job task by subscribing to the message queue, it competes for the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock registers the task information with the Zookeeper scheduling server and then releases the task lock.

[0031] Each distributed consumer obtains the operation command of the job task by monitoring the message queue, and obtains the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock;

[0032] After each distributed consumer detects that the job task is in the startup state, it queries the zookeeper scheduling server to query the status of each process node of the job task in turn. When the status of the process node is found to be "incomplete", it starts the process node as the currently executed process node, obtains the unexecuted shard nodes under the process node, and executes them.

[0033] This application proposes a method for coordinating and scheduling distributed systems and their tasks. It uses ZooKeeper's independent namespace design to achieve unified registration of distributed tasks and ZooKeeper's distributed lock mechanism to achieve state consistency of distributed tasks. It has the following main features:

[0034] Independence: The same task can be split according to rules and executed on different consumer ends, and they are independent of each other;

[0035] Consistency: The same task can be executed simultaneously on multiple consumers, regardless of whether it is started manually or scheduled. Scheduled starts are independent of time differences between consumers.

[0036] Perceptible: The execution status of the same task on multiple consumer terminals is mutually perceptible, and each consumer terminal can perceive the task execution status of other consumer terminals in real time.

[0037] Decentralization: Tasks are executed on multiple consumer terminals without the concept of a master consumer terminal node or elections. This means that the downtime of any consumer terminal will not affect the execution of the task.

[0038] Dynamic load balancing: The execution of a task is split into several shards. After each consumer finishes executing the previous set (or multiple shards), it will obtain the next shard, and so on until all shards for the current task have been allocated. More powerful consumers will receive more shards, achieving dynamic load balancing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the overall architecture of the distributed system of this application.

[0040] Figure 2 This is a flow chart of the coordinated scheduling method for job tasks in a distributed system in this application.

[0041] Figure 3 This is a diagram of the task data structure in Zookeeper in an embodiment of this application.

[0042] Figure 4 This is a distributed task registration flow chart of an embodiment of the present application.

[0043] Figure 5 This is a flow chart of the distributed task startup process according to an embodiment of the present application.

[0044] Figure 6 This is a flow chart of the distributed task execution process of an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] The overall architecture of the distributed system of this application is as follows Figure 1 As shown, it includes a management end, a distributed consumption end, and a Zookeeper scheduling server. To facilitate the understanding of the technical content of this application, some technical terms involved in this application are described as follows:

[0047] Zookeeper scheduling server: an open source distributed application coordination service component with a tree node data structure, and the value changes of any node in Zookeeper can be monitored.

[0048] Quartz: An open source scheduled task trigger component that can trigger tasks according to defined time strategies.

[0049] Kafka message queue: An open-source message queue component. A message queue is a mechanism for transmitting data across a network. It has two call modes: producer and consumer. Producers send messages (data) to a queue with a specific name. Consumers monitor the queue and, when they find data in the queue, retrieve the messages (data) in the order they were generated.

[0050] Trigger: A program language description that triggers an event at a specified time. In this application, it refers to a timing program implementation within Quartz.

[0051] Lock: A computer program processing method used to implement access control. It can be generally understood as a door. When someone (a program) enters the room, the door automatically locks, preventing others (other programs) from entering. When the person (a program) finishes their work and leaves the room, the door automatically unlocks, allowing others (other programs) to enter.

[0052] Task registration: refers to the process of writing the task configuration and execution logic information into ZooKeeper in the form of structured data.

[0053] In one embodiment, Figure 2 As shown, a coordinated scheduling method for job tasks in a distributed system includes:

[0054] Step S1: The management terminal writes the job task and operation command into the message queue according to the job task process.

[0055] The management end of this embodiment provides a visual management tool to better meet practical application needs. It primarily configures the specific workflow of a job task (e.g., SQL execution, Shell script execution, etc.) through a visual method. Through the "Publish" function on the interface, the job task is written to the message queue in JSON format. Distributed consumers obtain the task configuration information by subscribing to the message queue. The management end also supports operations such as starting, pausing, stopping, and deleting job tasks. When the user clicks the corresponding function, the management end transmits the operation command sequence to the distributed consumers via the message queue.

[0056] Step S2: After each distributed consumer obtains the job task by subscribing to the message queue, it competes for the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock registers the task information with the Zookeeper scheduling server and then releases the task lock.

[0057] After each consumer obtains the job task by subscribing to the message queue, it needs to register the job task to the Zookeeper scheduling server. The registration process is as follows Figure 4 As shown in the figure, due to distributed nature, each consumer will register task information with Zookeeper. To avoid duplication of task information in Zookeeper, this embodiment uses Zookeeper's task lock mechanism. Only the first consumer to obtain the task lock is allowed to register task information with Zookeeper. The second and subsequent consumer ends will not register task information with Zookeeper because they cannot obtain the task lock.

[0058] After completing the task registration, the first consumer to acquire the task lock releases the lock and enables task monitoring to ensure it correctly receives status changes for the job. The released lock is then acquired by other consumers. Since the job has already been registered, the lock is immediately released and task monitoring is enabled. This shows that the first consumer to acquire the task lock registers the task information with Zookeeper. When other consumers acquire the task lock again, they no longer need to register the task again because the job has already been registered.

[0059] The data structure diagram of a job registered in ZooKeeper is used to identify the task configuration information and the running status of the task, each task node, and each node shard. / jobs / {jobId} is the root node of a task, where {jobId} is the task ID; / jobs / {jobId} / lock is the global lock node for the task; / jobs / {jobId} / jobConfig is the node for configuring the specific execution process of the task; / jobs / {jobId} / state is the node for the current state of the task; and / jobs / {jobId} / nodes is the root path for storing the running status of each task process node.

[0060] like Figure 3 The zookeeper data structure diagram shown in the figure:

[0061] Consumers node: stores the IP addresses of distributed consumers;

[0062] consumers.ip{1-N} node: a distributed consumer IP address;

[0063] consumers.ip{1-N}.cpunode: CPU usage of a distributed consumer;

[0064] consumers.ip{1-N}.mem node: memory usage of a distributed consumer;

[0065] consumers.ip{1-N}.lastActiveTime node: the last heartbeat time of a distributed consumer;

[0066] globalLock node: a global lock for a task, indicating that the permission for global operations has been obtained by a consumer. Until the lock is released, other consumers have no right to perform global operations.

[0067] Jobs node: All tasks are stored under this node;

[0068] jobs.job{1-N} node: represents a distributed job task;

[0069] jobs.job{1-N}.name node: represents the name of a distributed job task;

[0070] jobs.job{1-N}.jobConfig node: represents the specific process node configuration of a distributed job task, indicating the execution order and execution logic of each process node, and is stored in JSON format;

[0071] jobs.job{1-N}.state node: indicates the current state of a distributed job task, which can be divided into waiting, executing, and completed;

[0072] jobs.job{1-N}.lastRunTime node: indicates the last execution time of a distributed job task;

[0073] jobs.job{1-N}.nodes: This node only stores the process node ID, which represents the specific process node. Its child nodes are used for the shard status information of the process node and the operation control lock of the process node.

[0074] jobs.job{1-N}.nodes.nodeId{1-N} node: stores the ID of a process node of a distributed job task;

[0075] jobs.job{1-N}.nodes.nodeId{1-N}.shardings node: stores the number of shards of a process node of a distributed job task. If there are 10 shards executing this task, there will be 10 sequence nodes from 0 to 9 under this node, which are used to store the status of this shard, indicating whether the shard node is claimed by a distributed consumer.

[0076] jobs.job{1-N}.nodes.nodeId{1-N}.lock node: Stores the execution control lock for a process node in a distributed job task. This indicates that the node has been locked by a distributed consumer. Other consumers cannot perform any operations on this node until the lock is released.

[0077] In this embodiment, task registration refers to the process of writing the task configuration information and execution logic information to be executed into Zookeeper in the form of structured data.

[0078] Step S3: Each distributed consumer obtains the operation command of the job task by monitoring the message queue, and obtains the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock.

[0079] Take the startup operation command as an example, Figure 5 As shown, the user sends a start operation command to the Kafka message queue on the management side. The consumer side obtains the start operation command message by monitoring the Kafka message queue.

[0080] Zookeeper's task lock mechanism is also used for startup commands. After receiving the startup command, each consumer end competes for the task lock from the Zookeeper scheduling server. Only the first consumer end to obtain the task lock will check the status of the job task. If it is in the "Waiting to Start" state, the task status will be changed to "Started" by changing the start node status to "Started" in the Zookeeper scheduling server.

[0081] Similarly, the first consumer end that obtains the task lock will release the task lock after completing the startup. The released task lock will be acquired by other consumers. Since the job task has been started, the task lock will be released directly.

[0082] It should be noted that the same method is used for other operation commands such as pause, stop, and delete, and will not be repeated here. The difference is that after the start operation command, each consumer end starts executing the job task, while after other operation commands, each consumer end stops executing the job task because the job task is in the inactive state.

[0083] Step S4: After each distributed consumer detects that the job task is in the startup state, it queries the zookeeper scheduling server in turn for the status of each process node of the job task. When the status of the process node is found to be "incomplete", the process node is started as the currently executed process node, and the unexecuted shard nodes under the process node are obtained and executed.

[0084] According to the ZooKeeper data structure diagram of this application, a job task includes multiple process nodes, which are numbered. Each process node is split into several independently executable shard nodes and numbered. The shard nodes are independent of each other, that is, the task execution has "independence." In a specific embodiment, the status of a job task includes waiting to start, started, and completed, while the status of a process node and a shard node includes waiting to execute and completed.

[0085] After the job status is changed to "start", because all consumer ends can monitor the change of the job status, each consumer end will start executing the job task at the same time, so the task execution is "consistent". The task execution process is as follows: Figure 6 shown.

[0086] In a specific embodiment, the process of querying the status of each process node of the job task from the zookeeper scheduling server in sequence, and starting the process node as the currently executed process node when the status of the process node is found to be "unfinished", further includes:

[0087] According to the order of each process node of the job task, the next process node will be started only when the status of the previous process node is "Complete".

[0088] Specifically, the task execution process will be executed node by node according to the task flow. When the previous process node is completed, that is, when the status of the previous process node is monitored to become "completed", the next process node will be started.

[0089] After starting a process node, each idle consumer can apply for an unexecuted shard node under the process node, and can only apply for one shard node until all shard nodes are applied for.

[0090] After obtaining the shard node number, the consumer starts executing the shard node and marks the shard node status as "Completed" upon completion. It also checks whether all shards have the "Completed" status. If so, the process node has completed execution and the process node status is marked as "Completed." Otherwise, the consumer continues to request the next idle shard node for the process node until all shard nodes are allocated for execution. When all process nodes for a job have the "Completed" status, meaning all process nodes have completed execution, the job status is changed to "Completed."

[0091] This application is a competitive dynamic task allocation method, called "dynamic load balancing". In this way, the more powerful the consumer end is, the more shard nodes it runs, and the weaker the consumer end is, the fewer shard nodes it runs. This shard node competition mode realizes true dynamic load balancing. At the same time, the downtime of the consumer end will not affect the operation of the overall task because it cannot compete for the execution of any shard node. When all shard nodes are executed, the execution consumer end of the last shard will update the task status of the process node to "completed".

[0092] Different consumer terminals perceive the progress and status of task execution by monitoring the status changes of Zookeeper task nodes, making the technical solution of this application "perceptible." Furthermore, since no additional management program is required to coordinate the execution of each task, a downtime will not affect the normal execution of the next task, thus creating a "decentralized" architecture.

[0093] In another specific embodiment, the coordinated scheduling method for job tasks in a distributed system of the present application further includes:

[0094] When each distributed consumer obtains the job task by subscribing to the message queue, if the job task is a scheduled task, the scheduled trigger is started on each consumer;

[0095] When the timing trigger is triggered, the task lock is competed for from the Zookeeper scheduling server, and the task lock is obtained from the Zookeeper scheduling server. The first consumer end that obtains the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock.

[0096] Some jobs are scheduled. When a distributed consumer receives a job task by subscribing to a message queue and discovers it's a scheduled task, it activates a Quartz scheduled trigger. This means registering the task's trigger with the Quartz task scheduler on that consumer. Registering a Quartz scheduled trigger on each consumer effectively ensures that at least one consumer can initiate the task, preventing failures due to single-machine failures.

[0097] Likewise Figure 5 As shown in the figure, when the timing trigger is triggered, the consumer end competes for the task lock from the Zookeeper scheduling server, and obtains the task lock from the Zookeeper scheduling server. The consumer end that first obtains the task lock modifies the status of the job task in the Zookeeper scheduling server to start, and then releases the task lock.

[0098] In a specific embodiment of the present application, the method for coordinated scheduling of job tasks in a distributed system further includes:

[0099] When a consumer fails to execute a shard node, the job status will be changed to "failed";

[0100] After other consumer ends detect the "failed" status of a job task, they stop all shard nodes that are executing the "failed" job task.

[0101] Specifically, when a consumer fails to execute a shard node, the job status is changed to "failed." Other consumers quickly learn of the job failure by monitoring the Zookeeper task status node. They then quickly stop all threads executing the shard node for the job and stop executing the next process node, effectively terminating the job simultaneously.

[0102] The management side of this application is responsible for visually registering and publishing tasks; the consumer side is responsible for executing tasks. Zookeeper is the coordination component for distributed task execution, storing task information and task status information. Kafka is the connection tool between the management side and the distributed task execution side, issuing command data and receiving task status data. This application splits a task into multiple shards, and the consumer side executes them in a competitive manner, obtaining a shard to execute when it is idle. This ensures that all consumer machines have tasks to execute, maximizing the utilization of consumer resources.

[0103] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for coordinated scheduling of job tasks in a distributed system, characterized in that: The distributed system includes a management end, a distributed consumer end, and a Zookeeper scheduling server. The coordinated scheduling method for job tasks in the distributed system includes: The management end writes the job tasks and operation commands into the message queue according to the job task process; After each distributed consumer obtains the job task by subscribing to the message queue, it competes for the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock registers the task information with the Zookeeper scheduling server and then releases the task lock. Each distributed consumer obtains the operation command of the job task by monitoring the message queue, and obtains the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock; After each distributed consumer detects that the job task is in the startup state, it queries the zookeeper scheduling server in turn for the status of each process node of the job task. When the status of the process node is found to be "incomplete", the process node is started as the currently executed process node, and the unexecuted shard nodes under the process node are obtained and executed. After completing the shard node, the shard status of the shard node is modified to "complete". After completing all shard nodes of the process node, the process node status is modified to "complete". When the status of all process nodes of the job task is "complete", the status of the job task is modified to "complete".

2. The coordinated scheduling method for job tasks in a distributed system according to claim 1, characterized in that: The process of querying the status of each process node of the job task from the zookeeper scheduling server in sequence, and starting the process node as the currently executed process node when the status of the process node is found to be "unfinished", further includes: According to the order of each process node of the job task, the next process node will be started only when the status of the previous process node is "Complete".

3. The coordinated scheduling method for job tasks in a distributed system according to claim 1, characterized in that: The coordinated scheduling method for job tasks in the distributed system further includes: When each distributed consumer obtains the job task by subscribing to the message queue, if the job task is a scheduled task, the scheduled trigger is started on each consumer; When the timing trigger is triggered, the task lock is competed for from the Zookeeper scheduling server, and the task lock is obtained from the Zookeeper scheduling server. The first consumer end that obtains the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock.

4. The coordinated scheduling method for job tasks in a distributed system according to claim 1, characterized in that: The coordinated scheduling method for job tasks in the distributed system further includes: When a consumer fails to execute a shard node, the job status will be changed to "failed"; After other consumer ends detect the "failed" status of a job task, they stop all shard nodes that are executing the "failed" job task.

5. A distributed system, characterized in that: The distributed system includes a management terminal, a distributed consumption terminal and a Zookeeper scheduling server, wherein: The management end writes the job tasks and operation commands into the message queue according to the job task process; After each distributed consumer obtains the job task by subscribing to the message queue, it competes for the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock registers the task information with the Zookeeper scheduling server and then releases the task lock. Each distributed consumer obtains the operation command of the job task by monitoring the message queue, and obtains the task lock from the Zookeeper scheduling server. The first consumer to obtain the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock. After each distributed consumer detects that the job task is in the startup state, it queries the zookeeper scheduling server in turn for the status of each process node of the job task. When the status of the process node is found to be "incomplete", the process node is started as the currently executed process node, and the unexecuted shard nodes under the process node are obtained and executed. After completing the shard node, the shard status of the shard node is modified to "complete". After completing all shard nodes of the process node, the process node status is modified to "complete". When the status of all process nodes of the job task is "complete", the status of the job task is modified to "complete".

6. The distributed system according to claim 5, characterized in that The process of querying the status of each process node of the job task from the zookeeper scheduling server in sequence, and starting the process node as the currently executed process node when the status of the process node is found to be "unfinished", further includes: According to the order of each process node of the job task, the next process node will be started only when the status of the previous process node is "Complete".

7. The distributed system according to claim 5, characterized in that The distributed system further includes: When each distributed consumer obtains the job task by subscribing to the message queue, if the job task is a scheduled task, the scheduled trigger is started on each consumer; When the timing trigger is triggered, the task lock is competed for from the Zookeeper scheduling server, and the task lock is obtained from the Zookeeper scheduling server. The first consumer end that obtains the task lock modifies the status of the job task in the Zookeeper scheduling server and then releases the task lock.

8. The distributed system according to claim 5, characterized in that The distributed system further includes: When a consumer fails to execute a shard node, the job status will be changed to "failed"; After other consumer ends detect the "failed" status of a job task, they stop all shard nodes that are executing the "failed" job task.

Images