Distributed scheduled task distribution processing method and system based on ZooKeeper

Abstract

The present invention provides a distributed scheduled task distribution processing method and system based on ZooKeeper, comprising: connecting a service to a ZooKeeper module; the scheduled task distribution service monitoring the ZooKeeper module to obtain all surviving services in the service and assigning corresponding numbers to the surviving services; obtaining all pending tasks to obtain task sets corresponding to the numbers; mapping the task sets' corresponding numbers to the corresponding services of the task processors, and distributing the task sets to the corresponding task processors for execution. By centrally managing the triggering and distribution of scheduled tasks, the present invention frees developers from implementing scheduled tasks and distributed locks, reducing development difficulty and improving development efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a distributed timed task distribution processing method and system based on ZooKeeper. Background Art

[0002] ZooKeeper is a distributed, open-source coordination service for distributed applications. It is an open-source implementation of Google's Chubby and a key component of Hadoop and HBase. It provides consistency services for distributed applications, including configuration maintenance, domain name services, distributed synchronization, and group services.

[0003] Patent document CN112433830A discloses a distributed task scheduling method, system and storage medium based on ZooKeeper. The method includes: using a scheduling center server to create temporary nodes to a ZooKeeper cluster, where each temporary node corresponds to a node serial number; using the scheduling center server to perform an election based on the node serial number of the temporary node to select a temporary node from all temporary nodes to trigger a task.

[0004] However, in patent document CN112433830A, only one server can obtain the scheduled task execution right, and cluster service resources cannot be fully utilized.

[0005] Patent document CN105100259A discloses a distributed scheduled task execution method and system, which includes: a system cluster module maintains a long connection with a registered server node and generates a server list; a task distribution module allocates a server to a received scheduled task according to the server list, and stores the binding relationship between the scheduled task and the server in a database; a task loading module periodically scans the scheduled task table in the database, loads the scheduled tasks to be executed within the period into the server, and creates a scheduled task; and a task execution module executes the scheduled task.

[0006] However, the method disclosed in patent document CN105100259A requires storing the binding relationship between the scheduled task and the server in a database. If the server corresponding to the binding relationship stored in the database suddenly crashes for some reason, the tasks associated with the server in the database cannot be executed. Secondly, when the number of scheduled tasks increases sharply, the binding relationship needs to be frequently updated and maintained, which affects the efficiency of the overall scheduled task execution. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a distributed scheduled task distribution processing method and system based on ZooKeeper.

[0008] According to the present invention, a distributed scheduled task distribution processing method based on ZooKeeper is provided, comprising:

[0009] Step S1: Connect the service with the ZooKeeper module;

[0010] Step S2: The scheduled task distribution service monitors the ZooKeeper module, obtains the service, processes it, and distributes it to the corresponding task processor for execution.

[0011] Preferably, step S2 is processed in an independent scheduled task distribution service module and includes the following sub-steps:

[0012] Step S2.1: Obtain all surviving services in the service and assign corresponding numbers to the surviving services;

[0013] Step S2.2: Obtain all pending tasks and obtain a task set corresponding to the number;

[0014] Step S2.3: Map the corresponding numbers of the task sets to the corresponding services of the task processors, and distribute the task sets to the corresponding task processors for execution.

[0015] Preferably, there are one or more task processors;

[0016] The task processor includes a task receiving interface, which can receive and process tasks distributed by the scheduled task distribution service module; can implement the interface according to scheduled tasks of different business types; and encode the task processor according to different business types so that the scheduled tasks and task processors are matched one-to-one.

[0017] Preferably, step S2.2 includes:

[0018] Step S2.2.1: Get all pending tasks and filter the number of tasks to be processed in the current task according to the task cycle configured;

[0019] Step S2.2.2: Get all task cycles and calculate the next most recent task cycle trigger time T1. If there is no pending task or the calculated next trigger cycle is greater than the set threshold T0, the next trigger cycle of the scheduled task distribution service is T0, otherwise it is T1.

[0020] Preferably, step S2.3 includes:

[0021] Step S2.3.1: The scheduled task distribution service calls the corresponding main service task receiving interface;

[0022] Step S2.3.2: The task receiver obtains the task set and traverses the task set, and distributes it to the corresponding task processor for processing according to the business type code;

[0023] Step S2.3.3: The task processor processes the corresponding task and determines whether the current task processing is abnormal. If not, the current task status in the database is updated to processing completed; if so, step S2.3.4 is triggered;

[0024] Step S2.3.4: Determine whether the current task version is greater than the set value. If so, update the database and set the current task status to failure and do not enter the next task cycle. If not, increase the current task data version number by one, triggering step S2.2 to continue execution in the next cycle.

[0025] According to the present invention, a distributed scheduled task distribution and processing system based on ZooKeeper is provided, comprising:

[0026] Module M1: connects the service with the ZooKeeper module;

[0027] Module M2: The scheduled task distribution service monitors the ZooKeeper module, obtains the service, processes it, and distributes it to the corresponding task processor for execution.

[0028] Preferably, the module M2 is processed in an independent scheduled task distribution service module, including the following submodules:

[0029] Module M2.1: Obtain all surviving services in the service and assign corresponding numbers to the surviving services;

[0030] Module M2.2: Obtain all pending tasks and obtain the task set corresponding to the number;

[0031] Module M2.3: Map the corresponding numbers of the task sets to the corresponding services of the task processors, and distribute the task sets to the corresponding task processors for execution.

[0032] Preferably, there are one or more task processors;

[0033] The task processor includes a task receiving interface, which can receive and process tasks distributed by the scheduled task distribution service module; can implement the interface according to scheduled tasks of different business types; and encode the task processor according to different business types so that the scheduled tasks and task processors are matched one-to-one.

[0034] Preferably, module M2.2 includes:

[0035] Module M2.2.1: Get all pending tasks and filter the number of tasks to be processed in the current task according to the task cycle configured;

[0036] Module M2.2.2: Get all task cycles and calculate the next most recent task cycle trigger time T1. If there is no pending task or the calculated next trigger cycle is greater than the set threshold T0, the next trigger cycle of the scheduled task distribution service is T0, otherwise it is T1.

[0037] Preferably, module M2.3 includes:

[0038] Module M2.3.1: Scheduled task distribution service calls the corresponding main service task acceptance interface;

[0039] Module M2.3.2: The task receiver obtains the task set and traverses the task set, and distributes it to the corresponding task processor for processing according to the business type code;

[0040] Module M2.3.3: The task processor processes the corresponding task and determines whether the current task processing is abnormal. If not, it updates the database current task status to complete. If so, it triggers step S2.3.4;

[0041] Module M2.3.4: Determine whether the current task version is greater than the set value. If so, update the database current task status to failure and do not enter the next task cycle; if not, increase the current task data version number by one, triggering module M2.2 to continue execution in the next cycle.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention centrally manages the triggering and distribution of scheduled tasks, freeing developers from the implementation of scheduled tasks and distributed locks, reducing development difficulty and improving development efficiency.

[0044] 2. The present invention monitors surviving services and adopts an independent scheduled task distribution service to automatically distribute the acquired tasks to be processed to the corresponding task processors, making full use of cluster service resources and solving the problem that distributed locks and distributed scheduled tasks can only obtain the scheduled task execution right by a single service node.

[0045] 3. By adding the concept of task version, the present invention improves the fault tolerance during the execution of scheduled tasks, and has good resistance to abnormal situations such as system downtime and network band in a short period of time.

[0046] 4. In response to the situation where the number of scheduled tasks increases dramatically, the present invention establishes a strong logical binding relationship between the task processor and the actual task, which does not cause additional performance overhead and saves resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0048] Figure 1 Schematic diagram of the system framework of the present invention.

[0049] Figure 2 Schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0051] The present invention uses an independent scheduled task distribution service to monitor the number of surviving main services on the registration center, number the main services, obtain tasks to be processed, take the remainder based on the number of surviving main services, and then map them to the main services. The tasks are distributed according to the mapping relationship.

[0052] Example 1

[0053] According to a distributed timed task distribution processing method based on ZooKeeper provided by the present invention, Figure 1 and Figure 2 As shown, including:

[0054] Step S1: Connect the service to the ZooKeeper module. The connection method includes registering the service to ZooKeeper.

[0055] It should be noted that there are one or more task processors in the present invention. The task processor includes a task receiving interface, which can receive and process tasks distributed by the scheduled task distribution service module; can implement the interface according to scheduled tasks of different business types; and encode the task processor according to different business types, so that scheduled tasks and task processors are matched one by one. Furthermore, according to the actual scenario, the scheduled tasks to be processed are classified, and type codes are defined for them to distinguish them. For example, the type code is defined as L n , where n represents the number of types. After the task types are distinguished, the task processor interface is defined, the task processor is implemented for tasks of different business types, and the processor is coded. For example, the coding positioning is DL n, where n represents the number of types. Then, scheduled tasks of the same type are matched one-to-one with task processors. That is, the L1 scheduled task corresponds to processor DL1, and the L2 scheduled task corresponds to processor DL2.

[0056] Step S2: The scheduled task distribution service monitors the ZooKeeper module, obtains the service, processes it, and distributes it to the corresponding task processor for execution.

[0057] Specifically, step S2 is processed in an independent scheduled task distribution service module and includes the following sub-steps:

[0058] Step S2.1: Obtain all surviving services in the service and assign corresponding numbers to the surviving services, for example, [0, N-1].

[0059] Step S2.2: Obtain all pending tasks and obtain the task set corresponding to the number. Specifically, the scheduled task distribution service retrieves all pending task records from the database, then takes the remainder of the N surviving services based on the task record primary key ID to obtain the task set numbered [0, N-1]. Task records are generated by the business system, including but not limited to being stored in the database task record table. The database generates a unique and ordered primary key ID for each task record, such as an auto-incrementing primary key in MySQL or a sequence in Oracle, and stores the task classification.

[0060] Specifically, the sub-steps of step S2.2 are as follows:

[0061] Step S2.2.1: Obtain all pending tasks from the database, read the configuration file to configure the task period for scheduled tasks of different task categories, and filter the number of tasks to be processed in the current task. The task period can be configured using corn expressions.

[0062] Step S2.2.2: Get all task cycles and calculate the next most recent task cycle trigger time T1. If there is no pending task or the calculated next trigger cycle is greater than the set threshold T0, the next trigger cycle of the scheduled task distribution service is T0, otherwise it is T1.

[0063] Step S2.3: Map the task set corresponding number to the task processor corresponding service, that is, map the task processor corresponding service [0, N-1] according to the task set number [0, N-1] and distribute the task set to the corresponding task processor for execution.

[0064] Specifically, the sub-steps of step S2.3 are as follows:

[0065] Step S2.3.1: The scheduled task distribution service calls the corresponding main service task receiving interface;

[0066] Step S2.3.2: The task receiver obtains the task set and traverses the task set, and distributes it to the corresponding task processor for processing according to the business type code. Specifically, the task classification of each task is obtained by traversing the task set, and then the task classification L is used to classify each task. n With the task processor DL n One-to-one matching relationship, specific tasks are assigned to corresponding task processors for processing.

[0067] Step S2.3.3: The task processor processes the corresponding task and determines whether the current task processing is abnormal. If not, it updates the database to indicate that the current task status is complete. If so, it triggers step S2.3.4. Abnormalities that occur during task processing include unknown abnormalities such as network fluctuations and environmental problems.

[0068] Step S2.3.4: Determine whether the current task version is greater than the set value. If so, update the database and set the current task status to failure and do not enter the next task cycle. If not, increase the current task data version number by one, triggering step S2.2 to continue execution in the next cycle.

[0069] Example 2

[0070] The present invention also provides a distributed timed task distribution and processing system based on ZooKeeper. Those skilled in the art can implement the distributed timed task distribution and processing system based on ZooKeeper by executing the step flow of the distributed timed task distribution and processing method based on ZooKeeper, that is, the distributed timed task distribution and processing method based on ZooKeeper can be understood as an optimal implementation of the distributed timed task distribution and processing system based on ZooKeeper.

[0071] According to the present invention, a distributed scheduled task distribution and processing system based on ZooKeeper is provided, comprising:

[0072] Module M1: Connects the service with the ZooKeeper module.

[0073] Module M2: The scheduled task distribution service listens to the ZooKeeper module, obtains the service, processes it, and then distributes it to the corresponding task processor for execution. There can be one or more task processors. The task processors include a task receiving interface that receives and processes tasks distributed by the scheduled task distribution service module. These interfaces can be implemented based on scheduled tasks of different business types. Task processors are coded based on different business types, ensuring a one-to-one correspondence between scheduled tasks and task processors.

[0074] Specifically, module M2 is processed in an independent scheduled task distribution service module and includes the following submodules:

[0075] Module M2.1: Obtain all surviving services in the service and assign corresponding numbers to the surviving services.

[0076] Module M2.2: Obtain all pending tasks and obtain the task set corresponding to the number. Module M2.2 includes:

[0077] Module M2.2.1: Get all pending tasks and filter the number of tasks to be processed in the current task according to the task cycle configured;

[0078] Module M2.2.2: Get all task cycles and calculate the next most recent task cycle trigger time T1. If there is no pending task or the calculated next trigger cycle is greater than the set threshold T0, the next trigger cycle of the scheduled task distribution service is T0, otherwise it is T1.

[0079] Module M2.3: Map the task set corresponding number to the task processor corresponding service, and distribute the task set to the corresponding task processor for execution. Module M2.3 includes:

[0080] Module M2.3.1: Scheduled task distribution service calls the corresponding main service task acceptance interface;

[0081] Module M2.3.2: The task receiver obtains the task set and traverses the task set, and distributes it to the corresponding task processor for processing according to the business type code;

[0082] Module M2.3.3: The task processor processes the corresponding task and determines whether the current task processing is abnormal. If not, it updates the database to indicate that the current task status is complete. If so, it triggers step S2.3.4. Abnormalities that occur during task processing include unknown abnormalities such as network fluctuations and environmental problems.

[0083] Module M2.3.4: Determine whether the current task version is greater than the set value. If so, update the database current task status to failure and do not enter the next task cycle; if not, increase the current task data version number by one, triggering step S2.2 to continue execution in the next cycle.

[0084] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0085] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A distributed timed task distribution processing method based on ZooKeeper, characterized in that: include: Step S1: Connect the service with the ZooKeeper module; Step S2: The scheduled task distribution service monitors the ZooKeeper module, obtains the service, processes it, and distributes it to the corresponding task processor for execution; Step S2 is processed in an independent scheduled task distribution service module and includes the following sub-steps: Step S2.1: Obtain all surviving services in the service and assign corresponding numbers to the surviving services; Step S2.2: Obtain all pending tasks and obtain a task set corresponding to the number; Step S2.3: Map the corresponding numbers of the task sets to the corresponding services of the task processors, and distribute the task sets to the corresponding task processors for execution; Step S2.2 includes: Step S2.2.1: Get all pending tasks and filter the number of tasks to be processed in the current task according to the task cycle configured; Step S2.2.2: Get all task cycles and calculate the next most recent task cycle trigger time T1. If there is no pending task or the calculated next trigger cycle is greater than the set threshold T0, the next trigger cycle of the scheduled task distribution service is T0, otherwise it is T1; Step S2.3 includes: Step S2.3.1: The scheduled task distribution service calls the corresponding main service task receiving interface; Step S2.3.2: The task receiver obtains the task set and traverses the task set, and distributes it to the corresponding task processor for processing according to the business type code; Step S2.3.3: The task processor processes the corresponding task and determines whether the current task processing is abnormal. If not, the current task status in the database is updated to processing completed; if so, step S2.3.4 is triggered; Step S2.3.4: Determine whether the current task version is greater than the set value. If so, update the database and set the current task status to failure and do not enter the next task cycle. If not, increase the current task data version number by one, triggering step S2.2 to continue execution in the next cycle.

2. The distributed scheduled task distribution processing method based on ZooKeeper according to claim 1, characterized in that: There are one or more task processors; The task processor includes a task receiving interface, which can receive and process tasks distributed by the scheduled task distribution service module; can implement the interface according to scheduled tasks of different business types; and encode the task processor according to different business types so that the scheduled tasks and task processors are matched one-to-one.

3. A distributed timed task distribution and processing system based on ZooKeeper, characterized in that: include: Module M1: connects the service with the ZooKeeper module; Module M2: The scheduled task distribution service monitors the ZooKeeper module, obtains the service, processes it, and distributes it to the corresponding task processor for execution; The module M2 is processed in an independent scheduled task distribution service module and includes the following submodules: Module M2.1: Obtain all surviving services in the service and assign corresponding numbers to the surviving services; Module M2.2: Obtain all pending tasks and obtain the task set corresponding to the number; Module M2.3: Map the task set corresponding number to the task processor corresponding service, and distribute the task set to the corresponding task processor for execution; Module M2.2 includes: Module M2.2.1: Get all pending tasks and filter the number of tasks to be processed in the current task according to the task cycle configured; Module M2.2.2: Get all task cycles and calculate the next most recent task cycle trigger time T1. If there are no pending tasks or the calculated next trigger cycle is greater than the set threshold T0, the next trigger cycle of the scheduled task distribution service is T0, otherwise it is T1; Module M2.3 includes: Module M2.3.1: Scheduled task distribution service calls the corresponding main service task acceptance interface; Module M2.3.2: The task receiver obtains the task set and traverses the task set, and distributes it to the corresponding task processor for processing according to the business type code; Module M2.3.3: The task processor processes the corresponding task and determines whether the current task processing is abnormal. If not, it updates the database current task status to complete. If so, it triggers step S2.3.4; Module M2.3.4: Determine whether the current task version is greater than the set value. If so, update the database current task status to failure and do not enter the next task cycle; if not, increase the current task data version number by one, triggering module M2.2 to continue execution in the next cycle.

4. The distributed scheduled task distribution processing system based on ZooKeeper according to claim 3, characterized in that: There are one or more task processors; The task processor includes a task receiving interface, which can receive and process tasks distributed by the scheduled task distribution service module; can implement the interface according to scheduled tasks of different business types; and encode the task processor according to different business types so that the scheduled tasks and task processors are matched one-to-one.

Images