Highly Available Distributed Concurrent Task Scheduling System and Method

Abstract

The present invention provides a highly available distributed concurrent task scheduling system and method, including: according to the task scheduling policy, calling a task processing link through a call request to process a task; and processing the task according to the call request. By constructing a distributed concurrent task scheduling system, the present invention realizes effective control of node concurrent tasks in a distributed system, reduces repeated query actions, avoids waste of computing resources, reduces bandwidth occupancy, and also adopts a dual detection mechanism to control the concurrent state controller in the system, avoiding the dirty read problem caused by cluster task concurrency; moreover, the distributed deployment method of the present invention ensures the high availability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and particularly to a highly available distributed concurrent task scheduling system and method. In particular, it is a concurrent task scheduling solution with high availability in a distributed system. Background Art

[0002] Patent document CN106100779A provides an emergency broadcast task regulation method and system based on timed polling and real-time scheduling to solve the problem that the prior art cannot effectively regulate tasks of complex types issued by multi-level platforms. The system includes a timed polling module, a real-time scheduling module, an instruction processing module, and a fault processing module. It uses the timed polling process and the real-time scheduling process to regulate the timed and real-time broadcast tasks of multi-level platforms, and finally performs unified processing by the instruction processing process. The cyclic broadcast tasks are processed in the form of subtasks.

[0003] However, the control strategy for node concurrent tasks in the prior art under a distributed system needs to be optimized. Summary of the Invention

[0004] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a highly available distributed concurrent task scheduling system and method.

[0005] According to a highly available distributed concurrent task scheduling system provided by the present invention, it includes:

[0006] An automatic task scheduling module: According to the task scheduling strategy, call the task processing module through a call request to perform task processing;

[0007] A task processing module: Perform task processing according to the call request.

[0008] Preferably, the automatic task scheduling module includes:

[0009] A task poller: Execute the task scheduling strategy. In the task scheduling strategy, regularly check the concurrent state controller to see if there is data to be processed; if there is data to be processed and no node currently acquires the concurrent state controller, initiate an action to acquire the concurrent state controller; if the acquisition fails, it is considered that it has been acquired in advance by other nodes in the cluster, and the current node's current poll ends; if the acquisition is successful, call the task processing module of the current node to perform task processing, and return the concurrent state controller after processing; among them, after obtaining the concurrent state controller, verify the acquisition result;

[0010] A concurrent state controller: Adopt a double detection mechanism;

[0011] Among them, after the application of each node is in the active state, the task poller starts working; if the concurrent state controller is in the ready state, the concurrent state controller is obtained and the acquisition status is checked again; if the acquisition is successful, the processing continues, otherwise, the current processing ends;

[0012] The concurrent state controller has several states: ready state, running state, and abnormal state; the acquisition of the concurrent state controller is an update action, which updates a piece of data in the ready state to the running state: the acquisition state refers to whether the ready state is successfully updated to the running state.

[0013] Preferably, the automatic task scheduling module has a copy on each node; when some nodes fail, the automatic task scheduling module provides scheduling services normally through the copies of the nodes that have not failed; all task pollers share a state controller, so that the task processing module will only run on one node at a time;

[0014] Among them, the concurrent state controller constrains the execution frequency of the task processing module on the node, so that a piece of data will only be effectively processed by one node and only once; if an exception occurs in the query, the task processing module obtains the data again for processing.

[0015] Preferably, the highly available distributed concurrent task scheduling system is deployed using a homogeneous distributed cluster, with identical copies on each node; service calls are periodically initiated to the core transaction system via the enterprise application integration platform EAIP; the highly available distributed concurrent task scheduling system obtains query parameters according to the scheduling mode, then constructs a request message based on the query parameters and initiates a query request to the core transaction system via the enterprise application integration platform EAIP; the query message is obtained from the core transaction system, parsed, and persisted in a database; and the query results corresponding to the query message are fed back to the user in real time;

[0016] The highly available distributed concurrent task scheduling system further includes:

[0017] Manual trigger module: provides users with real-time access to the task processing module. The manual trigger module triggers the task processing module to perform logical processing and feeds back the processing results to the user. The manual trigger module adjusts the concurrent state controller to disable the automatic task scheduling module based on the user's manual modifications on the page.

[0018] A highly available distributed concurrent task scheduling method provided by the present invention includes:

[0019] Automatic task scheduling step: According to the task scheduling strategy, the task processing step is called by calling the request to perform task processing;

[0020] Task processing steps: Perform task processing according to the said call request.

[0021] Preferably, the automatic task scheduling step includes:

[0022] Task polling step: Perform task polling; wherein, the task polling includes executing the task scheduling policy, in which the concurrent status controller is regularly checked to see if there is data to be processed; if there is data to be processed and no node currently obtains the concurrent status controller, an action to obtain the concurrent status controller is initiated; if the acquisition fails, it is considered that it has been obtained in advance by other nodes in the cluster, and the current node's current polling ends; if the acquisition is successful, the task processing step of the current node is called to perform task processing, and the concurrent status controller is returned after processing; wherein, after obtaining the concurrent status controller, the acquisition result is verified.

[0023] Concurrent status controller: Adopt a double detection mechanism.

[0024] Among them, after the applications of each node are in an active state, the task polling starts to work; if the concurrent status controller is in a ready state, the concurrent status controller is obtained, and the acquisition status is checked again; if the acquisition is successful, the processing continues, otherwise, the current processing ends.

[0025] The concurrent status controller has several states: ready state, running state, abnormal state; obtaining the concurrent status controller is an update action that updates a piece of data in the ready state to the running state: the acquisition status refers to whether the ready state is successfully updated to the running state.

[0026] Preferably, in the automatic task scheduling step, there is a copy on each node; when some nodes fail, the automatic task scheduling step provides scheduling services normally through the copies of the nodes that have not failed; all the task pollings share a status controller, so that the task processing step can only run on one node at a time.

[0027] Among them, the concurrent status controller restricts the execution frequency of the task processing step on the node, so that a piece of data can only be effectively processed by one node and only once; if an exception occurs during the query, the task processing step obtains the piece of data again for processing.

[0028] Preferably, in the highly available distributed concurrent task scheduling method, a homogeneous distributed cluster is adopted for deployment, with the same replicas on each node; service calls are initiated to the core transaction system at regular intervals through the Enterprise Application Integration Platform (EAIP); in the highly available distributed concurrent task scheduling method, query parameters are obtained according to the scheduling mode, and then a request message is constructed based on the query parameters and a query request is sent to the core transaction system through the Enterprise Application Integration Platform (EAIP). The query message is obtained, parsed from the core transaction system and persisted to the database, and at the same time, the query result corresponding to the query message is fed back to the user in real time;

[0029] The highly available distributed concurrent task scheduling method further includes:

[0030] Manual trigger step: Provide a service for the user to access the task processing step in real time, and the manual trigger step triggers the task processing step to perform logical processing and feeds back the processing result to the user; in the manual trigger step, according to the manual modification of the user on the page, the concurrent state controller is adjusted to deactivate the automatic task scheduling step.

[0031] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the highly available distributed concurrent task scheduling method are implemented.

[0032] According to an electronic device provided by the present invention, including a memory, a processor, and a computer program stored on the memory and executable on the processor, when the computer program is executed by the processor, the steps of the highly available distributed concurrent task scheduling method are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. By constructing a distributed concurrent task scheduling system, the present invention realizes the effective control of node concurrent tasks in a distributed system.

[0035] 2. By using a concurrent state controller to restrict the execution frequency of the task processing module on the node, the present invention ensures that a piece of data is only processed effectively by one node and only once. Of course, if an exception occurs during the query, the task processing module will obtain the piece of data again for processing, thereby reducing repeated query actions, avoiding waste of computing resources, and reducing bandwidth occupancy.

[0036] 3. The present invention adopts a dual detection mechanism to control the concurrent state controller in the system, avoiding the dirty read problem caused by cluster task concurrency; moreover, the distributed deployment method of the present invention ensures the high availability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objectives, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 It is a schematic structural diagram of a highly available concurrent task scheduling system under a distributed system.

[0039] Figure 2 It is a schematic diagram of the processing logic flow steps of the task processing module.

[0040] Figure 3 It is a schematic diagram of the processing logic flow steps of the automatic task scheduler.

[0041] Figure 4 It is a schematic diagram of the principle of the deployment mode of the automatic task scheduling module. Detailed Embodiments

[0042] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all fall within the protection scope of the present invention.

[0043] A highly available distributed concurrent task scheduling system provided by the present invention includes: the automatic task scheduling module, according to the task scheduling strategy, calls the task processing module to perform task processing by sending a request; the task processing module performs task processing according to the call request; the manual trigger module provides the user with real-time access to the service of the task processing module, the manual trigger module triggers the task processing module to perform logical processing, and feeds back the processing result to the user; the manual trigger module adjusts the concurrent state controller to deactivate the automatic task scheduling module according to the user's manual modification on the page. In the present invention, a manual trigger module is designed, and the user can immediately access the task processing module through this manual trigger module, avoiding the user's waiting.

[0044] Among them, the two core components of the automatic task scheduling module are: the task poller and the concurrent state controller. The automatic task scheduling module regularly checks the concurrent state controller through the task poller to see if there is data to be processed and if there is a node processing. If there is data to be processed and no node currently obtains the concurrent state controller, an action to obtain the concurrent state controller is initiated. To ensure that the concurrent state controller can be truly obtained, the acquisition result is verified. If the acquisition fails, it means that it has been obtained in advance by other nodes in the cluster, and the current node's current poll ends. If the concurrent state controller is successfully obtained, the task processing module of the current node is called to perform task processing, and the concurrent state controller is returned after the processing is completed.

[0045] A complete system is deployed on each node of the present invention. As long as one node is active, the system can provide normal services. As Figure 1 shown is the highly available distributed concurrent task scheduling system constructed by the present invention; the present invention will be described in more detail below.

[0046] The automatic task scheduling module includes two processors: a task poller and a concurrent status controller respectively. The highly available distributed concurrent task scheduling system is deployed using a homogeneous distributed cluster, and the highly available distributed concurrent task scheduling system has the same replicas on each node.

[0047] The highly available distributed concurrent task scheduling system is deployed in the form of a distributed cluster. The application cluster consists of 5 physical devices. In a variation, the number of physical devices can also be increased or decreased. According to business requirements, the scheduling system needs to initiate a service call to the core transaction system regularly through the Enterprise Application Integrated Platform (EAIP).

[0048] The automatic task scheduling module processes the concurrent status controller using a dual detection mechanism. Compared with single detection, it avoids the dirty read problem caused by concurrent cluster tasks. After the applications on each node become active, the task poller of the automatic task scheduling module starts to work. Taking 10 minutes as a time unit, it initiates a poll to the concurrent status controller. If the concurrent status controller is in the ready state, the concurrent status controller is obtained. And a secondary check is performed on the obtained status. If the acquisition is successful, the process continues; otherwise, this processing ends. The concurrent status controller has several states: ready state, running state, and abnormal state. The obtaining of the concurrent status controller is an update action that updates a piece of data in the "ready state" to the "running state". The obtaining of the status refers to whether the ready state is successfully updated to the running state.

[0049] The manual trigger module allows users to initiate an immediate request. The task processing module immediately performs logical processing and feeds back the processing result to the user. The task processing module undertakes the business processing function. It also supports call requests initiated by the automatic task scheduling module and the manual trigger module. In the present invention, the implementation logic is as Figure 2 shown. Query parameters are obtained according to the scheduling mode, and then a request message is constructed and a query request is initiated to the core transaction system through the EAIP. The query message is parsed and persisted to the database, and at the same time, the query result is immediately fed back to the user. Query exceptions or parsing exceptions may occur during the processing, and all exceptions are processed by the exception handler. Among them, the scheduling mode includes an automatic mode and a manual mode.

[0050] The concurrent state controller of the automatic task scheduling module of the present invention will be described in more detail below.

[0051] In a preferred example, the concurrent state controller is a concurrent state controller based on a dual-detection mechanism. In the automatic task scheduling module, a dual-detection mechanism is adopted for the concurrent state controller. The automatic task scheduling logic is as Figure 3 shown. In addition to being automatically controlled by a program, the concurrent state controller can also be manually modified by the user on the page, which helps to deactivate the automatic task scheduling module by adjusting the concurrent state controller in special scenarios.

[0052] In a preferred example, the automatic task scheduling module is a highly available automatic task scheduling module. In the automatic task scheduling module, there is a copy on each node. The deployment mode is as Figure 4 shown. When some nodes fail, the automatic task scheduling module can still provide scheduling services normally. All task pollers share a state controller to ensure that the task processing module runs on only one node at a certain moment. Thus, the effective use of computing resources and data consistency are guaranteed.

[0053] In more preferred examples, the present invention uses an automatic concurrent task scheduling method based on a dual-detection mechanism, supplemented by a manually triggered scheduling method. This ensures that while the scheduling system automatically processes tasks, it can also access the task processing module immediately.

[0054] Those skilled in the art know that in addition to implementing the systems, devices, and their respective modules provided by the present invention in the form of pure computer-readable program codes, the method steps can be logically programmed to enable the systems, devices, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc., to achieve the same program. Therefore, the systems, devices, and their respective modules provided by the present invention can be considered as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing methods or structures within the hardware component.

[0055] The present invention also provides a highly available distributed concurrent task scheduling method. Those skilled in the art can implement the highly available distributed concurrent task scheduling system by executing the process steps of the highly available distributed concurrent task scheduling method. That is, the highly available distributed concurrent task scheduling method can be understood as a preferred implementation manner of the highly available distributed concurrent task scheduling system. Specifically, the highly available distributed concurrent task scheduling method includes:

[0056] Automatic task scheduling steps: According to the task scheduling policy, call the task processing steps by making a request to process the task;

[0057] Task processing steps: According to the said call request, perform task processing.

[0058] The said automatic task scheduling steps include:

[0059] Task polling steps: Conduct task polling; among them, the said task polling includes executing the task scheduling policy. In the task scheduling policy, regularly check the concurrent state controller to see if there is data to be processed; if there is data to be processed and no node currently acquires the concurrent state controller, initiate an action to acquire the concurrent state controller; if the acquisition fails, it is considered that other nodes in the cluster have acquired it in advance, and the current node's current polling ends; if the acquisition is successful, call the task processing steps of the current node to process the task, and return the concurrent state controller after processing; among them, after obtaining the concurrent state controller, verify the acquisition result;

[0060] Concurrent state controller: Adopt a double-check mechanism;

[0061] Among them, after the applications of each node are in an active state, the said task polling starts to work; if the concurrent state controller is in a ready state, acquire the concurrent state controller and conduct a secondary check on the acquisition status; if the acquisition is successful, continue the processing, otherwise, end the current processing.

[0062] In the said automatic task scheduling steps, there is a copy on each node; when some nodes fail, the automatic task scheduling steps can normally provide scheduling services through the copies of the nodes that have not failed; all the said task pollings share one state controller, so that the task processing steps can only run on one node at a time.

[0063] In the said highly available distributed concurrent task scheduling method, it is deployed using a homogeneous distributed cluster, and there are the same copies on each node; regularly initiate a service call to the core trading system through the enterprise application integration platform EAIP; in the said highly available distributed concurrent task scheduling method, obtain query parameters according to the scheduling mode, then construct a request message according to the said query parameters and initiate a query request to the core trading system through the enterprise application integration platform EAIP, obtain, parse the query message from the core trading system and persist it to the database, and at the same time, real-time feedback the query result corresponding to the query message to the user;

[0064] The said highly available distributed concurrent task scheduling method further includes:

[0065] Manual triggering step: Provide the user with a service for real-time access to the task processing steps of triggering, and the manual triggering step triggers the task processing steps to perform logical processing and feedback the processing results to the user; in the manual triggering step, the concurrent state controller is adjusted to deactivate the automatic task scheduling step according to the user's manual modification on the page.

[0066] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the highly available distributed concurrent task scheduling method are implemented.

[0067] According to an electronic device provided by the present invention, including a memory, a processor, and a computer program stored on the memory and executable on the processor, when the computer program is executed by the processor, the steps of the highly available distributed concurrent task scheduling method are implemented.

[0068] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other.

Claims

1. A highly available distributed concurrent task scheduling system, characterized in that, Including: An automatic task scheduling module: According to the task scheduling policy, it calls the task processing module to process tasks by invoking requests. A task processing module: Processes tasks according to the said invocation request. The automatic task scheduling module includes: A task poller: Executes the task scheduling policy. In the task scheduling policy, it regularly checks the concurrent state controller to see if there is data to be processed. If there is data to be processed and no node currently acquires the concurrent state controller, it initiates an action to acquire the concurrent state controller. If the acquisition fails, it is considered that other nodes in the cluster have acquired it earlier, and the current node's current poll ends. If the acquisition is successful, it calls the task processing module of the current node to process tasks, and returns the concurrent state controller after processing. Among them, after obtaining the concurrent state controller, it verifies the acquisition result. A concurrent state controller: Adopts a double-check mechanism. Among them, after the applications of each node become active, the task poller starts to work. If the concurrent state controller is in the ready state, it acquires the concurrent state controller and conducts a secondary check on the acquisition status. If the acquisition is successful, it continues to process; otherwise, it ends the current processing. The concurrent state controller has several states: ready state, running state, and abnormal state. The acquisition of the concurrent state controller is an update action that updates a piece of data in the ready state to the running state. The acquisition status refers to whether the ready state is successfully updated to the running state. The highly available distributed concurrent task scheduling system is deployed using a homogeneous distributed cluster and has the same replicas on each node. It regularly initiates service calls to the core transaction system through the enterprise application integration platform EAIP. The highly available distributed concurrent task scheduling system obtains query parameters according to the scheduling mode, then constructs a request message according to the query parameters and initiates a query request to the core transaction system through the enterprise application integration platform EAIP, obtains, parses the query message from the core transaction system and persists it to the database, and at the same time feeds back the query result corresponding to the query message to the user in real time.

2. The highly available distributed concurrent task scheduling system according to claim 1, wherein The automatic task scheduling module has a replica on each node. When some nodes fail, the automatic task scheduling module provides scheduling services normally through the replicas of the non-failed nodes. All task pollers share a state controller, so that the task processing module can only run on one node at a time. Among them, the concurrent state controller restricts the execution frequency of the task processing module on the node, so that a piece of data will only be effectively processed by one node and only once. If an exception occurs during the query, the task processing module acquires the same piece of data again for processing.

3. The highly available distributed concurrent task scheduling system according to claim 1, wherein The highly available distributed concurrent task scheduling system further includes: Manual Trigger Module: Provides users with real-time access to the task processing module. The manual trigger module triggers the task processing module to perform logical processing and feeds back the processing results to the users. The manual trigger module adjusts the concurrent state controller to deactivate the automatic task scheduling module based on the manual modifications made by the users on the page.

4. A highly available distributed concurrent task scheduling method, characterized in that, It includes: Automatic Task Scheduling Step: According to the task scheduling policy, calls the task processing step to perform task processing by invoking a request. Task Processing Step: Performs task processing according to the said invocation request. The automatic task scheduling step includes: Task Polling Step: Conducts task polling. Among them, the task polling includes executing the task scheduling policy. In the task scheduling policy, it regularly checks the concurrent state controller to see if there is any data to be processed. If there is data to be processed and no node is currently obtaining the concurrent state controller, it initiates an action to obtain the concurrent state controller. If the acquisition fails, it is considered that it has been obtained by other nodes in the cluster in advance, and the current node's current polling ends. If the acquisition is successful, it calls the task processing step of the current node to perform task processing, and returns the concurrent state controller after processing. Among them, after obtaining the concurrent state controller, it verifies the acquisition result. Concurrent State Controller: Adopts a double-check mechanism. Among them, after the applications of each node become active, the task polling starts to work. If the concurrent state controller is in the ready state, it obtains the concurrent state controller and conducts a secondary check on the acquisition status. If the acquisition is successful, it continues to process; otherwise, it ends the current processing. The concurrent state controller has several states: ready state, running state, and abnormal state. Obtaining the concurrent state controller is an update action that updates a piece of data in the ready state to the running state. The acquisition status refers to whether the ready state has been successfully updated to the running state. In the high-availability distributed concurrent task scheduling method, it is deployed using a homogeneous distributed cluster, with the same replicas on each node. It regularly initiates a service call to the core transaction system through the Enterprise Application Integration Platform (EAIP). In the high-availability distributed concurrent task scheduling method, query parameters are obtained according to the scheduling mode, and then a request message is constructed based on the query parameters and a query request is sent to the core transaction system through the EAIP. It obtains, parses the query message from the core transaction system and persists it to the database, and at the same time feeds back the query results corresponding to the query message to the users in real time.

5. The high-availability distributed concurrent task scheduling method according to claim 4, wherein In the automatic task scheduling step, there is a replica on each node. When some nodes fail, the automatic task scheduling step provides scheduling services normally through the replicas of the nodes that have not failed. All the task pollings share a state controller, so that the task processing step can only run on one node at a time. Among them, the concurrent state controller restricts the execution frequency of the task processing step on the node, so that a piece of data will only be effectively processed by one node and only once. If an exception occurs during the query, the task processing step obtains the same piece of data again for processing.

6. The highly available distributed concurrent task scheduling method according to claim 4, wherein the highly available distributed concurrent task scheduling method further includes: A manual trigger step: providing a service for the user to access and trigger the task processing step in real time, and the manual trigger step triggers the task processing step to perform logical processing and feeds back the processing result to the user; in the manual trigger step, the concurrent state controller is adjusted to deactivate the automatic task scheduling step according to the manual modification made by the user on the page.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the highly available distributed concurrent task scheduling method according to any one of claims 4 to 6.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the computer program is executed by a processor, it implements the steps of the highly available distributed concurrent task scheduling method according to any one of claims 4 to 6.

Images