Distributed timing task construction method and system for multi-environment triggering

Abstract

The application belongs to the technical field of computers, and particularly relates to a distributed timing task construction method and system for multi-environment triggering. The method comprises the following steps: S1, when machine resources are sufficient, a business system registers its IP address information on a distributed task scheduling platform xxljob; the xxljob requests the registered business system IP according to the triggering rule of the timing task and through the http protocol, and completes the triggering of the timing task within the cluster; S2, when machine resources are insufficient, a timing task framework Quartz is used for deployment, the business system reads the triggering rule of the timing task set in a service configuration center Nacos, and loads the rule to the local; the local triggers the timing task scheduling periodically; when the timing task enters the scheduling stage, the business system drives to access a memory database Redis and performs a locking operation; after the locking succeeds, a node with the successful locking schedules a timing task executor to execute the timing task.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a method and system for constructing distributed timed tasks triggered by multiple environments. Background Technology

[0002] With the development of existing internet technologies, various business scenarios have emerged. More and more business scenarios require the scheduled processing of certain tasks. In a single-machine environment, scheduled tasks can be implemented quickly and conveniently. However, as business grows and system complexity increases, single-machine systems often expand into distributed systems. At this point, scheduled tasks need to be guaranteed to be triggered on time and only once within the system cluster to prevent multiple nodes from triggering the same task simultaneously, causing systemic risks. Therefore, distributed scheduled task triggering systems have emerged. This triggering system ensures the orderliness (sequential execution of scheduled tasks, preventing a single task from being executed multiple times simultaneously) and effectiveness (ensuring the task is delivered correctly at the scheduled trigger time) when multiple nodes trigger scheduled tasks.

[0003] Traditional distributed scheduled task triggering typically relies on a middleware system or service to distribute tasks to the service, mitigating the concurrency risk of multiple nodes simultaneously triggering the same scheduled task in a distributed system. However, in the early stages of business growth, due to server cost considerations, no additional machine hardware resources may be added. In this case, the system needs to solve the concurrency problem caused by multiple nodes simultaneously triggering the same task in a distributed system without relying on middleware services.

[0004] Currently, existing distributed scheduled task technologies have the following drawbacks: 1. Additional resource consumption occurs during deployment: Currently, the industry generally uses middleware systems (such as the distributed task scheduling platform xxxljob) to implement scheduled task triggering in distributed systems. The advantage is that middleware systems solve the problem of multiple nodes in a cluster concurrently and synchronously calling scheduled tasks, and provide mature functions such as dynamically adjusting task triggering frequency. However, deploying middleware systems incurs additional resource consumption, and if the middleware system itself needs to maintain high availability, it also requires multi-node deployment, which introduces additional concurrency control issues for the scheduling platform itself.

[0005] 2. Quartz cluster deployment suffers from poor concurrency handling, with issues such as locks being held for extended periods. After removing the middleware system responsible for scheduled task execution, Quartz is commonly used in the industry as the trigger for scheduled tasks. Quartz is a feature-rich task scheduling library that can be integrated into any Java application to solve scheduled task scheduling and execution problems in Java applications. However, native Quartz is not very user-friendly in distributed systems. In distributed concurrency scenarios, Quartz uses MySQL row locks for concurrency control by default. When multiple nodes in a cluster execute the same task, the Quartz scheduler attempts to acquire the MySQL row lock. The node that acquires the lock proceeds to task execution, while the node that does not acquire the lock abandons its execution. However, if the node that acquires the lock suddenly dies, the lock will be held indefinitely, preventing other surviving nodes from executing the scheduled task normally.

[0006] 3. Concurrency control during Quartz cluster deployment is cumbersome to implement: Quartz's cluster deployment solution uses MySQL row locks by default, which requires creating a corresponding table in MySQL, making the integration process rather cumbersome.

[0007] Therefore, it is very important to design a distributed timed task construction method and system for multi-environment triggering that can solve the problem of efficiently, stably and orderly triggering timed tasks in distributed systems under resource-constrained scenarios. Summary of the Invention

[0008] This invention aims to overcome the problems of existing distributed scheduled task technologies, such as additional resource consumption during deployment, unfriendly handling of concurrent scenarios when deploying Quartz clusters leading to long-term lock holding, and cumbersome access to concurrent control when deploying Quartz clusters. It provides a method and system for constructing distributed scheduled tasks for multi-environment triggering, which can solve the problem of efficiently, stably, and orderly triggering of scheduled tasks in distributed systems under resource-constrained scenarios.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A method for constructing distributed scheduled tasks triggered by multiple environments includes the following steps; S1, When the machine hardware performance resources are sufficient, the business system registers its own IP address information to the distributed task scheduling platform xxljob via the HTTP protocol; the distributed task scheduling platform xxljob is configured with the triggering rules of the scheduled tasks, which are used to trigger the scheduled task scheduling periodically; the distributed task scheduling platform xxljob requests the registered business system IP according to the triggering rules of the scheduled tasks and via the HTTP protocol to complete a scheduled task trigger within the cluster. S2 utilizes the Quartz timed task framework for deployment when machine hardware performance resources are limited. The business system reads the triggering rules of the timed tasks set in the Nacos service configuration center via HTTP protocol and loads them locally. The local system then triggers the timed task scheduler periodically according to the configured triggering rules. When the timed task enters the scheduling phase, the business system accesses the in-memory database Redis through the JDBC interface driver and performs a locking operation on the in-memory database Redis. After successful locking, the node that has successfully locked the data schedules the timed task executor to execute the timed task.

[0010] Preferably, step S2 further includes the following steps: If locking fails, the locking operation will be retried. After the retry time reaches the preset time, the current scheduled task will be abandoned and the scheduled task process will be exited directly.

[0011] Preferably, in step S2, during the timed task scheduling process, when the timed task executor successfully completes the scheduling or the scheduling fails and is interrupted, the lock is released.

[0012] Preferably, in step S2, the lock added during the locking operation is a Redis lock; the Redis lock is added using the setnx command.

[0013] Preferably, the key set by the setnx command has an expiration date, and the Redis lock is automatically released after the expiration date.

[0014] Preferably, the acquisition and release of the Redis lock are as follows: S21, set business service 1 and business service 2 to trigger scheduled tasks simultaneously and execute Lua scripts to attempt to acquire Redis locks; S22 executes a Lua script to acquire a Redis lock from the in-memory database Redis and set the key expiration time. S23, If business service 1 inserts successfully, business service 1 acquires the Redis lock, while business service 2 fails to acquire the Redis lock; business service 2 attempts to reacquire the Redis lock before the set timeout period expires, and abandons the execution of this scheduled task after the timeout period expires. S24, if business service 1 completes all tasks normally, the data in the in-memory database Redis is deleted, and the Redis lock is released; if business service 1 encounters an abnormal situation, the Redis lock is automatically released after the key expires, so that subsequent business service 2 can insert normally and acquire the Redis lock; the abnormal situation includes network outage, system crash and restart.

[0015] Preferably, the timed task consists of a timed task executor and a scheduler; the execution frequency of the timed task is controlled by adjusting the trigger interval and frequency of the scheduler.

[0016] This invention also provides a distributed timed task construction system for multi-environment triggering, comprising: The xxljob deployment module is used to register the IP address information of the business system to the distributed task scheduling platform xxljob via HTTP protocol when the machine hardware performance resources are sufficient. The distributed task scheduling platform xxljob is configured with triggering rules for scheduled tasks, which are used to periodically trigger scheduled task scheduling. The distributed task scheduling platform xxljob requests the registered business system IP according to the triggering rules of the scheduled tasks and via HTTP protocol to complete a scheduled task trigger within the cluster. The Quartz deployment module is used to deploy the Quartz scheduled task framework when machine hardware performance resources are insufficient. The business system reads the scheduled task triggering rules set in the service configuration center Nacos via HTTP protocol and loads them locally. The local system then triggers the scheduled task periodically according to the configured triggering rules. When the scheduled task enters the scheduling phase, the business system accesses the in-memory database Redis through the JDBC driver interface and performs a locking operation on the in-memory database Redis. After successful locking, the node that has successfully locked the database schedules the scheduled task executor to execute the scheduled task.

[0017] Compared with the prior art, the beneficial effects of this invention are: (1) This invention can solve the problem of distributed timed task triggering under extremely limited resources; this invention can refine the task definition method, provide the function of configurable adjustment of task scheduling frequency, reduce the work of program developers, so as to achieve cost reduction and efficiency improvement in program development, and controllable cost in the process of customer business going to the cloud; (2) This invention combines the advantages of xxljob, Quartz, Redis and Nacos, fully integrates the editability of Quartz timed tasks, the real-time and visibility of Nacos configuration, and the high performance of Redis in-memory database, and realizes a set of functions for visually modifying the scheduling frequency of timed tasks on a page, dynamically adjusting the scheduling rules of timed tasks, and orderly, efficient and stable control of the concurrent timing of timed tasks in the cluster; (3) This invention has extremely high stability, so that program developers do not need to pay attention to the scheduling, concurrency and stability of timed tasks, but can focus on business development. The deployment mode can be switched through simple configuration, which truly reduces the cost of program development and improves production efficiency. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an overall framework for a distributed timed task construction method for multi-environment triggering in this invention; Figure 2 This is a flowchart illustrating one aspect of the Redis distributed lock implementation mechanism in this invention. Figure 3 This is a flowchart illustrating the application of the distributed timed task construction method for multi-environment triggering provided in the embodiments of the present invention in a toolkit encapsulation. Figure 4 for Figure 3 This is a flowchart illustrating the process from service startup to triggering a scheduled task under the XXLJob deployment. Detailed Implementation

[0019] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0020] This invention proposes a method for constructing distributed scheduled tasks for multi-environment triggering. By integrating necessary components such as Nacos, Quartz, and Redis, it achieves distributed task scheduling similar to XXLJob, resulting in orderly and effective scheduled task triggering, flexible node expansion, and simple configuration for integration. Furthermore, it is interoperable with XXLJob's integration code, allowing for a single codebase that can be configured and deployed repeatedly, enabling a multi-environment adaptable distributed scheduled task triggering system that can be used multiple times with a single codebase. The overall technical architecture of this invention is as follows. Figure 1 As shown, the specific process is as follows: 1. After the service starts successfully, read the Nacos configuration for triggering scheduled tasks; 2. Read the scheduled tasks marked in the code and place them into the scheduled task pool; 3. Service startup complete, awaiting subsequent scheduled task triggering; 4. The scheduled task triggering process includes the following two specific scenarios: (1) When the machine hardware performance resources are sufficient, the business system registers its own IP address information to the distributed task scheduling platform xxljob through the HTTP protocol; the distributed task scheduling platform xxljob is configured with the triggering rules of the scheduled task, which is used to trigger the scheduled task scheduling periodically; the distributed task scheduling platform xxljob requests the registered business system IP according to the triggering rules of the scheduled task and through the HTTP protocol to complete the triggering of a scheduled task within the cluster and execute the scheduled task; the above overall process is a third-party scheduling mode.

[0021] (2) When the machine hardware performance resources are insufficient, the Quartz timed task framework is used for deployment. The business system reads the trigger rules of the timed task set in the service configuration center Nacos through the HTTP protocol and loads them locally. The local system triggers the timed task scheduling periodically according to the configured timed task trigger rules. When the timed task enters the scheduling stage, the business system accesses the in-memory database Redis through the interface JDBC driver and performs a locking operation on the in-memory database Redis. After the lock is successfully acquired, the node that has successfully acquired the lock schedules the timed task executor to execute the timed task. If the lock fails, the locking operation is retried. After the retry time reaches the preset time, the current timed task scheduling is abandoned and the timed task scheduling process is directly skipped. During the business scheduling process, the Redis lock will be renewed periodically until the timed task executor finishes execution (successfully or failed and interrupted), and then the Redis lock will be released. The above overall process is the node self-scheduling mode.

[0022] Quartz is a feature-rich task scheduling library that can be integrated into any Java application to solve the problem of scheduling and executing scheduled tasks in Java applications. It provides a quick way to implement scheduled tasks, enabling scheduled task triggering and execution in both single-machine and cluster environments. It is efficient, secure, and convenient; developers only need to call a few interfaces and make simple configurations to achieve the above functions.

[0023] Figure 1 In Chinese, Nacos stands for Dynamic Naming and Configuration Service, a dynamic service discovery, configuration management, and service management platform that makes it easier to build cloud-native applications. The Nacos service configuration center stores runtime configuration parameters for services, including Cron schedules (the regular intervals between scheduled task triggers), lock holding times, and lock wait times. Its primary function is to facilitate adjustments to the timing rules and lock durations of scheduled tasks.

[0024] The service's scheduled task pool is comprised of blocks of functional code. The scheduled task pool itself is the functional block for scheduled tasks within the service. This task pool primarily contains the scheduled task objects from the code. Scheduled tasks consist of an executor and a scheduler; the execution frequency of the scheduled tasks is controlled by adjusting the scheduler's trigger interval and frequency.

[0025] xxljob, a distributed task scheduling platform, provides a platform for scheduling scheduled tasks when machine resources are sufficient. It allows for visual modification of scheduled task trigger rules, retention of scheduled task logs, and load balancing rule allocation via a web interface.

[0026] Redis, an in-memory database, is used to improve service throughput and cache frequently queried data. In Quartz's cluster scheduled task mode, it handles the locking functionality triggered by Quartz scheduled tasks. Its Redis lock usage scenarios include... Figure 2 As shown: The implementation mechanism of Redis distributed locks is as follows: 1. Taking inventory services as an example, set up scheduled tasks for both inventory service 1 and inventory service 2 to execute Lua scripts and attempt to acquire Redis locks.

[0027] 2. A Lua script is used to acquire a lock from the in-memory Redis database and set its expiration time. The Lua script contains the following: It executes the `setnx` function to ensure that only one service inserts the lock. After successful insertion, it sets the key's expiration time. Because Redis is single-threaded, `setnx` will only succeed once; however, the Lua script guarantees atomicity. Therefore, if two services execute simultaneously, only one service will succeed.

[0028] 3. If Inventory Service 1 successfully inserts the lock, it acquires it. Inventory Service 2 will then fail to acquire the lock. If the timeout period has not yet expired, it will attempt to reacquire the lock; if the timeout expires, it will abandon the scheduled task.

[0029] 4. Inventory Service 1: At this point, either step 5 or step 6 may occur.

[0030] 5. Inventory Service 1 successfully completed all tasks and deleted the data records in the in-memory database Redis, releasing the lock.

[0031] 6. If Inventory Service 1 experiences an anomaly, such as network outage, system crash, or restart, the Redis lock will be automatically released after the key's expiration time. Inventory Service 2 can then insert data normally and acquire the lock.

[0032] This invention also provides a distributed timed task construction system for multi-environment triggering, comprising: The xxljob deployment module is used to register the IP address information of the business system to the distributed task scheduling platform xxljob via HTTP protocol when the machine hardware performance resources are sufficient. The distributed task scheduling platform xxljob is configured with triggering rules for scheduled tasks, which are used to periodically trigger scheduled task scheduling. The distributed task scheduling platform xxljob requests the registered business system IP according to the triggering rules of the scheduled tasks and via HTTP protocol to complete a scheduled task trigger within the cluster. The Quartz deployment module is used to deploy the Quartz scheduled task framework when machine hardware performance resources are insufficient. The business system reads the scheduled task triggering rules set in the service configuration center Nacos via HTTP protocol and loads them locally. The local system then triggers the scheduled task periodically according to the configured triggering rules. When the scheduled task enters the scheduling phase, the business system accesses the in-memory database Redis through the JDBC driver interface and performs a locking operation on the in-memory database Redis. After successful locking, the node that has successfully locked the database schedules the scheduled task executor to execute the scheduled task.

[0033] The following is passed Figure 3 and Figure 4 This invention illustrates the application of its method for constructing distributed scheduled tasks triggered by multiple environments in practical development. In practical applications, this invention encapsulates a toolkit to shield the implementation differences between xxljob and Quartz, thereby reducing development costs, unifying access standards, and improving stability. The encapsulation approach of the toolkit is as follows: Figure 3 As shown, taking inventory services as an example, the specific process is as follows: 1. After the inventory service starts successfully, scan the classes and methods annotated with @xxljob, then read the configuration for scheduled task triggering in the service configuration center Nacos. If it is in Quartz deployment mode, obtain the scheduled task configuration corresponding to the @xxljob task, and generate the corresponding task scheduler and executor by combining Quartz dynamic proxy. Add Redis lock implementation to the proxy class of the scheduled task scheduling business, and register the scheduler and executor to the container Spring.

[0034] 2. Wait for the inventory service to start successfully, and then trigger the scheduled task.

[0035] 3. If the Nacos service configuration center is configured with scheduled tasks in xxljob deployment mode, the process from the start of the inventory service to the triggering of the scheduled task is as follows: Figure 4 As shown, specifically: 3.1. Inventory Service 1 obtains the registration address information of xxljob from the service configuration center Nacos.

[0036] 3.2. Inventory Service 1 registers its own IP information with the task scheduling platform xxljob.

[0037] 3.3.xxljob periodically triggers the configured inventory service task scheduler, reads the relevant IP list of the inventory service, and sends an HTTP request to inventory service 1 according to the scheduling rules, so as to realize the triggering of a single-node scheduled task within the cluster.

[0038] 3.4. Inventory Service 1 receives the request and executes the scheduled task.

[0039] 4. In the Nacos Service Configuration Center, change the scheduled task to Quartz deployment mode and restart the inventory service.

[0040] 5. When in Quartz deployment mode, the specific process from the start of the inventory service to the triggering of the scheduled task is as follows: 5.1. Inventory Service 1 obtains Quartz task scheduling configurations from the Nacos service configuration center.

[0041] 5.2. The inventory service obtains the scheduled tasks to be executed through the @xxljob annotation in the code.

[0042] 5.3. Based on the task scheduling configuration of Nacos in the service configuration center and the scheduled tasks to be executed, a Quartz scheduler is dynamically proxied and Redis locking code is added to achieve seamless switching of scheduled tasks.

[0043] 5.4. Inventory Service 1 and Inventory Service 2 simultaneously trigger scheduled tasks.

[0044] 5.5. Inventory Service 1 acquires the Redis lock and executes a scheduled task.

[0045] 5.6. Once the scheduled task is completed, the proxy class will automatically release the Redis lock.

[0046] This invention innovatively integrates multiple technologies such as xxljob, Nacos, Redis, and Quartz, and achieves different scheduled task triggering methods through different configuration switching, solving the problem of efficiently, stably, and orderly triggering scheduled tasks in distributed systems under resource-constrained scenarios.

[0047] The innovative aspects of this invention are as follows: 1. This invention innovatively integrates xxljob and Quartz, enabling a single codebase to be deployed in various environments and scenarios, thus achieving scheduled task triggering in distributed systems; 2. This invention combines Redis and Quartz, using Redis's distributed locks to address the shortcomings of native MySQL distributed locks in Quartz cluster mode. It optimizes and resolves the poor performance and deadlock issues of MySQL locks in native Quartz cluster mode, providing a more stable and more available Quartz cluster deployment method.

[0048] 3. This invention integrates Nacos, Quartz, and xxljob. Based on Nacos's dynamic configuration refresh function, it enables visual page configuration adjustment for Quartz scheduled tasks and dynamically optimizes and adjusts Quartz scheduled task triggering rules. By analyzing the principle of Quartz's dynamic registration generation and parsing the relevant code configuration of xxljob, it reduces the workload of product development and improves the simplicity of the product.

[0049] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing distributed timed tasks triggered by multiple environments, characterized in that, Includes the following steps; S1, When the machine hardware performance resources are sufficient, the business system registers its own IP address information to the distributed task scheduling platform xxljob via the HTTP protocol; the distributed task scheduling platform xxljob is configured with the triggering rules of the scheduled tasks, which are used to trigger the scheduled task scheduling periodically; the distributed task scheduling platform xxljob requests the registered business system IP according to the triggering rules of the scheduled tasks and via the HTTP protocol to complete a scheduled task trigger within the cluster. S2 utilizes the Quartz timed task framework for deployment when machine hardware performance resources are limited. The business system reads the triggering rules of the timed tasks set in the Nacos service configuration center via HTTP protocol and loads them locally. The local system then triggers the timed task scheduler periodically according to the configured triggering rules. When the timed task enters the scheduling phase, the business system accesses the in-memory database Redis through the JDBC interface driver and performs a locking operation on the in-memory database Redis. After successful locking, the node that has successfully locked the data schedules the timed task executor to execute the timed task.

2. The method for constructing a distributed timed task triggered by multiple environments according to claim 1, characterized in that, Step S2 also includes the following steps: If locking fails, the locking operation will be retried. After the retry time reaches the preset time, the current scheduled task will be abandoned and the scheduled task process will be exited directly.

3. The method for constructing a distributed timed task triggered by multiple environments according to claim 2 is characterized in that, In step S2, during the timed task scheduling process, if the timed task executor successfully completes the scheduling or fails to complete the scheduling and is interrupted, the lock is released.

4. The method for constructing a distributed timed task triggered by multiple environments according to claim 1, characterized in that, In step S2, the lock acquired during the locking operation is a Redis lock; the Redis lock is added using the setnx command.

5. The method for constructing a distributed timed task triggered by multiple environments according to claim 4, characterized in that, The key set by the setnx command has an expiration date. After the expiration date, the Redis lock is automatically released.

6. The method for constructing a distributed timed task triggered by multiple environments according to claim 5, characterized in that, The acquisition and release of the Redis lock are as follows: S21, set business service 1 and business service 2 to trigger scheduled tasks simultaneously and execute Lua scripts to attempt to acquire Redis locks; S22 executes a Lua script to acquire a Redis lock from the in-memory database Redis and set the key expiration time. S23, If business service 1 inserts successfully, business service 1 acquires the Redis lock, while business service 2 fails to acquire the Redis lock; business service 2 attempts to reacquire the Redis lock before the set timeout period expires, and abandons the execution of this scheduled task after the timeout period expires. S24, if business service 1 completes all tasks normally, the data in the in-memory database Redis is deleted, and the Redis lock is released; if business service 1 encounters an abnormal situation, the Redis lock is automatically released after the key expires, so that subsequent business service 2 can insert normally and acquire the Redis lock; the abnormal situation includes network outage, system crash and restart.

7. The method for constructing a distributed timed task triggered by multiple environments according to claim 3, characterized in that, The scheduled task consists of a scheduled task executor and a scheduler; the execution frequency of the scheduled task is controlled by adjusting the trigger interval and frequency of the scheduler.

8. A system for constructing distributed timed tasks triggered by multiple environments, used to implement the method for constructing distributed timed tasks triggered by multiple environments as described in any one of claims 1-7, characterized in that, The distributed timed task construction system for multi-environment triggering includes: The xxljob deployment module is used to register the IP address information of the business system to the distributed task scheduling platform xxljob via HTTP protocol when the machine hardware performance resources are sufficient. The distributed task scheduling platform xxljob is configured with triggering rules for scheduled tasks, which are used to periodically trigger scheduled task scheduling. The distributed task scheduling platform xxljob requests the registered business system IP according to the triggering rules of the scheduled tasks and via HTTP protocol to complete a scheduled task trigger within the cluster. The Quartz deployment module is used to deploy the Quartz scheduled task framework when machine hardware performance resources are insufficient. The business system reads the scheduled task triggering rules set in the service configuration center Nacos via HTTP protocol and loads them locally. The local system then triggers the scheduled task periodically according to the configured triggering rules. When the scheduled task enters the scheduling phase, the business system accesses the in-memory database Redis through the JDBC driver interface and performs a locking operation on the in-memory database Redis. After successful locking, the node that has successfully locked the database schedules the scheduled task executor to execute the scheduled task.

Images