Node dynamic backoff method, distributed task scheduling method and system

Abstract

The application belongs to the technical field of task scheduling. A node dynamic backoff method, a distributed task scheduling method and system are provided. When the triggering time of the current timing task is reached, it is judged whether the triggering of the previous timing task is completed by the node, and then it is decided whether to back off the node, solving the problem of low efficiency of distributed task scheduling in the prior art. Compared with the database lock, the competition for the distributed lock is reduced, and better performance is achieved in the Redis high-concurrency scenario. The lua script corresponding to the timing task is maintained in Redis, supporting the sharding execution of the timing task, solving the problem of task sharding not supported by the existing Quartz framework, and expanding the applicable scenarios of the distributed task scheduling strategy.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, specifically to a dynamic node backoff method, a distributed task scheduling method, and a system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, both internet applications and enterprise applications inevitably require scheduled tasks to trigger and execute tasks at regular intervals. As enterprises develop more and more systems, they are gradually evolving from single applications to service clusters. In distributed cluster systems, a series of problems need to be solved through distributed task scheduling, such as the repeated execution of scheduled tasks, the lack of unified scheduled task configuration and scheduling, task node failover, and task execution monitoring.

[0004] Currently, the commonly used distributed task scheduling framework is Quartz. Quartz provides rich functionality to meet various complex scheduling needs, such as time-based scheduling, round-robin scheduling, and calendar-based scheduling. Quartz supports distributed and cluster modes, which can connect multiple schedulers together to share jobs and triggers. This enables Quartz to achieve job load balancing and failover in multi-node environments, improving system availability and stability.

[0005] However, with the increasing number of nodes and task scheduling frequency in distributed systems, the traditional Quartz framework can no longer meet the needs of distributed task scheduling. This is mainly reflected in the following aspects: (1) The Quartz framework relies on database locks to implement distributed task scheduling. Due to the performance limitations of database locks, the scheduling latency will increase significantly in large-scale scheduling scenarios, especially for multi-node, high-frequency, second-level timed tasks; (2) The Quartz framework needs to operate on the database during the scheduling process, and database transactions will also affect the scheduling efficiency; (3) The Quartz framework does not support the sharding of timed tasks when scheduling tasks. For timed tasks that can be sharded, the advantages of distributed systems cannot be fully utilized, and the execution efficiency of tasks is low. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a dynamic node backoff method, a distributed task scheduling method, and a system. By employing a dynamic node backoff strategy, the invention solves the problem of low efficiency in distributed task scheduling in existing technologies and improves concurrency performance.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for dynamic node backoff.

[0009] A dynamic node backoff method for any node in a service cluster includes the following process:

[0010] When the trigger time of the current timed task is reached, it is determined whether the previous trigger of the timed task was completed by the node itself. If so, the current timed task is attempted to be triggered. If not, the triggering is paused and the node waits for a set time, which is the node's backoff time.

[0011] Once the backoff time has elapsed, determine whether the scheduled task has been successfully triggered. If so, do not trigger it again; otherwise, attempt to trigger the scheduled task.

[0012] As a further limitation of the first aspect of the present invention, let t be the trigger time of any timed task that has been triggered and completed. s The node receives the Redis broadcast that the scheduled task has completed execution at time t. e , then t e With t s The difference is the scheduling time for one scheduling of the timed task;

[0013] Calculate the average scheduling time of the most recent m scheduling times before the trigger time of the timed task, and use the average value as the backoff time, where m is a positive integer greater than or equal to 2.

[0014] As a further limitation of the first aspect of the present invention, if the calculated backoff time exceeds the set backoff time upper limit, the node directly triggers the current timed task.

[0015] As a further limitation of the first aspect of the present invention, each node in the service cluster generates a node identifier when it starts up, and the node stores the previous trigger node identifier corresponding to the previous timed task being triggered.

[0016] After receiving the broadcast value of Redis after executing the triggered scheduled task once, the node modifies the value of the previously triggered node identifier to the node identifier carried in the broadcast value.

[0017] As a further limitation of the first aspect of the present invention, the node stores the number of broadcast executions, and after receiving the broadcast value after Redis has executed the triggered timed task once, the node updates the number of broadcast executions to the broadcast value.

[0018] As a further limitation of the first aspect of the present invention, when the node starts up, it actively requests the Redis to obtain the broadcast value in order to update the broadcast execution count.

[0019] As a further limitation of the first aspect of the present invention, the node stores the number of times the trigger is executed, and the number of times the trigger is executed is the number of times the timer trigger of the node is triggered;

[0020] Determine whether the number of times the trigger is executed is the same as the number of times the broadcast is executed. If they are the same, it means that the broadcast value from Redis has been received, Redis has completed the triggering of the scheduled task, and the triggering of the scheduled task ends.

[0021] If the number of times the trigger is executed is greater than the number of times the broadcast is executed, then the node carries its own trigger execution count and its own node identifier, and calls Redis to execute the Lua script corresponding to the scheduled task.

[0022] Secondly, the present invention provides a node dynamic backoff system.

[0023] A node dynamic backoff system for serving any node in a cluster, comprising:

[0024] The backoff waiting unit is configured to: when the triggering time of the current timed task is reached, determine whether the previous triggering of the timed task was completed by the node itself; if so, attempt to trigger the current timed task; if not, pause the triggering and wait for a set time, where the set time is the backoff time of the node.

[0025] The trigger judgment unit is configured to: when the backoff time is reached, determine whether the current timed task has been triggered. If so, it will not be triggered again; otherwise, it will attempt to trigger the current timed task.

[0026] Thirdly, the present invention provides a distributed task scheduling method.

[0027] A distributed task scheduling method includes the following process:

[0028] The node dynamic backoff method described in the first aspect of the present invention is used to trigger the scheduled task. When the scheduled task is triggered, Redis executes the Lua script corresponding to the scheduled task.

[0029] Determine whether the number of task executions maintained in Redis is less than the number of trigger executions sent by the node. If so, modify the number of task executions maintained in Redis to match the number of trigger executions sent by the node. If not, end the determination process.

[0030] Using Redis's publish-subscribe mechanism, the system broadcasts its task execution count and the node identifier that successfully triggered the scheduled task to all nodes in the service cluster.

[0031] As a further limitation of the third aspect of the present invention, when it is necessary to shard the scheduled task, the task shards are placed into the Redis-stream queue of the scheduled task in the Lua script, and all nodes in the service cluster are the consumer group of the Redis-stream queue.

[0032] Messages are retrieved from the Redis-stream queue for consumption. Once a message has been consumed by one member of the consumer group, the other members of the consumer group will not consume the message again.

[0033] Fourthly, the present invention provides a distributed task scheduling system.

[0034] A distributed task scheduling system includes multiple nodes that communicate with each other, and each node can call the Lua script corresponding to the scheduled task in Redis.

[0035] The Redis maintains the Lua script corresponding to the scheduled task. Each node uses the node dynamic backoff method described in the first aspect of the present invention to trigger the scheduled task. When the scheduled task is triggered, the Redis executes the Lua script.

[0036] Determine whether the number of task executions maintained in Redis is less than the number of trigger executions sent by the node. If so, modify the number of task executions maintained in Redis to match the number of trigger executions sent by the node. If not, end the determination process.

[0037] Using Redis's publish-subscribe mechanism, the system broadcasts its task execution count and the node identifier that successfully triggered the scheduled task to all nodes in the service cluster.

[0038] Fifthly, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;

[0039] A processor, adapted to execute computer programs;

[0040] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the node dynamic backoff method as described in the first aspect of the present invention; or implements the distributed task scheduling method as described in the third aspect of the present invention.

[0041] In a sixth aspect, the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed as the node dynamic backoff method described in the first aspect of the present invention; or, to execute the distributed task scheduling method described in the third aspect of the present invention.

[0042] In a seventh aspect, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the node dynamic backoff method as described in the first aspect of the present invention; or implements the distributed task scheduling method as described in the third aspect of the present invention.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1. This invention innovatively proposes a dynamic node backoff strategy. When the trigger time of the current scheduled task is reached, it is determined whether the previous scheduled task was triggered by a node, and then decides whether to back off the node. This solves the problem of low efficiency in distributed task scheduling in the prior art. Compared with database locks, it reduces lock contention and has better performance in high-concurrency Redis scenarios.

[0045] 2. This invention calculates the average scheduling time of the most recent m scheduling times before the triggering time of the timed task as the backoff time, avoiding the error caused by a single calculation and improving the accuracy of dynamic backoff control of nodes.

[0046] 3. In this invention, if the calculated backoff time exceeds the set backoff time limit, the node directly triggers the current timed task, thus avoiding the node's long-term backoff and ensuring the normal operation of the timed task.

[0047] 4. In this invention, the node stores the identifier of the last triggered node. After receiving the broadcast value after Redis executes a triggered timed task, the node modifies the value of the last triggered node identifier to the node identifier carried in the broadcast value, thereby enabling real-time judgment of the node identifier of the last triggered timed task and ensuring the accuracy of the node's dynamic backoff.

[0048] 5. In this invention, the nodes store the broadcast execution count. After receiving the broadcast value after Redis executes the triggered timed task once, the node updates the broadcast execution count to the broadcast value. Moreover, when each node starts, it actively requests Redis to obtain the broadcast value to update the broadcast execution count, thus ensuring the real-time nature of the broadcast execution count.

[0049] 6. In this invention, the node stores the number of times the trigger is executed. The number of times the trigger is executed is the number of times the timed trigger is triggered. It is determined whether the number of times the trigger is executed is the same as the number of times the broadcast is executed. Based on the determination result, the current trigger is terminated or the corresponding Lua script for the timed task is called to execute Redis, which improves the timeliness of the node's dynamic backoff.

[0050] 7. This invention innovatively proposes a distributed task scheduling method. Redis maintains the Lua scripts corresponding to the scheduled tasks, supports the sharded execution of scheduled tasks, solves the problem of task sharding not being supported by the existing Quartz framework, and expands the applicable scenarios of the distributed task scheduling strategy.

[0051] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0053] Figure 1 This is a flowchart illustrating the node dynamic backoff method provided in Embodiment 1 of the present invention;

[0054] Figure 2 This is a schematic diagram of the task partitioning strategy provided in Embodiment 1 of the present invention;

[0055] Figure 3 This is a schematic diagram of a node dynamic backoff system provided in Embodiment 2 of the present invention;

[0056] Figure 4 This is a flowchart illustrating the distributed task scheduling method provided in Embodiment 3 of the present invention;

[0057] Figure 5 This is a schematic diagram of a computer device provided in Embodiment 5 of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0060] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0061] Example 1:

[0062] This implementation proposes a dynamic node backoff method. Scheduled tasks are executed via Lua scripts in Redis, and each scheduled task undergoes multiple scheduling and executions. The following is a brief introduction to the technical terminology and related concepts involved in this solution:

[0063] Redis (Remote Dictionary Server) is an open-source, ANSI C-written, network-enabled, in-memory or persistent, log-structured, key-value database that provides APIs for multiple languages. Redis stores all data in memory, making read and write operations much faster; memory access speeds are significantly higher than disk access speeds. Redis supports various data structures, such as strings, hashes, lists, sets, and sorted sets. Redis supports master-slave replication and cluster deployment, enabling high availability and scalability. Master-slave replication copies data from one Redis instance to multiple slave instances; when the master instance fails, slave instances can take over the data. Cluster deployment combines multiple Redis instances into a single cluster, achieving distributed data storage and access.

[0064] Lua scripts in Redis guarantee the atomicity of a set of operations, which is crucial when executing multiple operations, especially when these operations are interdependent. For example, a Lua script can be used to implement an atomic counter or currency transaction. Lua scripts can also be used to implement transactional execution of a set of commands; that is, a group of commands can be packaged together and submitted to Redis as a whole. If any error occurs during this process, all commands will not be executed. For complex operations, using Lua scripts can reduce network latency and command parsing time, thereby improving performance. Lua scripts can be used to implement complex algorithms, such as sorting and hash calculations. Data transformation or processing can also be performed in Lua scripts, such as converting the value of a key from one format to another.

[0065] The node dynamic backoff method in this implementation is specifically used for any node in the service cluster and includes the following process:

[0066] When the trigger time of the current timed task is reached, it is determined whether the previous trigger of the timed task was completed by the node itself. If so, the current timed task is attempted to be triggered. If not, the triggering is paused and the node waits for a set time, which is the node's backoff time.

[0067] Once the backoff time has elapsed, determine whether the scheduled task has been successfully triggered. If so, do not trigger it again; otherwise, attempt to trigger the scheduled task.

[0068] In this implementation, a node refers to a replica of a service cluster deployed in a distributed system, and a scheduled task is a task that needs to be executed at a specified time.

[0069] In this implementation, each node in the cluster generates a unique identifier, called the node identifier, when it starts up.

[0070] In this implementation, each node in the service cluster maintains the following variables for the scheduled task:

[0071] Variable 1: Broadcast execution count; This variable represents the number of times the task was executed after Redis received the broadcast after executing the Lua script. Each node updates its own broadcast execution count to the broadcast value after receiving the broadcast. Each node will actively request Redis to obtain the broadcast execution count when it starts up.

[0072] Variable 2: Trigger execution count; the trigger execution count is the number of times the timed triggers of the tasks in this node are triggered.

[0073] Variable 3: Last trigger node identifier; each time a broadcast is received from Redis after executing a Lua script, the value of the last trigger node identifier is modified to the node identifier carried in the broadcast.

[0074] Variable 4: Backoff time; The backoff time is to prevent the waste of network resources caused by all nodes in the cluster calling Redis simultaneously when the task is triggered. This is especially important for frequently triggered tasks. The backoff time is when a node triggers a task, it will check whether it was the one that successfully triggered the task last time. If it was, it will trigger the task immediately. If it was not, it will wait for a period of time. If the scheduled task has not been completed, it will request Redis. This waiting period is called the backoff time.

[0075] This implementation innovatively proposes a method for calculating the backoff time, including:

[0076] For any node, the trigger time of the node's scheduled task (a scheduled task that has been triggered and completed) is denoted as ts, and the time when the node receives the Redis broadcast Lua script to complete execution is te. Then, the value of te-ts is the scheduling time of one scheduled task. If this is the i-th scheduling of this scheduled task, then the scheduling time is t(i).

[0077] The backoff time is the average of the scheduling times of the task in the most recent m times (i.e., the m times before the trigger time of the current scheduled task). If the current task has been executed n times, then the backoff time = (t(n) + t(n-1) + t(n-2) + ... + t(nm)) / m, where t(n) represents the scheduling time of the nth scheduling, t(n-1) represents the scheduling time of the (n-1)th scheduling, t(n-2) represents the scheduling time of the (n-2)th scheduling, and t(nm) represents the scheduling time of the nmth scheduling.

[0078] Variable 5: Maximum backoff time; Each scheduled task is configured with its own maximum backoff time to prevent the backoff time from exceeding the business tolerance limit due to cluster clock errors and network latency.

[0079] In this implementation, each node in the service cluster maintains its own timed trigger for the task, and the node's dynamic backoff strategy is triggered when the scheduled task is about to be executed.

[0080] More detailed steps of the node dynamic backoff method, such as Figure 1 As shown, it includes:

[0081] Step (1): Determine whether the node identifier that triggered the timed task last time is this node. If not, proceed to step (2). If yes, proceed to step (3).

[0082] Step (2): Determine whether the retreat time is greater than the maximum retreat time. If it is not greater, temporarily suspend the retreat time and then execute step (3). If it is greater, execute step (3) directly.

[0083] Step (3): Increment the trigger execution count by 1;

[0084] Step (4): Determine whether the number of times the trigger is executed is the same as the number of times the broadcast is executed. If they are the same, it means that the broadcast from Redis has been received and Redis has finished executing the Lua script corresponding to the scheduled task. Then, end the current triggering. If the number of times the trigger is executed is greater than the number of times the broadcast is executed, continue to step (5).

[0085] Step (5): The node carries its own trigger execution count and its own node identifier to call the Lua script corresponding to the Redis scheduled task.

[0086] Example 2:

[0087] like Figure 2 As shown, this implementation provides a dynamic node backoff system for any node in a service cluster, including:

[0088] The backoff waiting unit is configured to: when the triggering time of the current timed task is reached, determine whether the previous triggering of the timed task was completed by the node itself; if so, attempt to trigger the current timed task; if not, pause the triggering and wait for a set time, where the set time is the backoff time of the node.

[0089] The trigger judgment unit is configured to: when the backoff time is reached, determine whether the current timed task has been triggered. If so, it will not be triggered again; otherwise, it will attempt to trigger the current timed task.

[0090] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0091] Understandably, the various units of the above-mentioned node dynamic backoff system can be embedded as functional modules in the node's processor; of course, they can also be embedded in a processor configured separately for the node, which will not be elaborated here.

[0092] Example 3:

[0093] This implementation provides a distributed task scheduling method, in which each node executes the node dynamic backoff method described in Example 1, such as... Figure 3 As shown, Redis maintains the corresponding Lua script for the scheduled task and keeps track of the number of times the task is executed. The execution steps of the script are as follows:

[0094] Check if the number of times the task maintained in Redis has been executed is less than the number of times the trigger has been executed when the node calls the Lua script. If it is not less, the process ends; if it is less, continue to the next step.

[0095] The number of task executions maintained in Redis is modified to match the number of trigger executions sent by the node. If the scheduled task needs to be sharded, it will be sharded in this step (the sharding logic is determined by the specific business scenario of the task; for example, when analyzing device operation, the task may be sharded according to the device type), and then placed into the message queue of the scheduled task in the Redis stream data type; if it does not need to be sharded, the scheduled task will be placed directly into the stream message queue.

[0096] Using Redis's publish-subscribe mechanism, the system broadcasts its own task execution count and the node identifier that successfully triggered the scheduled task to the cluster. Each node subscribes to changes in the task execution count published by Redis through the publish-subscribe mechanism and synchronizes its own broadcast execution count to be consistent with that published by Redis.

[0097] This implementation leverages Redis's atomicity guarantee for Lua script execution, avoiding potential concurrency issues when multiple nodes simultaneously trigger Lua script calls. By placing sharded tasks into a task execution queue, which is then consumed by all nodes in the cluster, the scheduled tasks are executed in shards within the cluster.

[0098] In this implementation, a stream message queue and its consumer group are maintained in Redis for each scheduled task (once a message is consumed by a member of the consumer group, the member of the consumer group will not consume the message again, thus ensuring that task shards are not executed repeatedly). All nodes in the cluster are members of the consumer group. When the Lua script corresponding to the task puts a message into the message queue of the task, the nodes in the cluster act as consumers to consume it.

[0099] More specifically, the sharding logic for scheduled tasks is as follows: The sharding logic is implemented in a Redis-Lua script, dividing a scheduled task into shards. For example, a scheduled task to send SMS messages to a batch of users can be divided into different shards based on the city of the mobile phone number. This sharding logic is determined by the business logic of the scheduled task itself. Therefore, different scheduled tasks use different Lua scripts. Except for the initial concurrency control based on the number of task triggers and the final broadcast of the trigger message to the cluster, the sharding logic in the Lua scripts is different. For the sharded scheduled tasks, Redis-stream (an implementation of a message queue provided by Redis) is used for distribution. In the Lua script, the task shards are placed into the Redis-stream queue for that task. All nodes in the cluster act as consumers of this queue, retrieving messages from the queue for consumption.

[0100] like Figure 4 As shown, taking two shards as an example, in the Redis-lua script, task shard 1 and shard 2 are put into the stream message queue corresponding to the task. Node A, as a member of the consumer group, retrieves shard 1 from the stream message queue and executes it, and node B, as a member of the consumer group, retrieves shard 2 from the stream message queue and executes it.

[0101] In summary, this implementation uses Redis-lua scripts and a dynamic node backoff strategy to solve the problem of low efficiency in distributed task scheduling in existing technologies. Redis offers superior performance compared to database locks in high-concurrency scenarios. Combined with the dynamic node backoff strategy, it reduces lock contention and further improves concurrency performance. Moreover, task sharding is performed in the Redis-lua scripts, supporting task sharding functionality not supported by Quartz.

[0102] Example 4:

[0103] This implementation provides a distributed task scheduling system, which includes multiple interconnected nodes. Each node can call the Lua script corresponding to the scheduled task in Redis and uses the distributed task scheduling method described in Example 3 to control the scheduled tasks of each node.

[0104] Example 5:

[0105] like Figure 5 As shown, this implementation provides a computer device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means.

[0106] The communication interface 1002 is used to receive and send data. The computer-readable storage medium 1003 can be stored in the memory of the electronic device. The computer-readable storage medium 1003 is used to store computer programs, which include program instructions. The processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0107] The processor 1001 (or CPU (Central Processing Unit)) is the computing and control core of electronic devices. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve corresponding methods or functions.

[0108] The processor 1001 is configured to perform the following process:

[0109] When the trigger time of the current timed task is reached, it is determined whether the previous trigger of the timed task was completed by the node itself. If so, the current timed task is attempted to be triggered. If not, the triggering is paused and the node waits for a set time, which is the node's backoff time.

[0110] Once the backoff time has elapsed, determine whether the scheduled task has been successfully triggered. If so, do not trigger it again; otherwise, attempt to trigger the scheduled task.

[0111] Alternatively, the processor 1001 is configured to perform the following process:

[0112] Check if the number of times the task maintained in Redis has been executed is less than the number of times the trigger has been executed when the node calls the Lua script. If it is not less, the process ends; if it is less, continue to the next step.

[0113] In Redis, the number of task executions maintained should be updated to match the "trigger execution count" sent by the node.

[0114] If the task requires sharding, it will be sharded in this step (the sharding logic is determined by the specific business scenario of the task; for example, if the device is being analyzed, the task may be sharded according to the device type), and then placed into the task's message queue in the Redis stream data type; if it does not require sharding, the task will be placed directly into the task stream message queue.

[0115] Using Redis's publish-subscribe mechanism, broadcast the number of times your task has been executed and the identifier of the node that successfully triggered this scheduled task to the cluster.

[0116] It is understandable that the aforementioned computer equipment can be used directly as a node, i.e., a computer equipment used as a node; it is also understandable that in some other implementations, the aforementioned computer equipment can also be used as an external control terminal device, working through communication with the node, which will not be elaborated here.

[0117] Example 6:

[0118] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the electronic device and extended storage media supported by the electronic device. The computer-readable storage medium provides storage space that stores the processing system of the electronic device.

[0119] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or unstable memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0120] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to perform the following process:

[0121] When the trigger time of the current timed task is reached, it is determined whether the previous trigger of the timed task was completed by the node itself. If so, the current timed task is attempted to be triggered. If not, the triggering is paused and the node waits for a set time, which is the node's backoff time.

[0122] Once the backoff time has elapsed, determine whether the scheduled task has been successfully triggered. If so, do not trigger it again; otherwise, attempt to trigger the scheduled task.

[0123] Alternatively, the following process can be implemented:

[0124] Check if the number of times the task maintained in Redis has been executed is less than the number of times the trigger has been executed when the node calls the Lua script. If it is not less, the process ends; if it is less, continue to the next step.

[0125] In Redis, the number of task executions maintained should be updated to match the "trigger execution count" sent by the node.

[0126] If the task requires sharding, it will be sharded in this step (the sharding logic is determined by the specific business scenario of the task; for example, if the device is being analyzed, the task may be sharded according to the device type), and then placed into the task's message queue in the Redis stream data type; if it does not require sharding, the task will be placed directly into the task stream message queue.

[0127] Using Redis's publish-subscribe mechanism, broadcast the number of times your task has been executed and the identifier of the node that successfully triggered this scheduled task to the cluster.

[0128] It is understandable that the aforementioned computer-readable storage medium can be directly mounted on an electronic device and used as a node; it is also understandable that in some other implementations, the aforementioned computer-readable storage medium can also be mounted on an external control device, which works by communicating with the node, and will not be elaborated here.

[0129] Example 7:

[0130] This implementation provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the following process:

[0131] When the trigger time of the current timed task is reached, it is determined whether the previous trigger of the timed task was completed by the node itself. If so, the current timed task is attempted to be triggered. If not, the triggering is paused and the node waits for a set time, which is the node's backoff time.

[0132] Once the backoff time has elapsed, determine whether the scheduled task has been successfully triggered. If so, do not trigger it again; otherwise, attempt to trigger the scheduled task.

[0133] Alternatively, perform the following procedure:

[0134] Check if the number of times the task maintained in Redis has been executed is less than the number of times the trigger has been executed when the node calls the Lua script. If it is not less, the process ends; if it is less, continue to the next step.

[0135] In Redis, the number of task executions maintained should be updated to match the "trigger execution count" sent by the node.

[0136] If the task requires sharding, it will be sharded in this step (the sharding logic is determined by the specific business scenario of the task; for example, if the device is being analyzed, the task may be sharded according to the device type), and then placed into the task's message queue in the Redis stream data type; if it does not require sharding, the task will be placed directly into the task stream message queue.

[0137] Using Redis's publish-subscribe mechanism, broadcast the number of times your task has been executed and the identifier of the node that successfully triggered this scheduled task to the cluster.

[0138] Understandably, the aforementioned computer program products or computer programs can run as embedded in the node's processor; of course, they can also run on a processor specifically configured for the node, which will not be elaborated here.

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for dynamic node backoff, characterized in that, For any node in the service cluster, the following procedures are included: When the trigger time of the current timed task is reached, it is determined whether the previous trigger of the timed task was completed by the node itself. If so, the current timed task is attempted to be triggered. If not, the triggering is paused and the node waits for a set time, which is the node's backoff time. Once the backoff time has elapsed, determine whether the scheduled task has been successfully triggered. If so, do not trigger it again; otherwise, attempt to trigger the scheduled task.

2. The node dynamic backoff method as described in claim 1, characterized in that, Let t be the trigger time of any time when the scheduled task has been triggered and completed. s The node receives the Redis broadcast that the scheduled task has completed execution at time t. e , then t e With t s The difference is the scheduling time for one scheduling of the timed task; Calculate the average scheduling time of the most recent m scheduling times before the trigger time of the timed task, and use the average value as the backoff time, where m is a positive integer greater than or equal to 2.

3. The node dynamic backoff method as described in claim 2, characterized in that, If the calculated backoff time exceeds the set backoff time limit, the node will directly trigger the scheduled task.

4. The node dynamic backoff method as described in claim 1, characterized in that, Each node in the service cluster generates a node identifier upon startup, and the node stores the previous trigger node identifier corresponding to the last time the scheduled task was triggered. After receiving the broadcast value of Redis after executing the triggered scheduled task once, the node modifies the value of the previously triggered node identifier to the node identifier carried in the broadcast value.

5. The node dynamic backoff method as described in claim 1, characterized in that, The node stores the number of broadcast executions. After receiving the broadcast value after Redis has executed the triggered scheduled task once, the node updates the broadcast execution count to the broadcast value.

6. The node dynamic backoff method as described in claim 5, characterized in that, When the node starts, it actively requests the Redis to obtain the broadcast value in order to update the broadcast execution count.

7. The node dynamic backoff method as described in claim 5, characterized in that, The node stores the number of times the trigger is executed, which is the number of times the node's timer trigger is triggered. Determine whether the number of times the trigger is executed is the same as the number of times the broadcast is executed. If they are the same, it means that the broadcast value from Redis has been received, Redis has completed the triggering of the scheduled task, and the triggering of the scheduled task ends. If the number of times the trigger is executed is greater than the number of times the broadcast is executed, then the node carries its own trigger execution count and its own node identifier, and calls Redis to execute the Lua script corresponding to the scheduled task.

8. A node dynamic backoff system, characterized in that, For any node in the service cluster, including: The backoff waiting unit is configured to: when the triggering time of the current timed task is reached, determine whether the previous triggering of the timed task was completed by the node itself; if so, attempt to trigger the current timed task; if not, pause the triggering and wait for a set time, where the set time is the backoff time of the node. The trigger judgment unit is configured to: when the backoff time is reached, determine whether the current timed task has been triggered. If so, it will not be triggered again; otherwise, it will attempt to trigger the current timed task.

9. A distributed task scheduling method, characterized in that, The process includes the following: The node dynamic backoff method according to any one of claims 1-7 is used to trigger the scheduled task. When the scheduled task is triggered, Redis executes the Lua script corresponding to the scheduled task. Determine whether the number of task executions maintained in Redis is less than the number of trigger executions sent by the node. If so, modify the number of task executions maintained in Redis to match the number of trigger executions sent by the node. If not, end the determination process. Using Redis's publish-subscribe mechanism, the system broadcasts its task execution count and the node identifier that successfully triggered the scheduled task to all nodes in the service cluster.

10. The distributed task scheduling method as described in claim 9, characterized in that, When the scheduled task needs to be sharded, the task shards are placed into the Redis-stream queue of the scheduled task in the Lua script, and all nodes in the service cluster act as the consumer group of the Redis-stream queue. Messages are retrieved from the Redis-stream queue for consumption. Once a message has been consumed by one member of the consumer group, the other members of the consumer group will not consume the message again.

11. A distributed task scheduling system, characterized in that, It includes multiple nodes that communicate with each other, and each node can call the Lua script corresponding to the scheduled task in Redis; The Redis maintains the Lua script corresponding to the scheduled task. Each node uses the node dynamic backoff method described in any one of claims 1-8 to trigger the scheduled task. When the scheduled task is triggered, the Redis executes the Lua script. Determine whether the number of task executions maintained in Redis is less than the number of trigger executions sent by the node. If so, modify the number of task executions maintained in Redis to match the number of trigger executions sent by the node. If not, end the determination process. Using Redis's publish-subscribe mechanism, the system broadcasts its task execution count and the node identifier that successfully triggered the scheduled task to all nodes in the service cluster.

12. A computer device, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the node dynamic backoff method as described in any one of claims 1 to 7; or implements the distributed task scheduling method as described in claim 9 or 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and execute the node dynamic backoff method as described in any one of claims 1 to 7; or, execute the distributed task scheduling method as described in claim 9 or 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the node dynamic backoff method as described in any one of claims 1 to 7; or implements the distributed task scheduling method as described in claim 9 or 10.

Images