Two-level distributed task queue management method based on Redis ordered set

Abstract

This invention discloses a two-level distributed task queue management method based on Redis ordered sets, relating to the fields of distributed systems and task scheduling technology. It solves the problems of low scheduling efficiency, unbalanced load, imperfect exception handling mechanisms, and uncontrollable risks in the cleanup process inherent in traditional task queues in distributed environments. This invention achieves separate management of task metadata and data by constructing a two-level queue based on Redis ordered sets; it realizes adaptive optimal scheduling and intelligent consumer matching through a dynamic priority scoring and affinity matching model; it employs distributed locks and atomic operations to ensure concurrency safety and consistency, and combines configurable retry strategies and dynamic cleanup risk assessment to improve fault tolerance and resource reclamation reliability. This invention achieves efficient, reliable, and adaptive distributed task scheduling and management, significantly improving task execution efficiency and overall system stability.

Description

Technical Field

[0001] This invention belongs to the field of distributed systems and task scheduling technology, specifically a two-level distributed task queue management method based on Redis ordered sets. Background Technology

[0002] In the field of distributed systems, task queues are key components for achieving asynchronous processing, peak shaving, and system decoupling. Traditional task queue solutions, such as those based on message middleware like RabbitMQ and Kafka, while relatively mature in terms of reliability, have gradually revealed several bottlenecks in large-scale, high-concurrency scenarios with complex task attributes. Specifically, existing solutions often employ a single-layer queue structure, storing or tightly binding task metadata with specific task data. This results in the need to load large amounts of data during scheduling, impacting scheduling performance and flexibility. At the task scheduling level, traditional methods typically rely on simple first-in-first-out (FIFO) or fixed-priority strategies, lacking a comprehensive consideration of the real-time value of tasks, load, and differences in consumer nodes, easily leading to uneven load distribution, hotspot issues, and overall low execution efficiency. Furthermore, during concurrent consumption, there is a lack of efficient, fine-grained locking mechanisms to ensure the idempotency and consistency of task processing, and retry mechanisms after exceptions are often rigid. The resource cleanup process after task processing is also often overlooked, posing a risk of data loss due to premature or improper cleanup. Redis, as a high-performance in-memory data structure, is well-suited for building ordered queues due to its Sorted Set feature. However, existing Redis-based queue designs are typically simplistic and fail to systematically address the complex issues of scheduling, matching, security, and cleanup. Therefore, a more intelligent, robust, and efficient distributed task queue management method is urgently needed to meet the challenges of modern complex distributed application scenarios.

[0003] The following problems exist in the existing technology: Traditional task queues often store task metadata and specific business data together. Consumers need to load the complete task data when scheduling, which leads to huge network and memory overhead in high-concurrency scenarios, limiting scheduling performance and system scalability. Existing technical solutions mostly rely on simple first-in-first-out or fixed priority rules, which cannot dynamically respond to changes in the real-time value of tasks, data scale, system load and heterogeneous capabilities of consumers, and are prone to low system throughput, uneven load and delays in high-value tasks. In existing technologies, when multiple consumers compete for the same task concurrently, there is a lack of efficient atomic preemption mechanisms, which can easily lead to duplicate task processing or inconsistent states. At the same time, the retry mechanism after task processing failure is usually not flexible enough. Existing technical solutions lack an intelligent cleanup decision-making mechanism after the task is completed; immediate cleanup may result in the loss of data in unfinished stages, while prolonged lack of cleanup may lead to resource leakage. The risks of cleanup operations cannot be quantitatively assessed and dynamically managed. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a two-level distributed task queue management method based on Redis ordered sets to solve the above-mentioned technical problems.

[0005] The first aspect of this invention provides a two-level distributed task queue management method based on Redis ordered sets, comprising the following steps: S1: Construct a first-level task queue and a second-level data queue based on Redis; the first-level task queue is used to store the metadata identifiers of all tasks to be processed in order of scheduled execution time; the second-level data queue is created independently for each task in the first-level task queue and is used to store the specific data items of the corresponding task. S2: Multiple consumers concurrently retrieve task metadata identifiers that have reached their planned execution time from the first-level task queue, and update the sorting score of the identifier in the first-level task queue through atomic operations to mark it as entering the processing state. S3: Based on the acquired task metadata identifier, the consumer first evaluates its own suitability with the task through a dynamic affinity matching model, and then attempts to acquire the corresponding distributed lock based on the evaluation result; if the acquisition is successful, it locates the corresponding second-level data queue based on the task metadata identifier to obtain the task data, and then executes the pre-defined scalable configuration processing operation. S4: After all task data corresponding to the metadata identifier has been processed, release the distributed lock; then, after confirming that all data items in the second-level data queue have been processed, clean up the second-level data queue and finally delete the corresponding metadata identifier record in the first-level task queue.

[0006] Preferably, in step S1, a first-level task queue based on Redis is constructed. This first-level task queue is used to store the metadata identifiers of all tasks to be processed in order of their scheduled execution time, and includes the following steps: The metadata identifier of the task to be processed is used as a member, and the planned execution timestamp of the task is used as the score of the member. Both are stored in a Redis sorted set to complete the construction of the first-level task queue. Based on the feature that members are automatically sorted by score value, the score value represents the planned execution time. By initiating a range query to the Redis sorted set, all members with scores not greater than the current timestamp are requested. The metadata identifiers of all tasks whose planned execution time has arrived are filtered out from the constructed first-level task queue.

[0007] Preferably, in step S1, a second-level data queue based on Redis is constructed. This second-level data queue is created independently for each task in the first-level task queue and is used to store the specific data items of the corresponding task. This includes the following steps: In response to the insertion of a new task metadata identifier into the first-level task queue, an independent Redis sorted set is created and maintained as the corresponding second-level data queue based on the metadata identifier. Each second-level data queue only stores the task data items corresponding to its associated metadata identifier, and the data storage and processing of different tasks are independent of each other.

[0008] Preferably, step S2 includes the following steps: Multiple consumers concurrently query the first-level task queue to obtain the metadata identifiers of tasks that have reached their planned execution time; Each consumer executes a composite atomic command for at least one task metadata identifier obtained. This composite atomic command is essentially an atomic operation executed by Redis's Lua script. The logic of this atomic operation is to read the current score value of the target task metadata identifier in the first-level task queue, and then compare the current score value with the original scheduled execution timestamp. Only if the two are equal will the score value of the identifier be updated to the predefined locked score value; otherwise, failure will be returned. The compound atomic command updates the target task's score to a locked score representing its processing status only if the current score value identified by the target task's metadata is equal to its original scheduled execution timestamp; otherwise, it returns failure. If the composite atomic command is executed successfully, the consumer executing the command successfully obtains the task corresponding to the task metadata identifier and marks the task metadata identifier to enter the processing state; if the composite atomic command fails to execute, the current consumer abandons processing this task metadata identifier and tries to obtain it again; when multiple task metadata identifiers that have reached their planned execution time are obtained, an optimal scheduling process is performed to determine the target task metadata identifier to be processed first.

[0009] Preferably, in step S2, before updating the sorting score through atomic operations, when multiple task metadata identifiers that have reached their planned execution time are obtained from the first-level task queue, the consumer performs an optimal scheduling process, including the following steps: For each candidate task, the metadata identifier corresponds to the task. Calculate dynamic priority scores The calculation formula is as follows: ; Where t is the current time, For the task The planned execution timestamp The time decay constant, The coefficient is a power greater than 1. For the task The estimated business value, For the task The corresponding data item size, The current load rate, The maximum load threshold. For the task The estimated processing load, For average task load, This refers to the load adaptability factor. Based on the calculated dynamic priority score All candidate tasks are sorted in descending order; the consumer executes the composite atomic command to obtain the task with the highest dynamic priority score.

[0010] Preferably, step S3 includes the following steps: To select the most suitable task for processing, consumers evaluate their own suitability with the task through a dynamic affinity matching model. Based on the evaluation results, consumers decide whether to compete for the distributed lock corresponding to the task. After deciding to compete, they generate the corresponding distributed lock key based on the acquired task metadata identifier and attempt to acquire the distributed lock. If the distributed lock is successfully acquired, the consumer obtains exclusive access and processing rights to the unique second-level data queue corresponding to the task's metadata identifier. Subsequently, the consumer locates the second-level data queue based on the task metadata identifier, retrieves the task data item from it, and executes the processing operation of the preset scalable configuration; the processing operation of the scalable configuration includes at least a core processing stage for executing business logic and an exception handling stage for capturing and handling exceptions during the execution process; If the processing of any task data item fails, the task data item will be added back to its second-level data queue according to the preset retry strategy, and the sorting score of the task data item in the second-level data queue will be adjusted for subsequent reprocessing.

[0011] Preferably, in step S3, the suitability between the consumer and the task is evaluated using a dynamic affinity matching model, including the following steps: A dynamic affinity matching model is used to calculate the fit between consumers and the acquired tasks, where consumers... With the task Dynamic affinity The calculation formula is: ; in, To calculate the weighted normalized feature similarity between the consumer feature vector and the task requirement vector, For consumer-based deal with The dynamic trust level is calculated based on the historical success rate and average efficiency of the task type. For consumers To the mission Data storage nodes The network latency and topological distance are measured values. This is the network loss coefficient. For consumers Current local pending load, The preset load threshold, The sensitivity coefficient for load penalty; Consumers based on calculated dynamic affinity The value of determines whether to compete for the distributed lock of the task; and based on the dynamic affinity... Adjust its task processing priority.

[0012] Preferably, step S4 includes the following steps: After releasing the distributed lock, perform an eventual consistency check on the second-level data queue to confirm that there are no unprocessed data items in the queue. Obtain the task attributes, historical anomaly records, and operational health status of the task corresponding to the current metadata identifier. Calculate the risk value R of the current cleanup operation using a dynamic cleanup risk assessment model. The calculation formula is as follows: ; in, A comprehensive indicator to characterize the complexity of the current task. and These represent the recent and historical anomaly frequencies of the current task, respectively. This is the last time the current task was completed. To clear the waiting timeout threshold, This is a confidence value calculated based on the operational health status. , , , This is the adjustment coefficient; Based on the threshold range of the risk value R, the corresponding subsequent cleanup strategy is dynamically selected and executed. If the risk value is lower than the low threshold, it is immediately cleaned up sequentially, that is, the corresponding metadata identifier records in the second-level data queue and the first-level task queue are deleted. If it is in the middle range, the deletion operation is performed after a predetermined time delay. If it is higher than the high threshold, a strong verification process is initiated, and the deletion operation is performed after the verification is passed.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention achieves physical separation of metadata and task data by constructing a two-level queue, enabling the core scheduling process to operate only on lightweight metadata, greatly reducing overhead, improving task distribution efficiency under high concurrency, and achieving complete isolation of different task data processing. This invention uses a dynamic priority scoring and dynamic affinity matching model to make optimal decisions in real time based on task characteristics, status, and consumer capabilities, ensuring that high-value tasks are prioritized and consumer matching is maximized, thereby significantly improving overall processing efficiency and resource utilization. This invention combines composite atomic commands and distributed locks to provide fine-grained mutual exclusion guarantees for task preemption and data processing, effectively preventing duplicate consumption and data inconsistency; combined with a configurable retry strategy, it achieves flexible and reliable recovery from processing anomalies. This invention introduces a dynamic cleanup risk assessment model, which quantitatively analyzes multiple factors such as task complexity, anomaly history, and health status before cleanup, and dynamically selects differentiated cleanup strategies based on risk levels. This fundamentally balances data security and resource release efficiency, and avoids data loss or leakage caused by improper cleanup. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0015] Figure 2 This is a schematic diagram of the two-level queue data structure of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0017] Please see Figure 1 This invention is a two-level distributed task queue management method based on Redis sorted sets, comprising the following steps: S1: Construct a first-level task queue and a second-level data queue based on Redis; the first-level task queue is used to store the metadata identifiers of all tasks to be processed in order of scheduled execution time; the second-level data queue is created independently for each task in the first-level task queue and is used to store the specific data items of the corresponding task. S2: Multiple consumers concurrently retrieve task metadata identifiers that have reached their planned execution time from the first-level task queue, and update the sorting score of the identifier in the first-level task queue through atomic operations to mark it as entering the processing state. S3: Based on the acquired task metadata identifier, the consumer first evaluates its own suitability with the task through a dynamic affinity matching model, and then attempts to acquire the corresponding distributed lock based on the evaluation result; if the acquisition is successful, it locates the corresponding second-level data queue based on the task metadata identifier to obtain the task data, and then executes the pre-defined scalable configuration processing operation. S4: After all task data corresponding to the metadata identifier has been processed, release the distributed lock; then, after confirming that all data items in the second-level data queue have been processed, clean up the second-level data queue and finally delete the corresponding metadata identifier record in the first-level task queue.

[0018] Specifically, firstly, a two-level queue structure based on Redis sorted sets is initialized. The first-level task queue stores the metadata identifiers of all tasks to be processed and sorts them according to the planned execution timestamps. Simultaneously, a second-level data queue is created independently for each task metadata identifier in the first-level queue, specifically for storing the corresponding data item, thus achieving separation and independent management of metadata and business data. Subsequently, during the task scheduling phase, multiple consumers concurrently query the first-level task queue for the metadata identifiers of tasks whose planned execution times have arrived. For each candidate task obtained, the consumer attempts to update its sort score by executing a compound atomic command. Only when the task's score is still the original planned time (i.e., not preempted by other consumers) can it be successfully updated to a lock status value representing "processing," thus atomically preempting the task. After successful preemption, the consumer does not process immediately but first evaluates its own suitability for the task using a dynamic affinity matching model and decides whether to compete for the distributed lock corresponding to the task based on the evaluation result. If the lock is successfully acquired, the consumer locates the corresponding second-level data queue based on the task metadata identifier, retrieves the task data item from it, and executes the business logic with scalable configuration. If processing fails, the data item is reinserted into the queue according to a preset strategy. Finally, after all data items of the task have been processed, the consumer releases the distributed lock and, after confirming that the second-level data queue is empty, safely cleans up the second-level queue and ultimately deletes the corresponding task metadata identifier from the first-level task queue, thus completing the full lifecycle of a task from scheduling, execution to cleanup.

[0019] In one embodiment of the present invention, in step S1, a first-level task queue based on Redis is constructed. The first-level task queue is used to store the metadata identifiers of all tasks to be processed in sorted order according to their planned execution time, including the following steps: The metadata identifier of the task to be processed is used as a member, and the planned execution timestamp of the task is used as the score of the member. Both are stored in a Redis sorted set to complete the construction of the first-level task queue. Based on the feature that members are automatically sorted by score value, the score value represents the planned execution time. By initiating a range query to the Redis sorted set, all members with scores not greater than the current timestamp are requested. The metadata identifiers of all tasks whose planned execution time has arrived are filtered out from the constructed first-level task queue.

[0020] Specifically, a first-level task queue based on a Redis sorted set is constructed to store the scheduling metadata of all pending tasks. A sorted set is created in Redis as the storage medium for the first-level task queue. The key of this sorted set should be named with clear business meaning, such as `first_level_task_queue`. Each pending task (e.g., a user points settlement task or an order status update task) is generated with a globally unique task metadata identifier, such as a Universally Unique Identifier (UUID). Each task is also associated with a planned execution time, which is converted to a Unix timestamp format (precision in seconds or milliseconds) and used as the planned execution timestamp for that task. The core queue construction operation is completed using the Redis `ZADD` command (used to add one or more members to the sorted set and set their scores). Specifically, the task's metadata identifier is used as a member of the sorted set, and the task's planned execution timestamp is used as the member's score. For example, for a task with metadata identifier "task:001" and a planned execution time of Unix timestamp 1700000000, the command executed would be `ZADD first_level_task_queue 1700000000 "task:001"`. The Redis sorted set automatically sorts each member in ascending order based on its score. Since the score represents the planned execution time, the queue forms a sequence ordered from earliest to latest task planned execution time.

[0021] Consumers periodically or continuously query tasks whose scheduled execution time has arrived. This query process is implemented using Redis's ZRANGEBYSCORE command (which retrieves members in a sorted set whose scores fall within a specified range). The consumer obtains the current timestamp (denoted as current_timestamp) and then initiates a range query to retrieve all members whose scores are less than or equal to the current timestamp. The specific command format is ZRANGEBYSCORE first_level_task_queue-inf current_timestamp. This command returns a list containing metadata identifiers for all tasks whose scheduled execution time has arrived or has expired. The time complexity of this range query operation is O(log(N)+M), where N is the total number of members in the sorted set, and M is the number of members returned.

[0022] In distributed, multi-consumer concurrent access scenarios, a safe task "claiming" mechanism based on atomic operations is needed to prevent the same due task from being repeatedly acquired and processed by multiple consumers. A common implementation pattern is that after a consumer acquires a batch of due candidate task identifiers via the ZRANGEBYSCORE command, it does not immediately remove them from the queue. Instead, it performs a composite atomic operation on one of the target identifiers. Specifically, the ZSCORE command (used to retrieve the score of a specified member in an ordered set) checks whether the current score of the identifier in the current queue is still equal to its original planned execution timestamp. If they are equal, it indicates that the task is in an acquireable original state. The ZADD command then atomically updates the identifier's score to a predefined locked score value representing the "processing" state (e.g., a maximum value, or a composite value composed of the consumer identifier and timestamp). The logic of "checking score consistency and updating" must be executed as an indivisible atomic unit. If the operations of "checking task status" and "marking task as in progress" are not executed atomically, other consumers may perform the same check within a very short time window after the check and before the update, causing multiple consumers to think that they have obtained the task at the same time, resulting in duplicate processing of the task.

[0023] In addition, the method includes a task status monitoring and automatic recovery step to handle the problem of long-term task status stagnation caused by unexpected termination of consumer processes. This step periodically identifies task metadata identifiers in the first-level task queue whose sort score is marked as "processing" and whose duration has exceeded a preset timeout threshold. For each such identifier, its sort score is reset to the planned execution timestamp when it was initially inserted into the queue (or an earlier timestamp). After this reset operation, the task will be restored to the "pending execution" state and can be retrieved and processed again by other healthy consumers, realizing automatic fault recovery of task processing.

[0024] In one embodiment of the present invention, in step S1, a second-level data queue based on Redis is constructed. The second-level data queue is created independently for each task in the first-level task queue and is used to store the specific data items of the corresponding task. This includes the following steps: In response to the insertion of a new task metadata identifier into the first-level task queue, an independent Redis sorted set is created and maintained as the corresponding second-level data queue based on the metadata identifier. Each second-level data queue only stores the task data items corresponding to its associated metadata identifier, and the data storage and processing of different tasks are independent of each other.

[0025] Specifically, based on the first-level task queue, a dedicated second-level data queue is created for each task to store the task's specific data items, achieving separate management of metadata and business data. Metadata refers to descriptive information used for scheduling and management, such as the task's unique identifier and planned execution time, while business data refers to the specific content information that the task needs to process. The creation of the second-level data queue is triggered by the insertion event of a new task into the first-level task queue; when a new task's metadata identifier and its planned execution timestamp are successfully inserted into the first-level task queue via the ZADD command, the creation process of a corresponding second-level data queue is immediately triggered.

[0026] The second-level data queue also uses Redis sorted sets as its data structure. Its key name needs to establish a clear and unique association with the task metadata identifier in the first-level task queue. A common naming convention is to concatenate a fixed prefix with the task metadata identifier. For example, if the task metadata identifier in the first-level queue is `task:001`, then the key name for the second-level data queue created for it could be `task_data:task:001`. Each second-level data queue (sorted set) stores all the specific data items for the corresponding task. Each data item is stored as a member of the sorted set and needs to be assigned a score for sorting. The data item itself is usually a serialized (e.g., JSON format) business data string. The score is set according to the specific business logic requirements, such as: a sequence number assigned according to the processing order, a data item creation timestamp, or a business weight value representing processing priority. The `ZADD` command is used to add the data item and its score to the corresponding second-level data queue.

[0027] Each secondary data queue stores only the data item corresponding to its associated task metadata identifier. This design offers several key advantages: First, data items for different tasks are stored under different Redis keys, achieving complete physical storage isolation and preventing data ambiguity between tasks. Second, when processing a specific task, a consumer only needs to interact with the corresponding secondary data queue, allowing data acquisition and processing operations for different tasks to be completely parallel and independent, greatly supporting high-concurrency processing capabilities in a distributed environment. Furthermore, it allows for independent monitoring and management of individual secondary data queues (i.e., individual tasks), such as using the ZCARD command to query the number of members to understand the task's data scale in real time.

[0028] The lifecycle of the second-level data queue is tightly bound to its associated task metadata identifier, encompassing three phases: creation, usage, and cleanup. Its creation phase is synchronized with the insertion of task metadata identifiers into the first-level task queue, ensuring that each task has its own dedicated data container upon scheduling. During usage, consumers locate the corresponding second-level data queue based on the task metadata identifier and use commands such as ZRANGE and ZPOPMIN to read and process data items. If a data item fails to process, it can be re-inserted into the original queue with an adjusted score (e.g., increased latency) using the ZADD command, according to a preset retry policy, awaiting subsequent retries. Once all data items associated with a task have been processed, and after releasing the distributed lock and completing eventual consistency verification, the cleanup phase begins. At this point, the second-level data queue for that task is deleted, and the corresponding metadata identifier record is removed from the first-level task queue, achieving closed-loop management and timely reclamation of task resources.

[0029] The data structure relationship of the two-level queue can be found in [reference]. Figure 2 As shown in the figure, the first-level task queue stores task metadata identifiers in order of planned execution time (Score); each task metadata identifier (such as Task_1) is associated with an independent second-level data queue, which is used to store the specific data items of the task.

[0030] In one embodiment of the present invention, step S2 includes the following steps: Multiple consumers concurrently query the first-level task queue to obtain the metadata identifiers of tasks that have reached their planned execution time; Each consumer executes a composite atomic command for at least one task metadata identifier obtained. This composite atomic command is essentially an atomic operation executed by Redis's Lua script. The logic of this atomic operation is to read the current score value of the target task metadata identifier in the first-level task queue, and then compare the current score value with the original scheduled execution timestamp. Only if the two are equal will the score value of the identifier be updated to the predefined locked score value; otherwise, failure will be returned. The compound atomic command updates the target task's score to a locked score representing its processing status only if the current score value identified by the target task's metadata is equal to its original scheduled execution timestamp; otherwise, it returns failure. If the composite atomic command is executed successfully, the consumer executing the command successfully obtains the task corresponding to the task metadata identifier and marks the task metadata identifier to enter the processing state; if the composite atomic command fails to execute, the current consumer abandons processing this task metadata identifier and tries to obtain it again; when multiple task metadata identifiers that have reached their planned execution time are obtained, an optimal scheduling process is performed to determine the target task metadata identifier to be processed first.

[0031] Specifically, multiple consumers run concurrently, each independently initiating queries to the first-level task queue. Each consumer periodically or event-drivenly executes the Redis ZRANGEBYSCORE command, requesting all members whose scores (i.e., planned execution timestamps) are less than or equal to the current timestamp, thereby obtaining a list of currently expired task metadata identifiers (called candidate_task_ids). This query is a read-only operation, and concurrent execution by multiple consumers will not cause data conflicts.

[0032] When the consumer receives a list of task metadata identifiers containing multiple identifiers, it needs to select one for further processing. At this point, the consumer enters the optimal scheduling process. This process uses a dynamic priority scoring model to calculate, evaluate, and rank all candidate tasks in the list, ultimately selecting the single task metadata identifier with the highest priority as the target identifier for this processing (denoted as target_task_id). If there is only one candidate task, it is directly designated as the target identifier.

[0033] To absolutely prevent multiple consumers from simultaneously acquiring the same task, the operations of "checking the current status of the task" and "marking it as processing" are merged into an indivisible atomic operation. This atomic operation is implemented through Redis's Lua script mechanism, and its logic is as follows: First, the ZSCORE command (used to retrieve the score of a specified member in a sorted set) is used to read the current score of the target identifier in the current queue; then, the current score is compared with the input original plan execution timestamp; only if the two are strictly equal, the ZADD command (used to add or update the score of a member in a sorted set) is used to update the score to the locked value. These three steps are executed as an atomic unit in the Lua script, ensuring operational consistency and safety in a distributed concurrent environment. The logic of this Lua script is as follows: Input the metadata identifier of the target task (i.e., the target identifier), the original planned execution timestamp obtained during the query, and a lock score representing the processing status. The script first uses the ZSCORE command to obtain the target identifier's current score in the first-level task queue. Then, it checks if the current score equals the original planned execution timestamp (indicating that the task status has not been modified by any other consumer since the query and is still in the original pending state). Only then does the script execute the ZADD command, atomically updating the target identifier's score from the original planned execution timestamp to the lock score. The lock score is a specially constructed value (e.g., current timestamp + fixed timeout value) used to uniquely identify that the task has been locked in the queue and entered the subsequent business data processing flow. If the above condition (current score equals original planned execution timestamp) is met and the update is successful, the script returns a success flag, meaning the current consumer has successfully acquired the task. If the current score does not equal the original planned execution timestamp (indicating the task has been modified by another consumer), the script immediately returns a failure flag.

[0034] If the composite atomic command (an atomic operation executed by a Lua script) returns successfully, the current consumer is confirmed as the legitimate acquirer of the task's target identifier. If the command returns unsuccessful, it indicates that the attempt to acquire and lock the task failed due to contention. The current consumer immediately abandons processing the target identifier and retryes according to a preset strategy. The strategy includes selecting the next priority task from the task metadata identifier list as the new target identifier and repeating the atomic operation, or re-initiating the entire query process after a short random delay.

[0035] A safeguard step is added to handle exceptions where tasks are permanently locked due to consumer failure. This step periodically checks the first-level task queue to identify which tasks have a score (i.e., a locked score) that is lower than the current timestamp and indicates that they have been in a "processing" state for a long time. For tasks that have not been completed due to timeout, their scores are reset to the original planned execution timestamp or an earlier timestamp, restoring them to an executable state so that they can be retrieved and processed by other consumers.

[0036] In one embodiment of the present invention, in step S2, before updating the sorting score through atomic operations, when multiple task metadata identifiers that have reached their planned execution time are obtained from the first-level task queue, the consumer performs an optimal scheduling process, including the following steps: For each candidate task, the metadata identifier corresponds to the task. Calculate dynamic priority scores The calculation formula is as follows: ; Where t is the current time, For the task The planned execution timestamp The time decay constant, The coefficient is a power greater than 1. For the task The estimated business value, For the task The corresponding data item size, The current load rate, The maximum load threshold. For the task The estimated processing load, For average task load, This refers to the load adaptability factor. Based on the calculated dynamic priority score All candidate tasks are sorted in descending order; the consumer executes the composite atomic command to obtain the task with the highest dynamic priority score.

[0037] Specifically, when a consumer executes a ZRANGEBYSCORE query, if the number of candidate task metadata identifiers returned is greater than one, this optimal scheduling process is triggered; the input to this process is the set of candidate tasks. And the static properties of each task (such as the scheduled execution timestamp). Estimated business value Estimated processing load and the size of the data items in its second-level data queue Meanwhile, consumers need to obtain real-time status parameters, including the current time t and the current load rate. .

[0038] For each candidate task First, calculate the time urgency factor. ;in,( This represents the duration (in seconds) the task has been waiting, and its value is greater than or equal to 0. For the task The planned execution timestamp; Used to calculate task waiting time ( Normalization, in its physical sense, is a scaling unit for the time effect. Its value ranges from a single real number to a value greater than 0. It is set according to the business scenario; for example, it can be used to... Setting it to 3600 (seconds) indicates that the impact of the waiting time is measured in hours; This amplifies the urgency level of timed-out tasks; its value must be greater than 1, and is typically set to 2 or 3. The higher the value, the more significant the priority increase effect on timed-out tasks.

[0039] Next, calculate the value density factor. ;in, The estimated business value of a task is a dimensionless positive real number. The specific value is usually specified by the task submitter or obtained by mapping according to the type when the task is created. The larger the value, the higher the business importance of the task. The size of the task data items is usually the total number of members in the second-level data queue of the task (obtained via the ZCARD command), and is a non-negative integer.

[0040] Then calculate the load affinity factor. Among them, the current load rate The current load is a scalar value within the range [0,1], representing the overall busy level. It is calculated as the ratio of the number of currently active consumers to the maximum allowed number of consumers, or as the weighted average of the recent average CPU / memory utilization of all consumers; the maximum load threshold... This is typically set to a value slightly less than 1, such as 0.8 or 0.9, to reserve buffer capacity; average task load By estimating the load of completed tasks The load fit coefficient is obtained by performing a moving average calculation. Used to adjust the influence of the current load and task load matching degree on priority; its value is a real number greater than 0, and the initial value is usually set to 1. For the task The estimated processing load (i.e., task load) is a real number greater than or equal to 0. When a task is created, a non-negative real number is preset based on the task type, the expected amount of data to be processed, the required computing resources (such as CPU time and memory usage), or the actual resource consumption statistics of similar tasks in the past. This number represents the resource consumption of the task relative to other tasks. For example, tasks can be divided into lightweight, medium-weight, and heavyweight according to their processing complexity, and assigned load values ​​of 0.5, 1.0, and 2.0 respectively.

[0041] Final comprehensive dynamic priority score A higher value indicates a higher overall priority for processing the task at the current moment and in the current state. The consumer calculates all candidate tasks. Dynamic priority scoring Then, according to The values ​​are sorted in descending order; the task at the top of the list after sorting is the task with the highest overall priority at the current moment and in the current state. The consumer selects this task as its current target task and immediately executes a compound atomic command (Lua script) on the task's metadata identifier to attempt to acquire and lock the task; if the lock is successful, the subsequent data processing flow begins; if the competition fails, the next task in the list can be selected to retry the atomic operation, or the process can be rolled back and re-queried, depending on the strategy.

[0042] In one embodiment of the present invention, step S3 includes the following steps: To select the most suitable task for processing, consumers evaluate their own suitability with the task through a dynamic affinity matching model. Based on the evaluation results, consumers decide whether to compete for the distributed lock corresponding to the task. After deciding to compete, they generate the corresponding distributed lock key based on the acquired task metadata identifier and attempt to acquire the distributed lock. If the distributed lock is successfully acquired, the consumer obtains exclusive access and processing rights to the unique second-level data queue corresponding to the task's metadata identifier. Subsequently, the consumer locates the second-level data queue based on the task metadata identifier, retrieves the task data item from it, and executes the processing operation of the preset scalable configuration; the processing operation of the scalable configuration includes at least a core processing stage for executing business logic and an exception handling stage for capturing and handling exceptions during the execution process; If the processing of any task data item fails, the task data item will be added back to its second-level data queue according to the preset retry strategy, and the sorting score of the task data item in the second-level data queue will be adjusted for subsequent reprocessing.

[0043] Specifically, after a consumer updates the score of the task metadata identifier in the first-level queue to the "processing" status through an atomic operation, it does not immediately start processing the task. First, the consumer calls the dynamic affinity matching model and outputs a quantified fit score to evaluate its fit with the task. The consumer decides whether to actually compete for the processing right of the task based on this score. If the fit is higher than the preset competition threshold, it decides to participate in the subsequent distributed lock competition. If the fit is too low, the consumer can choose to give up the competition and reset the score of the task in the first-level queue back to its original planned execution timestamp, so that it can be acquired by other more suitable consumers. Then, the consumer itself returns to acquire other tasks.

[0044] Once a consumer decides to compete for a task, it needs to obtain exclusive access to that task to ensure that, in a distributed environment, the data for the same task is not processed by multiple consumers simultaneously, which could lead to state chaos. This access is implemented through a distributed lock. The key of the distributed lock is deterministically generated based on the task metadata identifier, for example, by adding the prefix "lock:" before the task metadata identifier to form a lock key such as "lock:task:001". This ensures that each task has one and only one corresponding distributed lock.

[0045] Consumers use Redis's SET command, along with the NX (Not eXists, set only if the key does not exist) and PX (set millisecond expiration time) options, to attempt to acquire a distributed lock in an atomic operation. For example, executing the command SET lock:task:001<consumer_identifier> The NX PX 30000 command sets the value of lock:task:001 to the unique identifier (e.g., UUID) of the current consumer and assigns it a short expiration time (e.g., 30 seconds) only if lock:task:001 does not exist. If the command executes successfully, it returns OK, indicating that the current consumer has successfully acquired the distributed lock for the task. If the key already exists, the command execution fails, indicating that the lock is already held by another consumer, and the current consumer fails to acquire the lock. If the acquisition of the distributed lock fails, the current consumer releases the processing mark it set in the first-level queue of the task through atomic operations and returns to the beginning of the task acquisition process or tries other tasks.

[0046] Successfully acquiring the distributed lock means that the consumer obtains exclusive access and processing rights to the unique second-level data queue corresponding to the task's metadata identifier. During the validity period of this distributed lock, no other consumer can successfully acquire the same distributed lock, and therefore cannot access the same second-level data queue. Based on the task's metadata identifier, the consumer derives the key name of the second-level data queue according to the naming rules consistent with its creation; for example, for the task metadata identifier task:001, its data queue key is task_data:task:001.

[0047] Consumers read one or more data items from the second-level data queue using Redis commands for processing. These data items typically contain serialized business data. Consumers execute pre-defined, scalable processing operations on the retrieved task data items. This processing is designed to be modular and configurable, comprising at least two distinct phases. The first phase is the core processing phase, which is the main stage for executing business logic. Its specific behavior is not hard-coded but defined through scalable configuration. For example, by configuring a "task type-processor" mapping table during initialization, the corresponding business logic processor is dynamically loaded and executed based on the type field in the task metadata identifier when processing a task. The second phase is the exception handling phase, a standardized fault-tolerance layer wrapped around the core processing phase. This phase intervenes before and after the execution of core logic, and when exceptions are caught. Its responsibilities include recording the start and end of processing logs, capturing all exceptions thrown by the core processing phase, classifying exceptions by type, recording detailed exception context information for analysis, and ultimately deciding whether to re-throw the exception to trigger a retry or mark it as a non-retryable error.

[0048] If the processing of any task data item is deemed to have failed during the exception handling phase and retry is allowed, the preset retry strategy is activated. First, the failed data item is added back to its original second-level data queue using the ZADD command, with the original data item as a member; at the same time, a new, adjusted sort score is set for the newly added data item. Common adjustment strategies include the following: First, fixed-delay retries, where the new score is set to the current timestamp plus a fixed delay interval (e.g., 5 seconds), making the data item available again after a short delay. Second, exponential backoff retries, where the new score is set to the current timestamp plus the initial delay multiplied by the backoff base raised to the power of the number of retries. For example, the first retry is delayed by 2 seconds, the second by 4 seconds (2 x 2 to the power of 1), the third by 8 seconds (2 x 2 to the power of 2), and so on, avoiding meaningless frequent retries for faulty tasks. Third, priority downgrading, where the new score is set to the original score (or a baseline score) plus a larger penalty value, lowering the data item's ranking position in the queue and prioritizing other data items. Typically, the number of retries is recorded in the data item or task metadata. When the number of retries exceeds the configured maximum threshold, the data item will no longer be added back to the queue but will be transferred to a dedicated "dead-letter queue" for manual or specific process subsequent inspection and processing, preventing infinite retry loops.

[0049] In one embodiment of the present invention, step S3 involves evaluating the fit between the consumer and the task using a dynamic affinity matching model, including the following steps: A dynamic affinity matching model is used to calculate the fit between consumers and the acquired tasks, where consumers... With the task Dynamic affinity The calculation formula is: ; in, To calculate the weighted normalized feature similarity between the consumer feature vector and the task requirement vector, For consumer-based deal with The dynamic trust level is calculated based on the historical success rate and average efficiency of the task type. For consumers To the mission Data storage nodes The network latency and topological distance are measured values. This is the network loss coefficient. For consumers Current local pending load, The preset load threshold, The sensitivity coefficient for load penalty; Consumers based on calculated dynamic affinity The value of determines whether to compete for the distributed lock of the task; and based on the dynamic affinity... Adjust its task processing priority.

[0050] Specifically, when the consumer (denoted as...) A task metadata identifier (corresponding to the task) was successfully marked from the first-level queue. After the state is being processed, the calculation of the dynamic affinity matching model is triggered before attempting to acquire the distributed lock for the task.

[0051] First, calculate the weighted normalized feature similarity between the consumer feature vector and the task requirement vector. Consumer feature vector It includes multiple dimensions to encode its processing capabilities, such as CPU architecture identifiers, programming language environment versions, etc.; task requirement vector. This includes requirements for each dimension, such as specifying the need for an ARM architecture or a specific GPU model for computation; each dimension can be represented numerically (e.g., version number), boolean, or categorically. For each dimension k (k ranges from 1 to K, where K is the total number of dimensions in the consumer feature vector and the task requirement vector), a weight is assigned. Calculate the feature similarity of two vectors in each dimension k. For numerical dimensions, adopt (in (This refers to the range of values ​​for this dimension); for Boolean or categorical dimensions, =1 if and only if the two are a perfect match, otherwise 0. The final weighted normalized feature similarity is the weighted sum of the similarities of each dimension: This value is a real number between [0,1], and the closer the value is to 1, the higher the feature matching degree.

[0052] Then calculate the dynamic trust level. For each consumer-task type pair Maintain records within a sliding time window (e.g., the last 24 hours or the last 100 processing sessions), including the number of successful processing sessions (Success), the total number of processing sessions (Total), and the efficiency set for each processing session (e.g., data volume processed / time taken). The success rate is calculated as follows: The average efficiency is calculated by averaging the efficiency set and normalizing it to the [0,1] interval. Dynamic trust level is a combination of success rate and average efficiency, that is... ;in, It is a balancing coefficient between [0,1] used to adjust the weight of success rate and efficiency in trust level.

[0053] Dynamic affinity is obtained through a dynamic affinity matching model. The calculation formula is: ;in, For consumers To the mission Data storage nodes Network latency and topological distance metrics are typically expressed as Round-Trip Time (RTT), measured in milliseconds (ms). Network loss coefficient. A constant greater than 0 is used to correlate network latency with topological distance metric. Converted to an amplification factor, its physical meaning is the proportion of performance loss per unit of network latency (e.g., 1ms); for example, if set... =0.01, then a delay of 100ms will make the network part in the denominator become 1+0.01∗100=2, which means that network loss halves the effective processing capacity; its value range is usually a small positive number (such as 0.001 to 0.1). The larger, A higher value leads to a lower dynamic affinity A value, consistent with the common knowledge that higher network latency results in lower processing efficiency. Local pending load. This is a non-negative integer or real number that quantifies the current busy level of the consumer. Specifically, it can be the number of pending task data items in the consumer's local memory queue, or the number of task subtasks that are being executed concurrently. Load threshold. This is a preset positive constant, representing the maximum load boundary that a single consumer can effectively handle; when ≥ This indicates that consumers are already overloaded. The sensitivity coefficient of load penalty. It is a constant greater than 0, which controls the strength of the load penalty on fitness. The larger, The faster the decay, The higher the value, the lower the dynamic affinity A value, and the more severe the penalty on the load; its typical range is 0.5 ≤ ≤3.

[0054] Consumers preset a competition threshold (Usually an empirical value greater than 0, such as 0.5); if the calculated If so, the consumer decides to participate in the subsequent competition for the distributed lock for that task; if If a consumer deems themselves unsuitable for the task, they will withdraw from the competition and be removed from the first-level queue according to the predetermined process. The "processing" flag is added to make it accessible to other consumers. In scenarios with multiple task candidates or resource scheduling, The value of A can be directly used to adjust the task processing priority; for example, when multiple consumers compete for the same task, it can be designed so that the consumer with the highest adaptability is more likely to acquire the lock; or when consumers sort multiple marked tasks locally, they can be processed in descending order based on the value of A.

[0055] In one embodiment of the present invention, step S4 includes the following steps: After releasing the distributed lock, perform an eventual consistency check on the second-level data queue to confirm that there are no unprocessed data items in the queue. Obtain the task attributes, historical anomaly records, and operational health status of the task corresponding to the current metadata identifier. Calculate the risk value R of the current cleanup operation using a dynamic cleanup risk assessment model. The calculation formula is as follows: ; in, A comprehensive indicator to characterize the complexity of the current task. and These represent the recent and historical anomaly frequencies of the current task, respectively. This is the last time the current task was completed. To clear the waiting timeout threshold, This is a confidence value calculated based on the operational health status. , , , This is the adjustment coefficient; Based on the threshold range of the risk value R, the corresponding subsequent cleanup strategy is dynamically selected and executed. If the risk value is lower than the low threshold, it is immediately cleaned up sequentially, that is, the corresponding metadata identifier records in the second-level data queue and the first-level task queue are deleted. If it is in the middle range, the deletion operation is performed after a predetermined time delay. If it is higher than the high threshold, a strong verification process is initiated, and the deletion operation is performed after the verification is passed.

[0056] Specifically, after the consumer completes processing all corresponding task data items based on the task metadata identifier and releases the distributed lock for that task, it does not immediately delete the first-level queue record and the second-level data queue for that task. An eventual consistency check must be performed to absolutely confirm that all data related to the task has been properly processed and that there are no remaining unprocessed or retryable items. The goal of the check is to confirm that the second-level data queue corresponding to the current task metadata identifier no longer contains any members. The check method is to use the Redis ZCARD command (which returns the total number of members in a specified sorted set) to query the cardinality of the second-level data queue. If the ZCARD return value is 0, it proves that the queue is empty and all data items have been processed.

[0057] Before the final cleanup, it is necessary to collect multi-dimensional status information of the task and calculate the cleanup risk value R using a dynamic cleanup risk assessment model. The calculation formula is as follows: Among them, the comprehensive index of task complexity Let be a non-negative real number used to quantify the complexity of task processing; its calculation is based on the static attributes and dynamic behavior of the task, such as the task's default processor type (whether it involves external service calls or transaction operations), the maximum size of data items reached by its secondary data queue, etc.; these factors can be synthesized into a scalar value through weighted summation or more complex aggregation functions (such as principal component analysis). ; A higher value indicates a more complex task, and a potentially higher risk of state remnants or hidden errors; for example, the overall task complexity index for a medium-complexity task. The possible value is 2. The task complexity impact factor is a real number greater than 0, which controls the complexity. The strength of its contribution to the basic component of the risk value; The larger, Follow The faster the growth and saturation, the better; its value is usually set by fitting historical task processing data (e.g., analyzing the correlation between historical task complexity and problems that occurred after cleanup) or based on domain experience, for example, set to 0.5.

[0058] Recent Abnormal Frequency This represents the number of times anomalies (such as processing failures, retries, timeouts) occurred in the current task within the most recent time window (e.g., within the last hour or the last 10 processing cycles) (e.g., a value of 3); historical anomaly frequency. This is the average anomaly frequency of the task over its entire lifecycle or a longer historical window, such as the total number of anomalies divided by the total number of processing operations (with a value of 10), or a moving average. For a very small positive number (such as...) ), used to prevent the denominator from being zero and to smooth the calculation.

[0059] t is the current timestamp; The timestamp marking the completion of the last data item in the current task should be obtained from the task context or audit logs; To clean up the wait timeout threshold, a preset normal number (e.g., 300 seconds) is used to define the recommended minimum wait period between the task's "logical completion" and "allowing physical cleanup" to accommodate possible message delays or the achievement of eventual consistency. This is the weighting coefficient for the time factor, with a value range of, for example, [0, 5], and a typical value of 1.2. It controls the relative importance of the waiting time factor in risk calculation. The hyperbolic tangent function is used to weight the waiting time... Mapped to the [0,1) interval; the time factor term adopts... The format indicates that the closer the cleanup operation is to the task's final completion time, the higher the risk; as the waiting time increases to near or exceeding the timeout threshold... The risk contribution gradually diminishes, reflecting the time buffer and risk trade-off required for eventual consistency convergence in a distributed system.

[0060] Running health status confidence value This is a confidence scalar between [0,1] calculated based on the operational health status during task processing. Health status H can be evaluated and aggregated from multiple dimensions, including whether the consumer instance executing the task remains active and healthy during and after processing, whether temporary resources requested during task processing (such as database connections and file handles) have been explicitly released, and whether audit logs for all key steps of the task (start, data processing, exceptions, and end) have been successfully persisted without interruption. A comprehensive confidence value is ultimately calculated by weighting these Boolean or hierarchical indicators. For example, a typical value is 0.9. The higher the value, the healthier and more predictable the running status is when the task ends, and the lower the cleanup risk. This is the confidence weighting coefficient for health status, with a value range of, for example, [0,3], and a typical value of 0.8. It represents the degree to which operational health status contributes to risk reduction. The higher the value, the better the health task ( The higher the risk level, the greater the risk reduction.

[0061] After calculating the risk value R, it is compared with two preset thresholds, and the low-risk threshold is selected. and high risk threshold (satisfy The specific value needs to be set based on the business's risk tolerance and historical cleanup data; for example, it can be set to 0.3 and 0.7 respectively. Based on the interval where R is located, dynamically select and execute one of the following three cleanup strategies: immediate sequential cleanup (when...). This indicates extremely low risk of cleanup; an atomic cleanup sequence is executed immediately: first, the second-level data queue of the task is deleted (using the DEL command); then, the corresponding task metadata identifier record is deleted from the first-level task queue (using the ZREM command, which removes one or more specified members from an ordered set). This strategy is suitable for simple, stable, healthy tasks that have waited sufficiently, aiming to quickly reclaim resources. Cleanup is delayed until a predetermined time has elapsed (when...). This indicates a moderate risk of cleanup. Instead of immediate cleanup, the task's metadata identifier score in the first-level queue is modified to a future point in time (e.g., current time + a predetermined delay, such as 60 seconds). Simultaneously, a callback is set, or the cleanup daemon re-triggers the final consistency check and risk assessment for the task after the delay expires. This strategy provides additional buffer time for potential delayed messages or final state convergence. Cleanup is performed after the strong verification process is initiated (when...). This indicates a high risk of cleanup and initiates a strong verification process. This process includes: re-querying the status of all downstream processes related to the task, reviewing the persistence of all data processing results, and even performing a simulated rollback test. Only after this strong verification process passes will the final queue deletion operation be performed. If the verification fails, an alert will be generated and the task status may be marked as "pending manual verification," pausing automatic cleanup.

[0062] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A two-level distributed task queue management method based on Redis sorted sets, characterized in that, Includes the following steps: S1: Construct a first-level task queue and a second-level data queue based on Redis; the first-level task queue is used to store the metadata identifiers of all tasks to be processed in order of scheduled execution time; the second-level data queue is created independently for each task in the first-level task queue and is used to store the specific data items of the corresponding task. S2: Multiple consumers concurrently retrieve task metadata identifiers that have reached their planned execution time from the first-level task queue, and update the sorting score of the identifier in the first-level task queue through atomic operations to mark it as entering the processing state. S3: Based on the acquired task metadata identifier, the consumer first evaluates its own suitability with the task through a dynamic affinity matching model, and then attempts to acquire the corresponding distributed lock based on the evaluation result. If the data is successfully retrieved, the corresponding second-level data queue is located based on the task metadata identifier to obtain the task data, and then the processing operation of the preset extensible configuration is executed. S4: After all task data corresponding to the metadata identifier has been processed, release the distributed lock; Subsequently, after confirming that all data items in the second-level data queue have been processed, the second-level data queue is cleaned up, and finally the corresponding metadata identifier record in the first-level task queue is deleted.

2. The two-level distributed task queue management method based on Redis ordered sets according to claim 1, characterized in that, In step S1, a first-level task queue based on Redis is constructed. This first-level task queue is used to store the metadata identifiers of all tasks to be processed in sorted order according to their planned execution time. The steps include: The metadata identifier of the task to be processed is used as a member, and the planned execution timestamp of the task is used as the score of the member. Both are stored in a Redis sorted set to complete the construction of the first-level task queue. Based on the feature that members are automatically sorted by score value, the score value represents the planned execution time. By initiating a range query to the Redis sorted set, all members with scores not greater than the current timestamp are requested. The metadata identifiers of all tasks whose planned execution time has arrived are filtered out from the constructed first-level task queue.

3. The two-level distributed task queue management method based on Redis ordered sets according to claim 2, characterized in that, In step S1, a second-level data queue based on Redis is constructed. This second-level data queue is created independently for each task in the first-level task queue and is used to store the specific data items of the corresponding task. The steps include: In response to the insertion of a new task metadata identifier into the first-level task queue, an independent Redis sorted set is created and maintained as the corresponding second-level data queue based on the metadata identifier. Each second-level data queue only stores the task data items corresponding to its associated metadata identifier, and the data storage and processing of different tasks are independent of each other.

4. The two-level distributed task queue management method based on Redis ordered sets according to claim 1, characterized in that, S2 includes the following steps: Multiple consumers concurrently query the first-level task queue to obtain the metadata identifiers of tasks that have reached their planned execution time; Each consumer executes a composite atomic command for at least one task metadata identifier obtained. This composite atomic command is essentially an atomic operation executed by Redis's Lua script. The logic of this atomic operation is to read the current score value of the target task metadata identifier in the first-level task queue, and then compare the current score value with the original scheduled execution timestamp. Only if the two are equal will the score value of the identifier be updated to the predefined locked score value; otherwise, failure will be returned. The compound atomic command updates the target task's score to a locked score representing its processing status only if the current score value identified by the target task's metadata is equal to its original scheduled execution timestamp; otherwise, it returns failure. If the composite atomic command is executed successfully, the consumer executing the command successfully obtains the task corresponding to the task metadata identifier and marks the task metadata identifier to enter the processing state; if the composite atomic command fails to execute, the current consumer abandons processing this task metadata identifier and tries to obtain it again; when multiple task metadata identifiers that have reached their planned execution time are obtained, an optimal scheduling process is performed to determine the target task metadata identifier to be processed first.

5. The two-level distributed task queue management method based on Redis ordered sets according to claim 4, characterized in that, In step S2, before updating the sorting score through atomic operations, when multiple task metadata identifiers that have reached their planned execution time are obtained from the first-level task queue, the consumer performs an optimal scheduling process, including the following steps: For each candidate task, the metadata identifier corresponds to the task. Calculate dynamic priority scores The calculation formula is as follows: ； Where t is the current time, For the task The planned execution timestamp The time decay constant, The coefficient is a power greater than 1. For the task The estimated business value, For the task The corresponding data item size, The current load rate, The maximum load threshold. For the task The estimated processing load, For average task load, This refers to the load adaptability factor. Based on the calculated dynamic priority score All candidate tasks are sorted in descending order; the consumer executes the composite atomic command to obtain the task with the highest dynamic priority score.

6. The two-level distributed task queue management method based on Redis ordered sets according to claim 1, characterized in that, S3 includes the following steps: To select the most suitable task for processing, consumers evaluate their own suitability with the task through a dynamic affinity matching model. Based on the evaluation results, consumers decide whether to compete for the distributed lock corresponding to the task. After deciding to compete, they generate the corresponding distributed lock key based on the acquired task metadata identifier and attempt to acquire the distributed lock. If the distributed lock is successfully acquired, the consumer obtains exclusive access and processing rights to the unique second-level data queue corresponding to the task's metadata identifier. Subsequently, the consumer locates the second-level data queue based on the task metadata identifier, retrieves the task data item from it, and executes the processing operation of the preset scalable configuration; the processing operation of the scalable configuration includes at least a core processing stage for executing business logic and an exception handling stage for capturing and handling exceptions during the execution process; If the processing of any task data item fails, the task data item will be added back to its second-level data queue according to the preset retry strategy, and the sorting score of the task data item in the second-level data queue will be adjusted for subsequent reprocessing.

7. The two-level distributed task queue management method based on Redis ordered sets according to claim 6, characterized in that, In step S3, a dynamic affinity matching model is used to evaluate the fit between the consumer and the task, including the following steps: A dynamic affinity matching model is used to calculate the fit between consumers and the acquired tasks, where consumers... With the task Dynamic affinity The calculation formula is: ； in, To calculate the weighted normalized feature similarity between the consumer feature vector and the task requirement vector, For consumer-based deal with The dynamic trust level is calculated based on the historical success rate and average efficiency of the task type. For consumers To the mission Data storage nodes The network latency and topological distance are measured values. This is the network loss coefficient. For consumers Current local pending load, The preset load threshold, The sensitivity coefficient for load penalty; Consumers based on calculated dynamic affinity The value of determines whether to compete for the distributed lock of the task; and based on the dynamic affinity... Adjust its task processing priority.

8. The two-level distributed task queue management method based on Redis ordered sets according to claim 1, characterized in that, S4 includes the following steps: After releasing the distributed lock, perform an eventual consistency check on the second-level data queue to confirm that there are no unprocessed data items in the queue. Obtain the task attributes, historical anomaly records, and operational health status of the task corresponding to the current metadata identifier. Calculate the risk value R of the current cleanup operation using a dynamic cleanup risk assessment model. The calculation formula is as follows: ； in, A comprehensive indicator to characterize the complexity of the current task. and These represent the recent and historical anomaly frequencies of the current task, respectively. This is the last time the current task was completed. To clear the waiting timeout threshold, This is a confidence value calculated based on the operational health status. , , , This is the adjustment coefficient; Based on the threshold range of the risk value R, the corresponding subsequent cleanup strategy is dynamically selected and executed. If the risk value is lower than the low threshold, it is immediately cleaned up sequentially, that is, the corresponding metadata identifier records in the second-level data queue and the first-level task queue are deleted. If it is in the middle range, the deletion operation is performed after a predetermined time delay. If it is higher than the high threshold, a strong verification process is initiated, and the deletion operation is performed after the verification is passed.

Images