ID verification and delay scheduling co-processing system and method based on Redis script
Through the ID verification and delay scheduling collaborative processing system based on Redis scripts, the problems of insufficient state expression ability and limited system fault tolerance in big data processing are solved, and the fine diversion of business logic and service stability in a high-concurrency environment are achieved, thereby improving the robustness of the system.
Patent Information
- Application Number
- CN202510891466.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2025-10-14
AI Technical Summary
Existing technologies in big data processing have problems such as insufficient state expression capabilities, strong coupling between time series and state, and limited system fault tolerance, which leads to simple and crude business processing logic, high risk of misjudgment, service anomalies and data loss in high-concurrency environments.
A Redis script-based ID verification and delay scheduling collaborative processing system is adopted, including a uniqueness verification module, a time window calculation module, a delay task scheduling module, a state cache module and a dynamic script hot loading module. Lua scripts are used to implement unique ID verification, delay processing and state logic separation to ensure atomicity and fault tolerance.
It achieves fine-grained diversion of business logic, avoids race conditions, ensures uninterrupted services, improves system robustness and state isolation capabilities, and is suitable for high-concurrency scenarios.
Smart Images
Figure CN120780431A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and in particular relates to a Redis script-based ID verification and delay scheduling collaborative processing system and method. Background Art
[0002] With the rapid development of big data technology, the scale of data collection and processing is growing exponentially. In practical applications, particularly in scenarios such as government and enterprise data aggregation, intelligent monitoring, and financial risk control, systems often need to collect, identify, filter, and process massive amounts of data in a very short period of time. When data volumes reach tens of millions or even higher, traditional data uniqueness verification and processing mechanisms face severe bottlenecks, making it difficult to meet the multiple requirements of high concurrency, low latency, and strong consistency.
[0003] Currently, most mainstream solutions implement verification of unique identifiers (such as order numbers, transaction IDs, etc.) through single-point caches or databases, but they have the following prominent problems:
[0004] Insufficient state expression capabilities: Existing methods typically only support binary judgments of "existence / non-existence" and cannot effectively distinguish between business states such as initial submission, short-term retry, and abnormal recovery. This leads to simplistic and crude business processing logic and the risk of misjudgment.
[0005] Strong coupling between timing and state: There is a lack of atomicity between uniqueness verification and subsequent asynchronous tasks (such as delayed retries and compensation operations), which may lead to inconsistent intermediate states under high concurrency.
[0006] The system has limited fault tolerance: In cases such as Redis cache script failure, network jitter, and master-slave switching, the existing solution cannot automatically restore the verification logic, which can easily lead to service anomalies or data loss. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to provide a Redis script-based ID verification and delay scheduling collaborative processing system and method to solve at least one of the above-mentioned problems existing in the prior art.
[0008] Based on the above objectives, one or more embodiments of the present application provide a Redis script-based ID verification and delay scheduling collaborative processing system, including:
[0009] The uniqueness verification module is used to receive the unique ID submitted externally, verify the unique ID by calling the Lua script in Redis, and return the status code corresponding to the verification status;
[0010] The time window calculation module is used to calculate the remaining delay time based on the current system timestamp and the timestamp of the historical records;
[0011] The delayed task scheduling module determines whether the unique ID needs to be delayed according to the remaining delay time calculated by the time window calculation module based on the status code returned by the uniqueness verification module;
[0012] The state cache module uses two independent Redis keys to determine the access status of the unique ID in the time window, which is used to separate the unique ID status from the timestamp logic;
[0013] The dynamic script hot loading module is used to detect the valid status of the Lua script stored in Redis called by the uniqueness verification module, and determine whether to automatically reload the Lua script based on the valid status of the Lua script.
[0014] Based on the above technical solution of the present invention, the following improvements can also be made:
[0015] Optionally, the status code returned by the uniqueness verification module is used to indicate one of three states: first submission, delayed processing, and repeated submission.
[0016] Optionally, when the delayed task scheduling module makes a judgment based on the returned status code, if the returned status code indicates delayed processing, the unique ID is added to the specified delay queue, and the remaining delay time required for the delay queue is configured based on the calculation result of the time window calculation module.
[0017] Optionally, the system further includes a service parameter configuration module, and the service parameter configuration module is used to configure queue parameters for different service queues.
[0018] Optionally, when the dynamic script hot loading module detects that the Lua script is in an invalid state, it automatically reloads the Lua script and records the number of reloads. When the number of reloads exceeds a set threshold, the set back or end command is executed.
[0019] According to a second aspect of the present invention, a method for collaboratively processing ID verification and delay scheduling based on a Redis script is provided, comprising the following steps:
[0020] After receiving the externally submitted unique ID, the Lua script stored in Redis is called to verify the unique ID and return a verification status code. The status code indicates whether the current status of the unique ID is one of the following: first submission, delayed processing, or repeated submission. The processing logic set according to the current status of the unique ID is as follows:
[0021] The status code indicates the current status of the unique ID. When the unique ID is submitted for the first time, it is only recorded;
[0022] When the status code indicates that the current status of the unique ID is delayed processing, the unique ID is added to the delay queue and retried by the system after a delay;
[0023] The state code indicates that the current state of the unique ID is duplicate submission, and the request is ignored.
[0024] Based on the above technical solutions of the application, the following improvements can be made:
[0025] Optionally, when the unique ID is added to the delay queue, the state and timestamp of the unique ID are updated according to the calculated remaining delay time.
[0026] Optionally, when the Redis server causes the Lua script to be invalid due to restart or cache invalidation, the system automatically reloads the Lua script, records the number of reloads, and generates a new SHA digest value for calling the reloaded Lua script, and when the number of reloads exceeds a set threshold, a set back or end command is executed.
[0027] Optionally, the state and timestamp of the unique ID are recorded by two independent Redis Keys, respectively, for logically separating the state and timestamp of the unique ID.
[0028] Optionally, the Lua script is executed in an atomic manner in Redis, ensuring no data race condition under concurrent requests.
[0029] The application has the beneficial effects that the application provides an ID verification and delay scheduling cooperative processing system and method based on Redis script, which can realize fine shunting of business logic, combine unique verification, timestamp calculation, state update, and delay queue scheduling into a single atomic operation, avoid race conditions, automatically reload the script if it is invalid and record the number of retries to ensure uninterrupted service, and can also ensure state isolation and improve system robustness. BRIEF DESCRIPTION OF DRAWINGS
[0030] Figure 1 The timing diagram of the distributed unique verification of the ID verification and delay scheduling cooperative processing system and method based on Redis script of the application embodiment.
[0031] Figure 2 The Redis unique judgment flowchart of the ID verification and delay scheduling cooperative processing system and method based on Redis script of the application embodiment. DETAILED DESCRIPTION
[0032] To make the purpose, technical solutions, and advantages of the present disclosure clearer, the present disclosure is further described in detail below with reference to specific embodiments and the accompanying drawings.
[0033] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0034] Example 1
[0035] like Figure 1 and Figure 2 As shown, one or more embodiments of the present application include a Redis script-based ID verification and delay scheduling collaborative processing system, including:
[0036] The uniqueness verification module is used to receive the unique ID submitted externally, verify the unique ID by calling the Lua script in Redis, and return the status code corresponding to the verification status;
[0037] The time window calculation module is used to calculate the remaining delay time based on the current system timestamp and the timestamp of the historical records;
[0038] The delayed task scheduling module determines whether the unique ID needs to be delayed according to the remaining delay time calculated by the time window calculation module based on the status code returned by the uniqueness verification module;
[0039] The state cache module uses two independent Redis keys to determine the access status of the unique ID in the time window, which is used to separate the unique ID status from the timestamp logic;
[0040] The dynamic script hot loading module is used to detect the valid status of the Lua script stored in Redis called by the uniqueness verification module, and determine whether to automatically reload the Lua script based on the valid status of the Lua script.
[0041] Among them, in this embodiment, the status code returned by the uniqueness verification module is used to indicate one of the three states of first submission, delayed processing and repeated submission. When the delayed task scheduling module makes a judgment based on the returned status code, if the returned status code indicates delayed processing, the unique ID is added to the specified delay queue, and the remaining delay time required for the delay queue is configured based on the calculation result of the time window calculation module. The system also includes a business parameter configuration module, which is used to configure queue parameters for different business queues. When the dynamic script hot loading module detects that the Lua script is in an invalid state, it automatically reloads the Lua script and records the number of reloads. When the number of reloads exceeds the set threshold, the set back or end command is executed.
[0042] The delay scheduling module uses Redisson's delay queue mechanism and supports multiple business queue identifiers (queueCode) to achieve scenario isolation. The state cache module combines the target ID with the queueKey and existsKey naming prefixes to form a unique Redis key-value pair, achieving logical separation of state and timestamp. The dynamic script hot reloading module uses the Lua script's SHA1 digest verification mechanism to check script validity and automatically reload and update the digest value when a "No matching script" error is detected.
[0043] Example 2
[0044] This embodiment provides an ID verification and delay scheduling collaborative processing system based on Redis scripts, including: a uniqueness verification service module (UniqueIdCheckService) for receiving a unique ID submitted externally, calling a Redis script to perform status judgment on the ID, which is divided into three types: first submission (0), delayed processing (1), and repeated submission (2).
[0045] The Redis client module (RedissonClient) calls Redis through the interface provided by Redisson, executes Lua scripts, and supports mechanisms such as script caching, SHA digest execution, and failure fallback.
[0046] The delay queue management module (RedisDelayQueueUtil) adds the ID to the corresponding business queue when the ID needs to be delayed (status code is 1), and configures the queue delay time, queue name, callback method, etc.
[0047] The business parameter configuration module (BusinessAutoProperties) supports configuring parameters such as prefixes and expiration times for different business queues to achieve dynamic support for multiple business scenarios.
[0048] Example 3
[0049] like Figure 1 and Figure 2 As shown, this embodiment provides a collaborative processing method for ID verification and delay scheduling based on Redis script, including the following steps:
[0050] After receiving the externally submitted unique ID, the Lua script stored in Redis is called to verify the unique ID and return a verification status code. The status code indicates whether the current status of the unique ID is one of the following: first submission, delayed processing, or repeated submission. The processing logic set according to the current status of the unique ID is as follows:
[0051] The status code indicates the current status of the unique ID. When submitted for the first time, the unique ID is only recorded;
[0052] When the status code indicates that the current status of the unique ID is delayed processing, the unique ID is added to the delay queue and retried by the system after a delay;
[0053] If the status code indicates that the unique ID is currently being submitted as a duplicate, the request is ignored.
[0054] Figure 1 This diagram illustrates how, in a distributed environment, the system completes uniqueness verification, status determination, and delayed task scheduling after a user submits a unique ID. The sequence diagram includes: the user requester; a distributed verification service (including a Redis server executing a Lua script); and the downstream business processing system. The diagram clearly demonstrates the system's differentiated handling of requests in three states: first, delayed, and duplicate.
[0055] Figure 2 This diagram illustrates the decision-making logic flow of a Lua script executed in Redis, including: loading and executing the Lua script; determining whether the queueKey exists; if the queueKey does not exist, determining whether the existsKey exists; if the existsKey also does not exist, writing the current timestamp; if the existsKey exists, calculating the remaining time and updating the queueKey; and returning different status codes for the Java business side to invoke processing logic (adding, adding to the delay queue, or discarding). This diagram highlights atomicity, multi-state judgment, and time window calculation mechanisms.
[0056] Specifically, when a unique ID is added to the delay queue, the status and timestamp of the unique ID are updated based on the calculated remaining delay time. When the Lua script becomes invalid due to a Redis server restart or cache failure, the system automatically reloads the Lua script, records the number of reloads, and generates a new SHA digest value for calling the reloaded Lua script. When the number of reloads exceeds the set threshold, the set fallback or end command is executed. The status and timestamp of the unique ID are recorded separately using two independent Redis keys to logically separate the status and timestamp of the unique ID. Lua scripts are executed atomically in Redis to ensure that there are no data race conditions under concurrent requests.
[0057] Specifically, the Lua script atomic execution process is as follows:
[0058] The Lua script uses 4 parameters: KEYS[1]: queue flag key (queueKey) of the ID to be processed; KEYS[2]: key to record the first submission timestamp (existsKey); KEYS[3]: current timestamp (in milliseconds); KEYS[4]: set uniqueness maintenance time (existTime).
[0059] Execution logic: If queueKey exists, {'2'} is returned, indicating that the ID is being processed. If existsKey exists, it indicates that the ID has been submitted again within the time window. The remaining expiration time is calculated and the queueKey is updated. If neither key exists, the current timestamp is recorded in existsKey and {'0'} is returned, indicating the first submission. This process is executed atomically, eliminating the need to worry about data overwriting or status corruption caused by concurrent access.
[0060] The business logic is diverted by the status code. The system processes the Lua script return value as follows:
[0061] Status code 0: represents the first submission, the ID is recorded but does not trigger subsequent business processing; Status code 1: represents the ID is repeated but recoverable, the ID is added to the delay queue, and the system will retry it periodically; Status code 2: indicates that the ID has been processed or submitted, and the system ignores the request to avoid duplicate processing.
[0062] The script hot reload mechanism: When a Redis server restarts or cache expires, Lua scripts may be lost. In this case, the system will catch a "No matching script" exception. The processing flow is as follows: re-call scriptLoad() to load the Lua script and generate a new SHA digest; update the local shaDigest string; throw a specific exception to prompt the business party to retry or downgrade; if the number of failures exceeds a threshold (such as 1), the system returns a failure, avoiding an infinite loop. This mechanism improves system robustness and is particularly suitable for scenarios such as complex networks and Redis master-slave switchover.
[0063] In addition, during the execution of the Redis Lua script, the EXISTS instruction is used to determine whether queueKey exists, and the GET instruction is used to obtain existKey and determine whether it exists. After completing the calculation of the delay time, the SET instruction is used to complete the status update.
[0064] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0065] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.
[0066] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system that is implemented in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0068] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0069] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.
Claims
1. ID verification and delay scheduling collaborative processing system based on Redis script, characterized by: include: The uniqueness verification module is used to receive the unique ID submitted externally, verify the unique ID by calling the Lua script in Redis, and return the status code corresponding to the verification status; The time window calculation module is used to calculate the remaining delay time based on the current system timestamp and the timestamp of the historical records; The delayed task scheduling module determines whether the unique ID needs to be delayed according to the remaining delay time calculated by the time window calculation module based on the status code returned by the uniqueness verification module; The state cache module uses two independent Redis keys to determine the access status of a unique ID in a time window; The dynamic script hot loading module is used to detect the valid status of the Lua script stored in Redis called by the uniqueness verification module, and determine whether to automatically reload the Lua script based on the valid status of the Lua script.
2. The ID verification and delay scheduling collaborative processing system based on Redis script according to claim 1 is characterized in that: The status code returned by the uniqueness verification module is used to indicate one of the three states: first submission, delayed processing, and repeated submission.
3. The ID verification and delay scheduling collaborative processing system based on Redis script according to claim 2 is characterized in that: When the delayed task scheduling module makes a judgment based on the returned status code, if the returned status code indicates delayed processing, the unique ID is added to the specified delay queue, and the remaining delay time required for the delay queue is configured based on the calculation result of the time window calculation module.
4. The ID verification and delay scheduling collaborative processing system based on Redis script according to claim 1 is characterized in that: The system further comprises a service parameter configuration module, and the service parameter configuration module is used to configure queue parameters for different service queues.
5. The ID verification and delay scheduling collaborative processing system based on Redis script according to claim 1 is characterized in that: When the dynamic script hot reloading module detects that the Lua script is in an invalid state, it automatically reloads the Lua script and records the number of reloads. When the number of reloads exceeds a set threshold, the set back or end command is executed.
6. The ID verification and delay scheduling collaborative processing method based on Redis script is characterized by: include: After receiving the externally submitted unique ID, the Lua script stored in Redis is called to verify the unique ID and return a verification status code. The status code indicates whether the current status of the unique ID is one of the following: first submission, delayed processing, or repeated submission. The processing logic set according to the current status of the unique ID is as follows: The status code indicates the current status of the unique ID. When the unique ID is submitted for the first time, it is only recorded; When the status code indicates that the current status of the unique ID is delayed processing, the unique ID is added to the delay queue and retried by the system after a delay; If the status code indicates that the unique ID is currently being submitted as a duplicate, the request is ignored.
7. The ID verification and delay scheduling collaborative processing method based on Redis script according to claim 6 is characterized in that: When a unique ID is added to the delay queue, the status and timestamp of the unique ID are updated according to the calculated remaining delay time.
8. The ID verification and delay scheduling collaborative processing method based on Redis script according to claim 6 is characterized in that: When a Lua script becomes invalid due to a Redis server restart or cache failure, the system automatically reloads the Lua script, records the number of reloads, and generates a new SHA digest value for calling the reloaded Lua script. When the number of reloads exceeds the set threshold, the set fallback or end command is executed.
9. The ID verification and delay scheduling collaborative processing method based on Redis script according to claim 7, characterized in that: The unique ID state and timestamp are recorded separately through two independent Redis keys to logically separate the unique ID state from the timestamp.
10. The ID verification and delay scheduling collaborative processing method based on Redis script according to claim 7, characterized in that: Lua scripts are executed atomically in Redis, ensuring no data race conditions under concurrent requests.