Cache key management method and system based on non-blocking traversal and storage medium

Abstract

The application provides a cache key management method and system based on non-blocking traversal and a storage medium. The method comprises the following steps: receiving a key management request, wherein the request at least contains a pattern matching expression of a key; establishing a connection with a cache database, performing incremental key retrieval based on a non-blocking traversal command; preprocessing a key set obtained through retrieval to obtain a candidate key set; performing a deletion operation on the candidate key set according to a predetermined strategy; during the retrieval and deletion process, performing dynamic flow control and abnormality processing based on runtime indexes; closing the traversal process, outputting an execution result and reporting monitoring data. Full-link incremental traversal based on SCAN avoids KEYS blocking; batch size, two-stage deletion and dynamic back pressure reduce business jitter; pipelining and UNLINK asynchronous release improve deletion concurrency; idempotent execution, failure retry and white list prefix prevent mistaken deletion; shard traversal / cluster iterator naturally adapts to clusters.

Description

Technical Field

[0001] This invention belongs to the field of distributed cache governance technology, specifically relating to a cache key management method, system, and storage medium based on non-blocking traversal. Background Technology

[0002] In large-scale online systems, operations and troubleshooting often require aggregation queries and cleanup of Redis keys that satisfy keywords or prefixes. Examples include periodic convergence of notification keys (such as "notify:*"), gray-scale data reclamation, and offline troubleshooting. The traditional KEYS command is a blocking O(N) operation, which can easily cause latency jitter or even avalanche in large key spaces. Although SCAN is a non-blocking incremental traversal, it still faces practical challenges in engineering, such as cursor management, batch parameters, network round trips, duplicate results, deletion blocking, failure retries, and cluster coverage. At the same time, directly executing synchronous DEL on a large number of keys may cause excessive overhead for the main thread to release data, and there are risks of concurrent writes and accidental deletions. Summary of the Invention

[0003] This invention provides a cache key management method, system, and storage medium based on non-blocking traversal; it offers a key fuzzy retrieval and batch deletion scheme that balances throughput, controllability, and security without blocking Redis, forming an integrated implementation capability of the method, system, and medium.

[0004] To achieve the above objectives, one or more embodiments of this application provide a cache key management method, system, and storage medium based on non-blocking traversal, the method comprising: Receive a key management request, the request containing at least a pattern matching expression for the key; Establish a connection to the cache database and perform incremental key retrieval based on non-blocking traversal commands; The retrieved key set is preprocessed to obtain a candidate key set; Perform a deletion operation on the candidate key set according to a predetermined strategy; During the retrieval and deletion process, dynamic flow control and anomaly handling are performed based on runtime metrics; Close the traversal process, output the execution results, and report the monitoring data.

[0005] Based on the above technical solution of the present invention, the following improvements can also be made: Optionally, the step of performing incremental key retrieval based on non-blocking traversal commands includes: Use cursor-based traversal commands to perform batch key retrieval of the cache database; The number of keys in a single retrieval is dynamically adjusted based on the real-time performance metrics of the cache database.

[0006] Optionally, the deletion operation on the candidate key set adopts a two-stage deletion strategy, including: In the first stage, candidate keys are marked as to be deleted or have a short lifespan. In the second stage, after the safe time window has passed, the final deletion operation is performed on the keys that are still marked.

[0007] Optionally, the deletion operation is executed in a batch pipeline manner, and the number of keys processed in a single batch is a configurable parameter.

[0008] Optionally, the exception handling includes retrying the key press exponential backoff strategy for failed operations and setting a maximum limit on the number of retries.

[0009] Optionally, the cache database is a distributed cache database cluster, and the method further includes the following steps: Identify each shard node in the distributed cache database cluster; Each shard node is connected to and the incremental key retrieval and deletion operations are performed on each shard node in turn to achieve key space coverage of the entire cluster.

[0010] Optionally, the step of performing the incremental key retrieval for each shard node is implemented by calling a client iterator that supports cluster traversal. The client iterator can automatically handle node routing and redirection, enabling seamless traversal of the entire cluster.

[0011] Optionally, the method also includes a hotkey protection mechanism, which implements a skip deletion or delayed deletion strategy for keys that have been accessed frequently recently.

[0012] According to another aspect of the present invention, a cache key management system based on non-blocking traversal is provided, comprising: The request parsing module is used to receive and parse key management requests; The traversal retrieval module is used to perform non-blocking incremental key retrieval; The data processing module is used to preprocess the search results; The secure deletion module is used to perform deletion operations according to a predetermined strategy; The flow control module is used for dynamic adjustment based on runtime metrics. The monitoring and reporting module is used to output execution results and report monitoring data; The modules work together to implement any of the methods described above.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0014] The beneficial effects of this invention are that it provides a cache key management method, system, and storage medium based on non-blocking traversal, which has the following advantages: non-blocking, with the entire link based on SCAN incremental traversal to avoid KEYS blocking; controllability, with batch size, two-stage deletion, and dynamic backpressure to reduce business jitter; high throughput, with pipelined and UNLINK asynchronous release to improve deletion concurrency; reliability, with idempotent execution, failure retries, and whitelist prefixes to prevent accidental deletion; and scalability, with sharded traversal / cluster iterators, naturally adaptable to clusters. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention; Figure 2 This is a schematic diagram of the module architecture of an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0017] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in one or more embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0018] like Figures 1-2 As shown, one or more embodiments of this application disclose a cache key management method, system, and storage medium based on non-blocking traversal. The method includes: Receive a key management request, the request containing at least a pattern matching expression for the key; The method establishes a connection with the cache database and performs incremental key retrieval based on non-blocking traversal commands. The steps of incremental key retrieval based on non-blocking traversal commands include: performing batch key retrieval on the cache database using cursor-based traversal commands; and dynamically adjusting the number of keys retrieved per retrieval based on the real-time performance metrics of the cache database. The cache database is a distributed cache database cluster. The method further includes the following steps: identifying each shard node in the distributed cache database cluster; establishing a connection with each shard node; and sequentially performing the incremental key retrieval and deletion operations on each shard node to achieve key space coverage across the entire cluster. The step of performing the incremental key retrieval on each shard node is implemented by calling a client iterator that supports cluster traversal. This client iterator can automatically handle node routing and redirection, achieving seamless traversal of the entire cluster.

[0019] The retrieved key set is preprocessed to obtain a candidate key set; The candidate key set is deleted according to a predetermined strategy. This deletion operation employs a two-stage strategy: a first stage, setting a deletion flag or a short-lived lifespan for each candidate key; and a second stage, after a safe time window, performing a final deletion operation on keys still marked. The deletion operation is executed in a batch pipeline manner, with the number of keys processed per batch being a configurable parameter. The method also includes a hotkey protection mechanism, implementing a skip deletion or delayed deletion strategy for keys with high recent access frequency.

[0020] During the retrieval and deletion process, dynamic flow control and exception handling are performed based on runtime metrics; the exception handling includes retrying key presses with an exponential backoff strategy for failed operations, and setting a maximum limit on the number of retries.

[0021] Close the traversal process, output the execution results, and report the monitoring data.

[0022] To further explain the specific technical solutions of the embodiments, the following embodiments are provided as examples: Example 1 This embodiment details a specific application of the present invention in an online operation and maintenance scenario, namely, to perform secure and efficient batch cleanup of a large number of notification keys in Redis that conform to a specific prefix (such as "notify:*").

[0023] Application Scenarios and Initialization Configuration: In the daily operation and maintenance of a large e-commerce platform, the system generates a large number of temporary notification keys daily (such as "notify:order:12345", "notify:user:67890"). These keys usually need to be cleaned up after the business logic is executed to release memory and keep the key space clean. Directly using the blocking KEYS"notify:*" command in a production environment poses a risk of causing service avalanche. Therefore, the non-blocking method of this invention is adopted for safe cleanup.

[0024] The initial configuration is as follows: Matching pattern: pattern="notify:*" Scan parameters: Initial single scan count = 1000, timeout = 60 seconds, maximum scan steps = 10000 (to prevent infinite loop).

[0025] Deletion strategy: Choose two-stage deletion to improve security. In the first stage, set a short TTL (e.g., 5 seconds) as a marker to be cleaned up. In the second stage, perform actual deletion after a delay window Δt=30 seconds.

[0026] Batch parameters: pipelined batch size batchSize=500.

[0027] Dynamic backpressure threshold: target_latency = 10 milliseconds. Dynamic backpressure is triggered when the P99 latency of a Redis instance exceeds this threshold.

[0028] Method execution steps Step S1: Schema Construction and Connection Establishment The application receives the cleanup task and constructs a matching pattern "notify:*". Then, it establishes a connection with the target Redis instance using a Redis client (such as Lettuce). Before execution, it verifies whether the "notify:" prefix is ​​in the whitelist of allowed operations.

[0029] Step S2: Non-blocking cursor incremental traversal Start a SCAN command loop with an initial cursor value of 0. Deserialize the byte keys returned by SCAN using a string serializer to ensure correct key name parsing. Remove null and duplicate keys from the list of keys returned by each SCAN operation, adding valid key names to the candidate key set (in this example, a HashSet is used for temporary storage). During the iteration, monitor the current Redis latency in real time. If the P99 latency exceeds 10 milliseconds, dynamically reduce the count value of the next SCAN operation using an adaptive algorithm (e.g., from 1000 to 500), and insert a short sleep (e.g., sleep_ms=10 milliseconds) before the next request to reduce the load on the Redis server.

[0030] Step S3: Candidate Key Aggregation This example only performs key name retrieval, so after traversal, a set of deduplicated candidate keys is obtained. The total number of candidate keys is recorded, and a batch number (batch_id) and a trace ID (trace_id) are generated for subsequent auditing.

[0031] Step S4: Two-stage secure deletion Phase 1 (Marking Phase): The candidate key set is processed in batches (500 keys per batch). For each batch of keys, a SETEXkey5 "pending_delete" command is sent to Redis using a pipeline to set a 5-second expiration mark for each key to be deleted. This operation completes quickly and almost does not block the main thread.

[0032] Phase Two (Actual Deletion Phase): Wait a 30-second safety window. After that, process these keys in batches again. For each batch of keys, use a Lua script to atomically perform the following operations: Check if the "pending_delete" flag on the key exists and if its value is correct.

[0033] If the tag exists, execute the UNLINK command to delete the key asynchronously (to avoid blocking caused by synchronous release of large keys).

[0034] If the tag does not exist or the key has been modified by another client, the deletion is skipped to ensure that newly written data is not accidentally deleted.

[0035] This Lua script guarantees the atomicity of tag verification and deletion operations.

[0036] Step S5: Exception Handling and Retry During the traversal or deletion process, if a network timeout or command execution failure occurs, the failed key is added to the retry queue. The retry strategy uses exponential backoff: the first retry waits 1 second, the second 2 seconds, the third 4 seconds, with a maximum of 3 retries. Keys exceeding the maximum number of retries are recorded in the failure list and an alarm is triggered, notifying operations personnel for manual intervention.

[0037] Step S6: Statistical Reporting and Resource Cleanup After the cursor traversal is complete, close the Redis connection. Collect and output the execution statistics for this task, including: Total number of keys scanned, number of candidate keys matched, number of keys successfully marked in the first stage, number of keys successfully deleted in the second stage, number of keys that failed to be deleted and their list, and the time taken for the task to complete at P95 / P99.

[0038] These metrics are integrated with the Prometheus and Grafana monitoring systems via Micrometer. Simultaneously, detailed execution records, including batch_id and trace_id, are output to the application logs for auditing and troubleshooting.

[0039] Through the implementation of this embodiment, the system can safely and smoothly complete the batch cleanup task of "notify:*" prefix keys in a production environment Redis instance (with a key space of tens of millions). The entire process does not cause significant server-side latency fluctuations, the risk of accidental deletion is extremely low, and it effectively improves operational efficiency and system reliability. This embodiment demonstrates the feasibility and superiority of the method of the present invention in a real-world scenario.

[0040] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0042] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0044] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0045] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A cache key management method based on non-blocking traversal, characterized in that, Includes the following steps: Receive a key management request, the request containing at least a pattern matching expression for the key; Establish a connection to the cache database and perform incremental key retrieval based on non-blocking traversal commands; The retrieved key set is preprocessed to obtain a candidate key set; Perform a deletion operation on the candidate key set according to a predetermined strategy; During the retrieval and deletion process, dynamic flow control and anomaly handling are performed based on runtime metrics; Close the traversal process, output the execution results, and report the monitoring data.

2. The cache key management method based on non-blocking traversal according to claim 1, characterized in that, The steps for incremental key retrieval based on non-blocking traversal commands include: Use cursor-based traversal commands to perform batch key retrieval of the cache database; The number of keys in a single retrieval is dynamically adjusted based on the real-time performance metrics of the cache database.

3. The cache key management method based on non-blocking traversal according to claim 1, characterized in that, The deletion operation on the candidate key set adopts a two-stage deletion strategy, including: In the first stage, candidate keys are marked as to be deleted or have a short lifespan. In the second stage, after the safe time window has passed, the final deletion operation is performed on the keys that are still marked.

4. The cache key management method based on non-blocking traversal according to claim 1, characterized in that, The deletion operation is executed using a batch pipelined approach, and the number of keys processed in a single batch is a configurable parameter.

5. The cache key management method based on non-blocking traversal according to claim 1, characterized in that, The exception handling includes retrying the key press exponential backoff strategy for failed operations, and setting a maximum limit on the number of retries.

6. The cache key management method based on non-blocking traversal according to claim 1, characterized in that, The cache database is a distributed cache database cluster, and the method further includes the following steps: Identify each shard node in the distributed cache database cluster; Each shard node is connected to and the incremental key retrieval and deletion operations are performed on each shard node in turn to achieve key space coverage of the entire cluster.

7. The cache key management method based on non-blocking traversal according to claim 6, characterized in that, The incremental key retrieval step for each shard node is implemented by calling a client iterator that supports cluster traversal. The client iterator can automatically handle node routing and redirection, enabling seamless traversal of the entire cluster.

8. The cache key management method based on non-blocking traversal according to claim 1, characterized in that, The method also includes a hotkey protection mechanism, which implements a skip deletion or delayed deletion strategy for keys that have been accessed frequently recently.

9. A cache key management system based on non-blocking traversal, characterized in that, include: The request parsing module is used to receive and parse key management requests; The traversal retrieval module is used to perform non-blocking incremental key retrieval; The data processing module is used to preprocess the search results; The secure deletion module is used to perform deletion operations according to a predetermined strategy; The flow control module is used for dynamic adjustment based on runtime metrics. The monitoring and reporting module is used to output execution results and report monitoring data; The modules cooperate with each other to implement the method described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images