A cache database time jump awareness and data expiration strategy automatic repair method

Abstract

The application discloses a cache database time jump awareness and data expiration strategy automatic repairing method, which comprises the following steps: when detecting a time jump, calculating a shard index to which a primary key key belongs according to a configured total number N of primary key shards; starting a refresh thread according to a configured number of concurrent refreshes, and each thread only refreshes a TTL key-value pair in one shard each time; after the current shard is refreshed, the thread is automatically switched to the next to-be-processed shard until N shards are processed.

Description

Technical Field

[0001] This invention belongs to the field of distributed caching database technology, specifically relating to a method for detecting time jumps in caching databases and automatically repairing data expiration strategies. Background Technology

[0002] Caching databases are an indispensable component of modern distributed systems, used to store frequently accessed data to alleviate pressure on backend databases and improve system response speed. TTL (Time To Live) is the core mechanism of caching databases, used to set the expiration time of key-value pairs. When data exceeds the TTL setting, the system automatically marks it as expired and deletes it.

[0003] In mainstream caching databases such as Redis and Memcached, the TTL mechanism is typically implemented based on system timestamps. When a client sets a key-value pair with an expiration time, the server records the current timestamp plus the TTL value as the expiration time. A background expired key cleanup task periodically scans and deletes key-value pairs whose current timestamps have exceeded their expiration time.

[0004] A time jump refers to a discontinuous change in the server system time, mainly including the following two situations:

[0005] (1) Time jump forward, the system time is manually or automatically adjusted to a future point in time, such as through NTP (Network Time Protocol) synchronization, manual modification of the system time, etc.

[0006] (2) Time jumps backward, the system time is manually or automatically adjusted to a certain point in the past, such as time reset after system restart, NTP synchronization callback, etc.

[0007] Time jumps can delay, shorten, or even expire the TTL (Time-To-Live) lifecycle of data in the cached database, leading to the following serious problems:

[0008] (1) Cache breakdown. When a large amount of data expires prematurely due to time jump, a large number of requests will directly penetrate the cache and reach the backend database, causing a sudden increase in database pressure.

[0009] (2) Cache avalanche. When a large amount of cached data expires at the same time, or when a batch of data expires due to a time jump, it will cause system-level cache failure and make the service unavailable.

[0010] (3) Data inconsistency. When the time jump is made, data that should have expired continues to exist, which may cause the business to read expired or invalid data.

[0011] Existing caching database implementations (such as Redis and Memcached) do not provide mechanisms for detecting and repairing time jumps. While NTP services can synchronize time, they cannot detect or repair TTL exceptions caused by time jumps. Some application-layer solutions add time validation logic to the business code, but this approach is highly intrusive, complex to implement, and cannot guarantee coverage of all scenarios. Summary of the Invention

[0012] Purpose of the invention: To address the problem that existing cache databases cannot detect and repair TTL exceptions caused by time jumps, this invention proposes a method for TTL lifecycle awareness and automatic repair of cache databases in time jump scenarios.

[0013] Technical solution: In the first aspect, this invention proposes a method for automatic repair of cache database time jump awareness and data expiration strategy, including:

[0014] When a time jump is detected, the shard index to which the primary key belongs is calculated based on the configured total number of primary key shards N;

[0015] The refresh thread is started according to the configured number of concurrent refreshes. Each thread refreshes only the key-value pairs with TTL in one shard at a time. After the current shard is refreshed, the thread automatically switches to the next shard to be processed until N shards are processed.

[0016] Secondly, this invention proposes a method for automatic repair of cache database time jump awareness and data expiration strategy, including:

[0017] When the system starts up, N independent sharding hash tables are created based on the configured total number of primary key shards N; each sharding hash table is configured with an independent read-write lock, and the lock granularity is refined from a global lock to a sharding-level lock.

[0018] When a data insertion or access request for the primary key is received, two layers of hash lookup are performed sequentially:

[0019] First-level hash: Based on the hash value of the primary key, determine the target shard hash table to which the primary key belongs;

[0020] Second-level hashing: Within the target shard hash table, the storage location of the primary key within the bucket of the target shard hash table is determined based on the hash value of the primary key.

[0021] After the location is completed, only the read-write lock corresponding to the target shard hash table is acquired and held to complete the data insertion or access operation;

[0022] When a system time jump is detected, M refresh threads are started according to the configured concurrent refresh quantity M, where M is an integer greater than or equal to 1 and less than or equal to N;

[0023] Each refresh thread performs a refresh operation on only one key-value pair with TTL in a sharded hash table at a time, and holds only the read-write lock corresponding to that sharded hash table during the refresh operation.

[0024] After the current sharded hash table is refreshed, the refresh thread automatically switches to the next sharded hash table to be processed, until all N sharded hash tables have been processed.

[0025] Furthermore, the target shard hash table to which the primary key belongs is determined according to the following formula:

[0026] shard = hash(key) % N

[0027] Where hash(key) represents calculating the hash value of the primary key; N is the total number of primary key shards; and shard represents the shard hash table to which the primary key belongs.

[0028] Furthermore, the storage location of the primary key within the bucket of the target sharded hash table is calculated as hash'(key) % M, where M is the number of buckets in the sharded hash table.

[0029] Furthermore, the following steps are followed to detect time jumps:

[0030] At the start of the detection period, record the current system timestamp T1;

[0031] Enter sleep mode according to the preset detection interval;

[0032] After hibernation ends, obtain the current system timestamp T2;

[0033] Calculate the expected timestamp T2' = T1 + Interval, where Interval represents the detection interval;

[0034] Calculate the time error delta = |T2 - T2'|, and compare the time error with a preset threshold; if delta <= the preset threshold, the system time is determined to be normal; if delta > the preset threshold, a time jump is determined to have occurred.

[0035] Furthermore, it is integrated into the cache database process, running as an internal component of the cache database and communicating with the cache database via the Redis protocol.

[0036] Furthermore, it is deployed as an external process to the cache database, communicating with the cache database via the Redis protocol.

[0037] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0038] At least one processor;

[0039] and a memory communicatively connected to the at least one processor;

[0040] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute a cache database time jump detection and automatic repair method for data expiration strategy.

[0041] Fourthly, the present invention proposes a computer-readable storage medium storing computer instructions, which are used to enable a processor to implement a cache database time jump detection and automatic repair method for data expiration strategies when executed.

[0042] Fifthly, the present invention proposes a computer program product, which includes a computer program that, when executed by a processor, implements a method for automatic repair of cache database time jump perception and data expiration strategy.

[0043] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0044] (1) The present invention can effectively prevent cache breakdown and cache avalanche caused by time jump, and ensure the stability and availability of the system in high concurrency scenarios.

[0045] (2) The present invention can accurately identify two types of time jumps, namely, time jumps before and after, through a dynamic timing detection mechanism. The detection accuracy can be flexibly configured through threshold parameters.

[0046] (3) The present invention is simple to implement and does not require modification of business code. It only needs to deploy the detection and repair module at the cache database level to take effect.

[0047] (4) This invention is applicable to various TTL-based caching database systems, including mainstream products such as Redis and Memcached.

[0048] (5) Users can customize parameters such as detection time interval, jump threshold, and concurrent refresh quantity according to business needs to achieve fine control.

[0049] (6) The hash sharding strategy supports horizontal scaling and can flexibly adjust the number of shards according to the data size to adapt to cache clusters of different sizes. Attached Figure Description

[0050] Figure 1 This is a diagram illustrating the impact of time jumps on TTL expiration.

[0051] Figure 2 This is an architecture diagram of a cache database time jump detection and automatic data expiration strategy repair method proposed in this invention;

[0052] Figure 3 Here is a flowchart for time jump detection;

[0053] Figure 4 This is a schematic diagram of the hash sharding refresh strategy. Detailed Implementation

[0054] Example 1:

[0055] To achieve accurate and efficient detection of system time jump events, including time jumps before and after, and to automatically repair all key-value pairs with TTL after a time jump is detected to avoid cache breakdown and cache avalanche, and to avoid excessive latency impact on business requests when refreshing a large number of TTL key-value pairs, this invention proposes a method for cache database time jump perception and automatic data expiration policy repair, including the following steps:

[0056] Step 1: Employ a dynamic timing detection mechanism to detect time jumps.

[0057] As one implementation method, this step includes:

[0058] S100: At the start of the detection cycle, record the current system timestamp T1.

[0059] S110: Enter sleep mode according to the preset detection interval.

[0060] S120: After hibernation ends, obtain the current system timestamp T2.

[0061] S130: Calculate the expected timestamp according to the formula T2' = T1 + Interval.

[0062] S140: Calculate the time error delta = |T2 - T2'| and compare it with the preset threshold (Threshold).

[0063] S150: If delta <= Threshold, the system time is considered normal; if delta > Threshold, a time jump has occurred.

[0064] Step 2: Upon detecting a time jump, to avoid cache avalanche, a hash sharding strategy is used to refresh key-value pairs with TTL in batches.

[0065] As one implementation method, this step includes:

[0066] S200: When the system starts, it initializes two key parameters according to the configuration parameters: the total number of primary key shards N and the number of concurrent refreshes (Concurrent Shards). N represents the number of logical shards into which all key-value pairs are divided; Concurrent Shards represents the number of threads actually started during concurrent refresh, with each thread responsible for processing one shard, and switching to process the next shard after processing.

[0067] S210: Calculate the shard index to which the primary key belongs using the following formula:

[0068] shard = hash(key) % N

[0069] Where hash(key) represents calculating the hash value of the primary key; N is the total number of shards for the primary key; and the calculated result shard is the shard number to which the key should be stored.

[0070] S220: Concurrency Control: The system starts refresh threads according to the number configured in Concurrent Shards. Each thread refreshes only the key-value pairs with TTL in one shard at a time. After the current shard is refreshed, the thread automatically switches to the next shard to be processed until all N shards have been processed. By limiting the number of shards participating in the refresh at the same time, cache avalanche caused by the simultaneous expiration or reconstruction of all key-value pairs is avoided.

[0071] Step 3: Perform a two-level hash transformation on the original single-threaded hash table structure to achieve multi-threaded concurrency safety.

[0072] As an implementation approach, considering that native Redis uses a single-threaded model and manages all data through a single global hash table, directly introducing multi-threaded access would pose thread safety issues. Therefore, this step modifies its storage structure as follows:

[0073] S300: When the system starts, it initializes the total number of shards N and the number of concurrent connections ConcurrentShards according to the configuration parameters, and creates N independent sub-hash tables (sharded hash tables); each sub-hash table is configured with an independent lock (such as a mutex lock or a read-write lock), and the lock granularity is refined from a global lock to a shard-level lock.

[0074] S310: First-level hash location (shard location): When inserting or accessing a primary key, first calculate hash(key), and then determine which sub-hash table the primary key falls into according to the formula shard = hash(key) % N.

[0075] S320: Second-level hash location (bucket location): Within the determined sub-hash table, the hash of the primary key is calculated again (e.g., hash'(key) % M, where M is the number of buckets in the sub-hash table) to locate the specific storage location in the sub-hash table and perform insertion, query, or deletion operations.

[0076] S330: Multi-threaded concurrent execution: The system starts worker threads according to Concurrent Shards, and different threads can operate on different sub-hash tables in parallel; when a thread operates on a certain sub-hash table, it only holds an independent lock for that shard and does not block each other; after the same thread finishes processing the current shard, it automatically switches to the next shard until all shards have been processed.

[0077] S340: Coordination with step 2: Since key-value pairs are naturally distributed in independent sub-hash tables according to shards, when step 1 detects a time jump triggering a TTL batch refresh, the refresh operation can be performed independently at the shard granularity. Each time, only the corresponding sub-hash table needs to be locked, without global locking. This further reduces the blocking of normal read and write requests during the refresh process while achieving multi-threaded concurrency, and avoids cache avalanche.

[0078] In this embodiment of the invention, the primary key is stored and retrieved through two layers of hashing (first locating the shard, then locating the position within the bucket). Under the premise of ensuring uniform data distribution, the scope of lock contention is reduced from the global scope to a single shard, thereby realizing the multi-threaded concurrent security transformation of the caching system.

[0079] Example 2:

[0080] This invention uses a Redis caching database as an example to describe the technical solution proposed in Embodiment 1. The implementation environment includes: a single-machine Redis instance, a detection and repair module, and a configuration center (for dynamically adjusting parameters).

[0081] This invention provides two deployment methods for users to choose from based on their actual business scenarios:

[0082] (1) Embedded mode. The embedded mode directly integrates the detection and repair proposed in Example 1 into the cache database process, and runs as an internal component of the database.

[0083] Table 2 Features of Embedded Mode

[0084]

[0085] (2) External call mode. The external call mode deploys the detection and repair proposed in Example 2 as an independent process and communicates with the cache database through the Redis protocol.

[0086] Table 3 Characteristics of External Call Patterns

[0087]

[0088] The embodiments of the present invention include the following steps:

[0089] Step 1: Select the deployment method (embedded mode or external call mode) according to business needs.

[0090] Step 2: Deploy the detection and repair module and establish a connection with the Redis instance.

[0091] Step 3: Configure parameters such as detection interval, jump threshold, and number of concurrent shards according to the business scenario.

[0092] Step 4: Start the timed detection task to begin periodic time jump detection.

[0093] Step 5: When a time jump is detected, the TTL refresh process is automatically triggered.

[0094] Step 6: Monitor the refresh progress, record logs, and resume normal detection cycles after the repair is complete.

[0095] Upon detecting a time jump event, this embodiment of the invention can automatically trigger the TTL refresh process. The embedded mode, due to direct access to the memory data structure, has higher refresh efficiency; while the external call mode, although slightly less efficient, offers better flexibility and maintainability. Both modes effectively prevent cache avalanche and ensure stable system operation.

[0096] The embodiments of this invention are currently optimized primarily for single-machine Redis instances. For Redis cluster mode, detection and repair need to be deployed independently on each node, with time jump detection and TTL refresh operations being independent of each node. Future development can extend this to a cluster coordination mode, enabling cross-node time jump consistency detection.

Claims

1. A method for automatic repair of cache database time jump detection and data expiration strategy, characterized in that: include: When a time jump is detected, the shard index to which the primary key belongs is calculated based on the configured total number of primary key shards N; N is an integer greater than 1; The refresh thread is started according to the configured number of concurrent refreshes. Each thread refreshes only the key-value pairs with TTL in one shard at a time. After the current shard is refreshed, the thread automatically switches to the next shard to be processed until N shards are processed.

2. A method for automatic repair of cache database time jump detection and data expiration strategy, characterized in that: include: When the system starts, N independent shard hash tables are created based on the configured total number of primary key shards N. Each of the aforementioned sharded hash tables is configured with an independent read-write lock, with the lock granularity refined from a global lock to a shard-level lock; When a data insertion or access request for the primary key is received, two layers of hash lookup are performed sequentially: First-level hash: Based on the hash value of the primary key, determine the target shard hash table to which the primary key belongs; Second-level hashing: Within the target shard hash table, the storage location of the primary key within the bucket of the target shard hash table is determined based on the hash value of the primary key. After the location is completed, only the read-write lock corresponding to the target shard hash table is acquired and held to complete the data insertion or access operation; When a system time jump is detected, M refresh threads are started according to the configured concurrent refresh quantity M, where M is an integer greater than or equal to 1 and less than or equal to N; Each refresh thread performs a refresh operation on only one key-value pair with TTL in a sharded hash table at a time, and holds only the read-write lock corresponding to that sharded hash table during the refresh operation. After the current sharded hash table is refreshed, the refresh thread automatically switches to the next sharded hash table to be processed, until all N sharded hash tables have been processed.

3. The method for automatic repair of cache database time jump detection and data expiration strategy according to claim 2, characterized in that: The target shard hash table to which the primary key belongs is determined using the following formula: shard = hash(key) % N Where hash(key) represents calculating the hash value of the primary key; N is the total number of primary key shards; and shard represents the shard hash table to which the primary key belongs.

4. The method for automatic repair of cache database time jump detection and data expiration strategy according to claim 2, characterized in that: The storage location of the primary key within the bucket of the target sharded hash table is calculated as hash'(key) % M, where M is the number of buckets in the sharded hash table.

5. A method for automatic repair of cache database time jump detection and data expiration strategy according to claim 1 or 2, characterized in that: To detect time jumps, follow these steps: At the start of the detection period, record the current system timestamp T1; Enter sleep mode according to the preset detection interval; After hibernation ends, obtain the current system timestamp T2; Calculate the expected timestamp T2' = T1 + Interval, where Interval represents the detection interval; Calculate the time error delta = |T2 - T2'|, and compare the time error with a preset threshold; if delta <= the preset threshold, the system time is determined to be normal; if delta > the preset threshold, a time jump is determined to have occurred.

6. The method for automatic repair of cache database time jump detection and data expiration strategy according to claim 1 or 2, characterized in that: It is integrated into the cache database process and runs as an internal component of the cache database, communicating with the cache database via the Redis protocol.

7. A method for automatic repair of cache database time jump perception and data expiration strategy according to claim 1 or 2, characterized in that: Deployed as an external process to the cache database, it communicates with the cache database via the Redis protocol.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the cache database time jump perception and data expiration strategy automatic repair method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method for automatic repair of cache database time jump perception and data expiration strategy according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a method for automatic repair of cache database time jump perception and data expiration strategy as described in any one of claims 1-7.

Images