A method of Redis-based distributed lock and cache

Abstract

The application discloses a Redis-based distributed lock and cache method. The application solves the problem that a first program cannot normally set a result or other application programs repeatedly execute after a lock expires due to the first program running for too long by creating two threads or processes, one for setting the lock and the other for adding an expiration time to the lock during the running of the first process after the first setting of the lock succeeds. Then, the set method provided by the official library of redis is used in the first thread, which is different from the setnx method used at present. The set method has other two parameters, ex and nx, which means that the expiration time can be set at the same time as setnx, and the situation of deadlock that may occur when two commands are executed is avoided. After the program is executed, the lock is not deleted, but the data in the lock is changed, and other programs take data according to the form of the value corresponding to the key.

Description

Technical Field

[0001] This invention relates to the field of backend development technology, and in particular to a method for distributed locking and caching based on Redis. Background Technology

[0002] With rapid technological advancements, increasing data volumes, and the growing prevalence of distributed systems, applications are often deployed across multiple machines (multi-nodes). In some scenarios, to ensure data integrity, it's crucial that the same task runs on only one node at a time—meaning a method can only be executed by one thread at a time. In multi-machine deployments, different machines run different processes, necessitating thread safety within a multi-process environment. Therefore, distributed locks have emerged.

[0003] Currently, the optimal approach is based on Redis, using locking: executing `SETNX`, and if successful, then executing `EXPIRE` to add the expiration time. In this approach, `SETNX` and `EXPIRE` are executed in two steps, which are non-atomic operations. If `SETNX` succeeds but `EXPIRE` fails, a deadlock may occur. Therefore, solutions using Lua scripts, such as Redisson, have emerged. However, this data is a Java package, and if based on other languages, Lua scripts still need to be implemented, which is relatively cumbersome. Furthermore, the results are typically stored in another Redis library, not making efficient use of library space. Existing methods have the following shortcomings:

[0004] 1) Directly executing two steps based on Redis may result in a deadlock;

[0005] 2) Implementing it yourself based on Lua scripts is relatively troublesome;

[0006] 3) Set different Redis databases for locks and caches.

[0007] Therefore, existing technologies have shortcomings and need to be improved. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a method for distributed locking and caching based on Redis, which effectively reduces the cost of understanding and development, requires no third-party libraries, and improves the degree of customization.

[0009] The technical solution of the present invention is as follows: a method for distributed locks and caches based on Redis is provided, including the following steps.

[0010] S1: Create two threads or processes, one to set the lock, and the other to add an expiration time to the lock during the first process after the first one has successfully set the lock.

[0011] S2: The application generates a unique ID and a unique cache key.

[0012] S3: Read cached data. Query the data in Redis using the unique cache key generated in step S2. If cached data exists, check if it starts with '{'. If it does, it's normal cached data, and the data is returned directly. If it doesn't start with '{', it means another application has set it, and step S3 is repeated for 2-15 seconds. If no cached data exists, proceed to step S4 to set the cache lock.

[0013] S4: Set a cache lock. If no cached value was read in step S3, it means there is no cached data and no application is running. Use the `set` method to set the lock; the key is the unique cache key from step S2, and the value is the unique ID from step S2. Set the `nx` parameter to `true`. This method returns `true` on success and `false` on failure. This can be used to determine if the application's cache lock was set successfully. If the lock is set successfully, the normal program will begin execution. If the lock fails, it means another application preemptively set the lock. In this case, the application should wait 2-15 seconds before executing steps S3 and S4.

[0014] S5: After step S4 is executed successfully, that is, after the cache lock is set successfully, the normal program starts to run. After the normal program runs, the result needs to be set into the cache data. At this time, the thread or process that extends the expiration time in step S6 is stopped, and then the result is set into Redis.

[0015] S6: Open multiple threads or processes within the same application, using the same unique cache key and unique ID. After the first program successfully executes step S4, continuously query the data and update the expiration time, indicating that the program is still running normally; stop the thread or process directly after the program finishes executing.

[0016] This is achieved by creating two threads or processes: one for setting the lock, and the other, which, after the first thread successfully sets the lock, adds an expiration time to the lock during the first thread's execution. This solves the problem of the first program running for too long, causing the lock to expire and preventing the setting of results, or other applications finding no lock and executing repeatedly. The first thread uses the `set` method provided by the Redis official library. Unlike the existing `setnx` method, the `set` method has two additional parameters, `ex` and `nx`. This means that the expiration time can be set simultaneously with `setnx`, avoiding the deadlock that might occur when executing two separate commands. After the program finishes execution, the lock is not deleted; instead, the data in the lock is modified. Other programs will then retrieve the data based on the value corresponding to this key.

[0017] Furthermore, in step S6, this thread or process also performs the judgment in step S3 to determine whether the condition starts with '{' to decide whether to extend the time. If the data setting is successful, the expiration time will not be adjusted.

[0018] Furthermore, after the application generates a unique ID, it determines whether the lock is set by this application based on the value corresponding to the set Redis key. If it is a lock set by this application, the data in the lock is modified to the cached value; otherwise, the operation is canceled.

[0019] Furthermore, the expiration time can be extended based on the unique ID. By assigning the unique ID to another thread or process, this application can extend the expiration time of data whose value is this ID while it is executing its normal program. This prevents situations where the current application is still running but the unique cache key has expired, causing other programs to reset the unique cache key and run, resulting in a mismatch when querying the unique cache key's ID after the program ends.

[0020] Furthermore, the unique ID of an application is usually generated using the time sequence of the UUID, which ensures that different processes at the same time are basically not duplicated.

[0021] Furthermore, a unique cache key is generated based on one or more fields, which is used by applications to read specified data based on certain conditions, i.e., the dimension of the cache.

[0022] Furthermore, the unique cache key is set based on the user's ID and the existing method category, so data for this user and category can be retrieved from Redis by using the user ID and method category.

[0023] Furthermore, in step S6, the result data after executing the program is converted into a JSON string and returned to Redis.

[0024] By adopting the above scheme, the present invention provides a method for distributed locking and caching based on Redis, which has the following technical effects:

[0025] 1. Avoid lock expiration to reduce the risk of deadlock.

[0026] 2. The same set of logic was used to implement both distributed lock and data caching functions, reducing the overhead of deleting and resetting locks.

[0027] 3. Flexible caching dimensions and data read judgment.

[0028] 4. It solves the problem of repeated execution on multiple machines in a distributed environment. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] Please see Figure 1 This invention provides a method for distributed locks and caching based on Redis, comprising the following steps.

[0032] S1: Create two threads or processes, one to set the lock, and the other to add an expiration time to the lock during the first process after the first one has successfully set the lock.

[0033] S2: The application generates a unique ID and a unique cache key.

[0034] S3: Read cached data. Query the data in Redis using the unique cache key generated in step S2. If cached data exists, check if it starts with '{'. If it does, it's normal cached data, and the data is returned directly. If it doesn't start with '{', it means another application has set it, and step S3 is repeated every 2-15 seconds. In this embodiment, step S3 is repeated every 5 seconds. If no cached data exists, proceed to step S4 to set the cache lock.

[0035] S4: Set a cache lock. If no cached value was read in step S3, it means there is no cached data and no application is running. Use the `set` method to set the lock; the key is the unique cache key from step S2, and the value is the unique ID from step S2. Set the `nx` parameter to `true`. This method returns `true` on success and `false` on failure. This can be used to determine if the application's cache lock was set successfully. If the lock is set successfully, the normal program begins execution. If the lock fails, it means another application preemptively set the lock. This application should wait 2-15 seconds before executing steps S3 and S4. In this embodiment, the application should wait 5 seconds before executing steps S3 and S4.

[0036] S5: After step S4 is executed successfully, that is, after the cache lock is set successfully, the normal program starts to run. After the normal program runs, the result needs to be set into the cache data. At this time, the thread or process that extends the expiration time in step S6 is stopped, and then the result is set into Redis.

[0037] S6: Open multiple threads or processes within the same application, using the same unique cache key and unique ID. After the first program successfully executes step S4, continuously query the data and update the expiration time, indicating that the program is still running normally; stop the thread or process directly after the program finishes executing.

[0038] In step S6, this thread or process also performs the judgment in step S3 to determine whether the condition starts with '{' to decide whether to extend the time. If the data setting is successful, the expiration time will not be adjusted.

[0039] In this embodiment, after the application generates a unique ID, it determines whether the lock is set by this application based on the value corresponding to the set Redis key. If it is a lock set by this application, the data in the lock is modified to the cached value; otherwise, the operation is canceled.

[0040] In this embodiment, the expiration time is extended based on a unique ID. By assigning the unique ID to another thread or process, this application can allow another thread or process to extend the expiration time of data whose value is this ID while executing normal programs. This prevents the occurrence of locks that are not set by this application.

[0041] In this embodiment, the unique ID of the application is usually generated using the time sequence of the UUID, which can ensure that different processes at the same time are basically not duplicated.

[0042] In this embodiment, a unique cache key is generated based on one or more fields, which is used by the application to read the cache of specified data according to certain conditions, i.e., the dimension of the cache.

[0043] In this embodiment, the unique cache key is set based on the user's ID and the existing method category, so that data for this user and category can be retrieved from Redis by user ID + method category.

[0044] In this embodiment, in step S6, the result data after executing the program is converted into a JSON string and returned to Redis.

[0045] Based on the above processes, we have implemented a distributed runtime lock and data caching service for third-party calls, as detailed in the following example.

[0046] This service uses a web interface, allowing external users to call third-party services and obtain results without needing to interface with third parties. Furthermore, in a distributed environment, this service ensures that multiple calls to third parties do not result in duplicate charges, and the results will be cached for a specified period of time.

[0047] 1. This service first determines which third-party interface to call based on the incoming request parameter (merchant). For example, the third-party interface WhatsAppDetection is used to detect whether a certain mobile number has been registered with WhatsApp. WhatsAppDetection is the parameter (merchant).

[0048] 2. When this service calls the corresponding third-party interface based on the parameter (merchant), it will generate a unique ID. We use a time-based UUID to ensure that there will be virtually no duplicate IDs on different machines at different times.

[0049] 3. Generate a unique cache key, which is the condition for caching and setting a distributed lock. For example, the parameter required by WhatsAppDetection above is the user's mobile phone number. We should use WhatsAppDetection plus the user's mobile phone number to form a unique cache key. This can ensure that the distributed lock and cache will play a role when subsequent requests repeatedly request the same vendor's interface and this mobile phone number. Assuming the user's mobile phone number is 12345678, the unique cache key is WhatsAppDetection_12345678, that is, using the vendor and user as dimensions.

[0050] 4. Before calling the WhatsAppDetection interface, first query the key in Redis, namely WhatsAppDetection_12345678. If an empty field (i.e., null or None) is returned, it means that no call has been made before, and the logic of setting a distributed runtime lock is executed (step S5). If the value returned does not start with '{', it means that this third-party is being executed by another process and the result has not yet been cached. In this case, you should proceed to (step S6) and wait for a specified time before querying again. If the result returned starts with '{', it means that another program has previously called WhatsAppDetection for this phone number and written cached data. You can directly convert this result into a JSON string and return it.

[0051] The following Python 3.10 code contains steps S4, S5, and S6.

[0052] def read_cache(self,interval=5,ex=180):

[0053] while 1:

[0054] if(data:=redis_cli.get(self.key))and data.startswith('{'):

[0055] return json.loads(data)

[0056] elif not redis_cli.set(self.key,self._identifier,ex=ex,nx=True):

[0057] time.sleep(interval)

[0058] else:

[0059] break

[0060] 5. If there is no call identical to WhatsAppDetection_12345678, the current process can set the run lock. The run lock is set using set(nx=True). You can check if the lock setting was successful. If it is successful, you can exit the loop to execute the third-party call. If it fails, it means that another process has set the run lock first. The current process should wait for a period of time before executing again (step S4).

[0061] 6. If other processes are executing the same third-party call (WhatsAppDetection_12345678), the current service should not execute it again, because executing it again will call it a second time, which will lead to duplicate billing. It should wait for other processes to complete execution and write to the cache before querying the cache data (execute step S4), or execute the call itself after other processes time out. Therefore, an infinite loop and a period of sleep are used here to wait.

[0062] 7. Execute the third-party call. Here, the return result of the read_cache method (step S4) is used as the judgment condition. If the read_cache return result is empty, it means that the lock was successfully set, and the actual third-party call is performed. After the execution is completed, the result is set to the cache. If the return result is not empty, it must be the actual cached result data, and it is returned directly.

[0063] The Python 3.10 code is as follows:

[0064]

[0065]

[0066] 8. Extending the expiration time is done in a separate thread to prevent the Redis key from expiring due to excessive program execution time. This would cause other processes to execute the current call repeatedly after finding an empty value in Redis in step S4.

[0067] In summary, this invention provides a method for distributed locking and caching based on Redis, which has the following technical effects:

[0068] 1. Avoid lock expiration to reduce the risk of deadlock.

[0069] 2. The same set of logic was used to implement both distributed lock and data caching functions, reducing the overhead of deleting and resetting locks.

[0070] 3. Flexible caching dimensions and data read judgment.

[0071] 4. It solves the problem of repeated execution on multiple machines in a distributed environment.

[0072] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for distributed locking and caching based on Redis, characterized in that, Includes the following steps: S1: Create two threads or processes, one to set the lock, and the other to add an expiration time to the lock during the first process after the first thread successfully sets the lock. S2: The application generates a unique ID and a unique cache key; S3: Read cached data and query the data in Redis based on the unique cache key generated in step S2; If there is cached data, check if it starts with '{'; if it starts with '{', it means it is normal cached result data, and the data is returned directly; if it does not start with '{', it means that other applications have set it, and start repeating step S3 every 2-15 seconds. If there is no cached data, proceed to step S4 to set the cache lock; S4: Set a cache lock. If no cache value is read in step S3, it means there is no cached data and no application is running. Use the set method to set the lock. The key is the unique cache key of step S2, and the value is the unique ID of step S2. The nx parameter is set to true. In this way, it will return true if the setting is successful and false if the setting fails. You can use this to determine whether the cache lock of this application is set successfully. If the setting is successful, the normal program will start. If the setting fails, it means that another application has set the cache lock first. In this case, the application should wait for 2 to 15 seconds before executing steps S3 and S4. S5: After step S4 is executed successfully, that is, after the cache lock is set successfully, the normal program starts to run. After the normal program runs, the result needs to be set into the cache data. At this time, the thread or process that extends the expiration time in step S6 is stopped, and then the result is set into Redis. S6: Open multiple threads or processes within the same application, using the same unique cache key and unique ID. After the first program successfully executes step S4, continuously query the data and update the expiration time, indicating that the program is still running normally; stop the thread or process directly after the program finishes executing.

2. The method for distributed locking and caching based on Redis according to claim 1, characterized in that, In step S6, this thread or process also performs the judgment in step S3 to determine whether the condition starts with '{' to decide whether to extend the time. If the data setting is successful, the expiration time will not be adjusted.

3. The method for distributed locking and caching based on Redis according to claim 1, characterized in that, After the application generates a unique ID, it determines whether the lock is set by this application based on the value corresponding to the set Redis key. If it is a lock set by this application, the data in the lock is modified to the cached value; otherwise, the operation is canceled.

4. The method for distributed locking and caching based on Redis according to claim 3, characterized in that, Extending expiration time based on a unique ID allows the application to extend the expiration time of data whose value is this ID by assigning the unique ID to another thread or process while the application is executing its normal program.

5. A method for distributed locking and caching based on Redis according to claim 1, characterized in that, Application unique IDs are usually generated using the time sequence of UUIDs, which ensures that different processes at the same time are unlikely to duplicate each other.

6. The method for distributed locking and caching based on Redis according to claim 1, characterized in that, A unique cache key is generated based on one or more fields. It is used by applications to read specified data based on certain conditions, i.e., the dimension of the cache.

7. A method for distributed locking and caching based on Redis according to claim 6, characterized in that, The unique cache key is set based on the user's ID and the existing method category, so that data for this user and category can be retrieved from Redis by user ID + method category.

8. A method for distributed locking and caching based on Redis according to claim 1, characterized in that, In step S6, the result data after executing the program is converted into a JSON string and returned to Redis.

Images