Redis multi-data-source dynamic switching method

Abstract

The invention relates to a Redis multi-data-source dynamic switching method, which comprises the following steps of: when a system is started, reading and analyzing an external configuration file in a YAML / Properties format by a dynamic configuration analysis module, and constructing configuration data after acquiring a Redis instance connection parameter and a connection pool parameter; in business code execution, when a method with a user-defined annotation is called, an annotation driving interception module triggers and extracts a data source name to a registration center to search for a corresponding instance, the data source name is bound to a current thread context through ThreadLocal, Redis methods such as get and set are called through a unified operation interface module, the thread context is cleaned after operation is completed, and the corresponding instance is stored in the registration center. If abnormity occurs in the whole process, standardized abnormity is thrown out, and detailed error information is recorded for debugging. According to the method, through annotation driving, dynamic configuration analysis and a thread security isolation mechanism, the multi-data source management process is simplified, and the flexibility and maintainability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems and data storage, and in particular to a Redis multi-data source dynamic switching method. Background Art

[0002] In today's microservices architecture, the complexity of business operations often requires systems to connect to multiple Redis (a remote dictionary server, a memory-based key-value storage system) instances. For example, to achieve database isolation, data from different business modules may be stored in different Redis instances. In multi-environment deployments, development, testing, and production environments each require independent Redis instances. However, traditional approaches to this problem present numerous problems.

[0003] First, there's a significant hard-coded dependency. Developers directly specify the Redis instance in their code, tightly tying business logic to the data source. Changes to Redis instance information, such as the address and port, require modifications to a large amount of business code, increasing development costs and introducing new errors.

[0004] Secondly, configuration redundancy is a prominent issue. Each Redis instance requires separate configuration of connection parameters, including the host address, port number, password, and various connection pool parameters such as the maximum number of connections, the minimum number of idle connections, and the connection timeout. As the number of Redis instances increases, the configuration files become lengthy and complex, significantly increasing the difficulty of maintenance and the risk of configuration errors.

[0005] Finally, dynamic switching is inefficient and poses security risks. Traditional dynamic switching requires manual management of the connection pool, a process that can easily lead to thread safety issues. For example, when multiple threads compete for connection resources, data inconsistencies may occur. Manual management can also easily lead to resource leaks, impacting system stability and performance.

[0006] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0007] In response to the defects in the existing technology, the purpose of the present invention is to provide a Redis multi-data source dynamic switching method, which simplifies the multi-data source management process and improves the system flexibility and maintainability through annotation drive, dynamic configuration parsing and thread safety isolation mechanism.

[0008] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0009] A Redis multi-data source dynamic switching method, characterized by comprising the following steps:

[0010] During the system startup phase, the dynamic configuration parsing module reads the external configuration file;

[0011] The dynamic configuration parsing module automatically parses the configuration file content according to the grammatical rules to obtain the connection parameters of each Redis instance and the connection pool configuration parameters;

[0012] Build connection pool configuration data based on connection parameters and connection pool configuration parameters;

[0013] During the execution of business code, when a method with a custom annotation is called, the annotation-driven interception module is triggered;

[0014] The annotation-driven interception module first extracts the data source name from the annotation, and then searches the registry for the Redis instance with the corresponding name.

[0015] After finding the corresponding Redis instance, the annotation-driven interception module binds the found instance to the current thread context with the help of the thread local variable ThreadLocal;

[0016] When performing Redis data operations, the corresponding Redis operation methods are called through the unified operation interface module;

[0017] When the Redis data operation is completed, the unified operation interface module clears the current thread context information;

[0018] During the entire data processing process, if an abnormal situation occurs, the system will throw a standardized exception and record detailed error information.

[0019] Based on the above technical solution, when building the connection pool configuration data, the dynamic configuration parsing module will check the validity of the parameters. If the parameters are invalid, the system will throw a corresponding exception and prompt the user to check the configuration file.

[0020] On the basis of the above technical solution, the configuration file includes an immediate use flag;

[0021] Based on the value of the immediate use flag, for instances that need to be used immediately, the dynamic configuration parsing module will immediately create and initialize the corresponding Redis connection pool instance based on the parsed connection parameters and connection pool configuration parameters;

[0022] For instances that are not marked for immediate use, the system only records their configuration information and performs initialization operations when the instance is actually needed for business, thus achieving lazy loading.

[0023] Based on the above technical solution, the custom annotation specifies that the annotation can only be used on the method through the meta-annotation @Target(ElementType.METHOD), and specifies that the annotation is valid at runtime through @Retention(RetentionPolicy.RUNTIME);

[0024] The custom annotation defines a value attribute to specify the data source name.

[0025] Based on the above technical solution, create an AOP aspect to intercept methods marked with custom annotations.

[0026] Based on the above technical solution, the specific steps of AOP interception marked with custom annotations include:

[0027] Define the aspect class and use the @Around annotation to specify the intercepted method expression;

[0028] In the aspect's around notification method, first get the custom annotation on the target method and get the data source name;

[0029] According to the data source name, obtain the corresponding Redis connection pool or RedisTemplate instance from the registration center;

[0030] Use thread-local variables to bind the obtained Redis instance to the current thread for use in subsequent business operations.

[0031] Based on the above technical solution, the unified operation interface module will verify the permissions of the current thread before calling the Redis operation method to ensure that the thread has the operation permission for the target Redis instance; if the permission is insufficient, the system throws an insufficient permission exception and records the relevant access information.

[0032] Based on the above technical solution, when performing Redis data operations, the unified operation interface module will dynamically adjust the priority of the operation based on the operation type and the load of the target Redis instance; for high-load Redis instances, read operations are executed before write operations to ensure the overall performance of the system.

[0033] Based on the above technical solution, the system will regularly check the connection status of Redis instances during operation. If a connection abnormality is found in a Redis instance, the system will automatically mark the instance as unavailable and temporarily redirect the business requests using the instance to the backup Redis instance, while recording the abnormality information and redirection operation log.

[0034] Based on the above technical solution, the dynamic configuration parsing module supports encryption of the configuration file when reading it; before parsing, the configuration file is first decrypted using a preset key to ensure the security of the Redis instance connection information.

[0035] The method for dynamically switching multiple Redis data sources described in the present invention has the following beneficial effects:

[0036] 1. The dynamic configuration parsing module implements automatic loading and management of multiple Redis instances. The annotation-driven interception module implements "zero-intrusion" data source switching through custom annotations and AOP technology. The thread-safe context module uses ThreadLocal to ensure operational consistency in a multi-threaded environment. The unified operation interface module provides developers with a simple and easy-to-use calling interface.

[0037] 2. It solves the three major problems of complex configuration, inefficient switching, and thread safety in Redis multi-data source scenarios. The modular design meets industrial-grade application standards. In actual applications, it can greatly improve the maintainability and execution efficiency of the system and is suitable for high-concurrency fields such as e-commerce, finance, and the Internet of Things.

[0038] 3. Developers only need to add custom annotations to business methods to easily switch data sources. Compared with traditional methods, in some scenarios, it can reduce 80% or even more redundant configuration code, significantly improving development efficiency and reducing development difficulty.

[0039] 4. Through dynamic connection pool management and parameter optimization, the latency of Redis operations is reduced (in some scenario tests, the latency of Redis operations is reduced by 30%-50%, which is suitable for high-concurrency scenarios), which can better meet the system performance requirements in high-concurrency scenarios.

[0040] 5. Supports dynamic addition or removal of Redis instances at runtime. Only configuration files need to be modified without changing the code, meeting the needs of system elastic scaling and improving system flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention has the following accompanying drawings:

[0042] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0043] Figure 1 Flowchart of Example 1 of a Redis multi-data source dynamic switching method described in the present invention. DETAILED DESCRIPTION

[0044] The present invention will be described in further detail below with reference to the accompanying drawings. The detailed description, which is provided for illustrative purposes only and includes various details to aid understanding of the embodiments of the present invention, should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.

[0045] like Figure 1 As shown, the present invention provides a Redis multi-data source dynamic switching method, comprising the following steps:

[0046] During the system startup phase, the dynamic configuration parsing module reads the external configuration file;

[0047] Exemplarily, the configuration file supports YAML (YAML Ain't Markup Language, a concise, easy-to-read and write data serialization format, commonly used for configuration files and data exchange) or Properties (a simple key-value pair configuration file format in the Java platform, with a .properties file extension). The following is an example of a configuration file in YAML format:

[0048]

[0049]

[0050] The meanings of the parameters or terms in this example are as follows:

[0051] redis: represents the root node of the Redis multi-data source configuration. The dynamic configuration parsing module parses all data source definitions under this node.

[0052] datasources: defines a collection of multiple Redis data sources. Each child node (such as ds1 and ds2) represents an independent Redis instance.

[0053] ds1, ds2: data source names, uniquely identifying different Redis instances. In annotation-driven switching, @RedisTemplateName("ds1") is used to specify the instance to use;

[0054] The following basic connection parameters are also included in the example:

[0055] host: the host address of the Redis server. The dynamic configuration parsing module needs to read this parameter to establish a connection.

[0056] port: The port number of the Redis server. The default value is 6379. Usually, each instance must be explicitly specified.

[0057] password: The access password of the Redis server, which is part of the connection parameters and is used for authentication to ensure the security of the data source;

[0058] The example also includes the following connection pool configuration (pool):

[0059] max-active: The maximum number of active connections in the connection pool. The "Connection Pool Intelligent Optimization" module will dynamically adjust resources based on this parameter to avoid connection exhaustion.

[0060] max-idle: The maximum number of idle connections, which is the upper limit of the number of connections the connection pool retains when idle, to reduce the overhead of frequent connection creation;

[0061] min-idle: The minimum number of idle connections, ensuring that a minimum number of connections are always available, improving the response speed of sudden requests;

[0062] max-wait: The maximum waiting time (in milliseconds) to obtain a connection from the connection pool. If a connection is not obtained within the time limit, an exception will be thrown to prevent the thread from being blocked for a long time.

[0063] The dynamic configuration parsing module automatically parses the configuration file (for example, reading and parsing the configuration file line by line), parses it according to the syntax rules (referring to the syntax rules of YAML or Properties), and obtains the connection parameters of each Redis instance and the connection pool configuration parameters;

[0064] Exemplarily, the connection parameters include: host address, port, password;

[0065] Exemplarily, the connection pool configuration parameters include: maximum number of connections max-active, maximum number of idle connections max-idle, minimum number of idle connections min-idle, and maximum waiting time max-wait;

[0066] Build connection pool configuration data based on connection parameters and connection pool configuration parameters;

[0067] During the execution of business code, when a method with a custom annotation is called, the annotation-driven interception module is triggered;

[0068] The annotation-driven interception module first extracts the data source name from the annotation, and then searches the registry for the Redis instance with the corresponding name.

[0069] For example, taking the custom annotation @RedisTemplateName as an example, when a method with this annotation is called, the annotation-driven interception module starts working; if the method is marked with @RedisTemplateName("ds1"), the module will obtain the data source name "ds1";

[0070] For example, the registry maintains information about all configured Redis instances. The registry is a Spring container or a custom instance registry.

[0071] After finding the corresponding Redis instance, the annotation-driven interception module uses the thread-local variable ThreadLocal to bind the found instance to the current thread context. During the subsequent execution of the current thread, all Redis-related operations will be performed based on this bound instance. The thread-local variable provides an independent variable copy for each thread, ensuring that data source binding operations between different threads do not interfere with each other, thereby ensuring thread safety.

[0072] When performing Redis data operations, the corresponding Redis operation methods are called through the unified operation interface module;

[0073] Exemplarily, the unified operation interface module provides common Redis operation methods, such as get for obtaining data, set for storing data, delete for deleting data, etc.

[0074] When calling these operation methods, the unified operation interface module will obtain the bound Redis instance from the current thread context. For example, when executing a get operation, it will first obtain the bound Redis instance from the thread local variable, and then call the corresponding method of the instance to perform the data read operation;

[0075] In a multi-threaded environment, thread-local variables ThreadLocal play a key role. Since each thread has its own independent copy of the thread-local variable, operations on the Redis instance by different threads are isolated from each other, effectively avoiding contention problems that may arise when multiple threads access concurrently, and ensuring thread safety.

[0076] When the Redis data operation is completed, the unified operation interface module clears the current thread context information. To prevent memory leaks, the unified operation interface module clears the Redis instance bound to the thread local variable to ensure that thread resources are released in a timely manner and system performance is not affected by long-term resource occupation.

[0077] During the entire data processing process, if an abnormal situation occurs, the system will throw a standardized exception and record detailed error information, making it easier for developers to quickly locate and solve the problem;

[0078] For example, when the annotation-driven interception module cannot find the Redis instance specified in the annotation in the registry, the system will throw a RedisDataSourceNotFoundException with specific error information, such as "Cannot find the Redis data source named ds1";

[0079] For example, when performing Redis data operations through the unified operation interface module, if a connection exception occurs, such as network interruption or Redis service unavailability, the system will throw a RedisConnectionException exception and explain the cause of the exception in detail, such as "A network failure occurred when connecting to Redis instance ds1."

[0080] In this embodiment, dynamic binding of data sources is achieved by declaring data sources with custom annotations. Business code does not need to pay attention to the connection and switching details of the underlying data source, truly achieving "zero-intrusion" switching and effectively reducing the coupling between business logic and data sources.

[0081] In this embodiment, based on the Spring Boot Starter mechanism, the system can automatically assemble multiple data sources. During the startup process, Redis instances are loaded on demand according to the configuration file, and unused instances are lazily initialized, reducing system resource usage and improving resource utilization efficiency.

[0082] In this embodiment, a dynamic connection pool parameter adaptation algorithm is built in to automatically optimize key connection pool parameters such as the maximum number of connections and timeout period according to the parameters in the configuration file, thereby improving the throughput of Redis operations to adapt to high-concurrency business scenarios.

[0083] In this embodiment, thread local variables (ThreadLocal) are used to isolate the data source context between different threads, ensuring that in asynchronous scenarios, the data source operations of each thread are independent and safe, avoiding thread safety issues caused by competition among multiple data sources, and enhancing system stability.

[0084] Based on the above technical solution, when building the connection pool configuration data, the dynamic configuration parsing module will check the validity of the parameters. If the parameters are invalid, the system will throw a corresponding exception and prompt the user to check the configuration file.

[0085] For example, when building a connection pool, the dynamic configuration parsing module automatically verifies the legitimacy of parameters (such as the maximum number of connections is a positive integer and the port number is within the valid range) through the built-in dynamic connection pool parameter adaptation algorithm, and optimizes the configuration according to business scenario requirements.

[0086] When writing configuration files, the connection parameters and connection pool configuration parameters must be set in accordance with the actual business scenario and system performance requirements. For example, for Redis instances corresponding to business modules with large concurrent access volumes, the maximum number of connections and the maximum number of idle connections can be appropriately increased to improve the system's concurrent processing capabilities.

[0087] On the basis of the above technical solution, the configuration file includes an immediate use flag;

[0088] Based on the value of the immediate use flag, for instances that need to be used immediately, the dynamic configuration parsing module will immediately create and initialize the corresponding Redis connection pool instance based on the parsed connection parameters and connection pool configuration parameters;

[0089] For instances that are not marked for immediate use, the system only records their configuration information and performs initialization operations when the instance is actually needed for business, thus achieving lazy loading.

[0090] For example, the eager-init parameter in the configuration file indicates whether instances should be initialized immediately. If true, the connection pool is automatically created at system startup; if false, initialization is delayed until the first use. This lazy loading mechanism avoids initializing all Redis instances at system startup, saving system resources and improving system startup speed.

[0091] Based on the above technical solution, the custom annotation specifies that the annotation can only be used on the method through the meta-annotation @Target(ElementType.METHOD), and specifies that the annotation is valid at runtime through @Retention(RetentionPolicy.RUNTIME);

[0092] The custom annotation defines a value attribute to specify the data source name.

[0093] For example, the design of custom annotations follows the interception rules of Spring AOP, ensuring the accuracy and reliability of dynamic switching by limiting method-level scope and runtime retention strategy. The code for defining the custom annotation @RedisTemplateName is as follows:

[0094] @Target(ElementType.METHOD)

[0095] @Retention(RetentionPolicy.RUNTIME)

[0096] public@interface RedisTemplateName{

[0097] String value();

[0098] }.

[0099] In the above code:

[0100] @Target(ElementType.METHOD) is a Java meta-annotation used to specify the target of a custom annotation. ElementType.METHOD indicates that the annotation can only be used at the method level. By restricting the @RedisTemplateName annotation to business methods, accurate AOP interception is ensured.

[0101] @Retention(RetentionPolicy.RUNTIME) is a Java meta-annotation that specifies the annotation's retention policy. RetentionPolicy.RUNTIME indicates that the annotation is retained at runtime and can be read through reflection. This ensures that annotation information (such as the data source name) can be dynamically retrieved at runtime, enabling the switching logic to be implemented.

[0102] @interface is a keyword for defining annotations in Java. A custom annotation is declared through the public@interface annotation name, that is, the @RedisTemplateName annotation is defined to mark the data source used by the method.

[0103] String value() defines a string attribute named value in a custom annotation. You can specify the attribute value using @RedisTemplateName("ds1") . The value attribute passes the data source name (such as ds1 or ds2) for AOP to resolve and bind to the corresponding instance.

[0104] Based on the above technical solution, create an AOP aspect to intercept methods marked with (with) custom annotations.

[0105] Before the method executes, the aspect retrieves the data source name specified in the annotation, retrieves the corresponding Redis instance from a registry (such as the data source instance registry maintained by the Spring container), and binds it to the current thread's context. This binding is implemented using ThreadLocal, ensuring data source isolation in a multi-threaded environment and preventing concurrency conflicts.

[0106] For example, the AOP aspect process specifically includes:

[0107] Intercept annotation method: Use @RedisTemplateName to mark the method that needs to dynamically switch the data source.

[0108] Bind the data source: Before the method is executed (@Before), obtain the Redis instance from the registration center and store it in the thread context.

[0109] Clean up the context: After the method is executed (@After), clear the thread context to prevent memory leaks.

[0110] Thread safety specifically includes:

[0111] Use ThreadLocal to ensure that each thread has independent data source binding to avoid multi-thread competition.

[0112] The registry (RedisTemplateRegistry) manages instances through a thread-safe ConcurrentHashMap.

[0113] Exception handling specifically includes:

[0114] If the data source is not found, a RedisDataSourceNotFoundException is thrown (a custom exception class is required).

[0115] Extensible exception handling logic (such as retry mechanisms or downgrade strategies).

[0116] Based on the above technical solution, the specific steps of AOP interception of methods marked with (with) custom annotations include:

[0117] Define the aspect class and use the @Around annotation to specify the intercepted method expression, for example, intercept all methods marked with the @RedisTemplateName annotation. You can also replace @Around with @Before as needed, for example, use @Before to bind the data source and use @After to clean up the context.

[0118] In the aspect's around notification method, first get the custom annotation on the target method and get the data source name;

[0119] According to the data source name, obtain the corresponding Redis connection pool or RedisTemplate instance from the registration center;

[0120] Use thread-local variables to bind the obtained Redis instance to the current thread for use in subsequent business operations.

[0121] Furthermore, after the Redis data operation is completed, the unified operation interface module cleans up the current thread context information. For example, after executing the business method, in the final logic of the surrounding notification (such as the finally block), the Redis instance binding in the thread local variable is cleaned up to ensure resource release.

[0122] Based on the above technical solution, the unified operation interface module will verify the permissions of the current thread before calling the Redis operation method to ensure that the thread has the operation permission for the target Redis instance; if the permission is insufficient, the system throws an insufficient permission exception and records the relevant access information.

[0123] In this embodiment, permission verification logic is inserted before the unified operation interface module (such as RedisOperationService) executes Redis operation methods (such as get and set). It is triggered when the data source has been bound to the current thread (via ThreadLocal) but no specific operation has been performed. It includes the following steps:

[0124] Step 1: Get the currently bound data source name (such as ds1) from the thread context;

[0125] Step 2: Query the preset permission policy (such as allowed roles, IP ranges, etc.) based on the data source name;

[0126] Step 3: Get the security context information of the current thread (such as user identity, role, IP address). For example, the security context information obtains the user role by integrating Spring Security's SecurityContextHolder, parses the client IP from the HttpServletRequest, and supports custom parser extensions.

[0127] Step 4: Compare the permission policy with the current context to decide whether to allow the operation;

[0128] Step 5: If the permission is insufficient, throw RedisAccessDeniedException and log it; otherwise, perform the operation normally.

[0129] The following schemes can be used to configure and load permission policies:

[0130] Expand the security policy parameters in the Redis data source configuration, for example:

[0131] redis:

[0132] datasources:

[0133] ds1:

[0134] host:192.168.1.100

[0135] security:

[0136] allowed-roles:[ADMIN,OPERATOR]#Allowed role list

[0137] allowed-ips:[192.168.1.0 / 24]#Allowed IP segment

[0138] ds2:

[0139] host:192.168.1.101

[0140] security:

[0141] allowed-roles:[OPERATOR];

[0142] This example defines a security access control policy for a Redis multi-data source, including two Redis instances (ds1 and ds2). Each instance is configured with connection information (host) and security rules (security), and access permissions to the Redis instances are restricted by roles and IP addresses.

[0143] redis: represents the global configuration root node of the Redis multi-data source.

[0144] datasources: defines a collection of multiple Redis data sources. Each child node (such as ds1 and ds2) represents an independent Redis instance.

[0145] The data source ds1 is configured as follows:

[0146] host: the host address of the Redis instance ds1;

[0147] security: Security policy configuration block, including the following sub-items:

[0148] allowed-roles: a list of roles that are allowed to access the instance;

[0149] The example value [ADMIN,OPERATOR] means that only users with the ADMIN or OPERATOR role can access ds1;

[0150] allowed-ips: The client IP address range allowed to access the instance;

[0151] The example value [192.168.1.0 / 24] allows access only from IP addresses 192.168.1.1 to 192.168.1.254.

[0152] The data source ds2 is configured as follows:

[0153] host: the host address of the Redis instance ds2;

[0154] security: Security policy configuration block, including the following sub-items:

[0155] allowed-roles: a list of roles that are allowed to access the instance;

[0156] The example value [OPERATOR] means that only users with the OPERATOR role can access ds2;

[0157] allowed-ips: If not explicitly configured, it means that there is no restriction on IP addresses (the default behavior may need to be determined based on the actual implementation). Furthermore, when allowed-ips is not explicitly configured, all IP addresses are allowed access by default. If strict restrictions are required, you can explicitly prohibit them by using deny-all-ips:true.

[0158] Policy loading is set as the dynamic configuration parsing module loading the security policy into memory (such as ConcurrentHashMap) during initialization; the permission rules of each data source are associated with its instance information (such as connection parameters) and stored.

[0159] Taking the example shown in the preceding code as an example, the system reads this configuration file when it starts up and loads the connection parameters and security policies of ds1 and ds2;

[0160] When a user attempts to access a Redis instance, the system obtains the current user's role and client IP from the security context, compares the allowed-roles and allowed-ips rules, and decides whether to allow access.

[0161] If the permission is insufficient, RedisAccessDeniedException is thrown and the log is recorded (such as user role, IP address, and target data source).

[0162] Based on the above technical solution, when performing Redis data operations, the unified operation interface module will dynamically adjust the priority of the operation according to the operation type (such as read operation, write operation) and the load of the target Redis instance; for high-load Redis instances, read operations are executed before write operations to ensure the overall performance of the system.

[0163] For example, the following modules are added:

[0164] Load monitoring module: collects load indicators of each Redis instance (such as CPU usage, memory usage, current number of connections, QPS, etc.) in real time to provide data support for dynamic scheduling.

[0165] Priority scheduling module: Dynamically adjusts the order of operation execution based on the operation type (read / write) and instance load. For example, read operations are prioritized during high load to alleviate the pressure of write operations.

[0166] Unified Operation Interface Module Extension: Based on the existing data source routing function, load awareness and priority scheduling logic are embedded, transparent to the business code. The business layer still calls redisService.get(key) or redisService.set(key, value) without being aware of the underlying scheduling strategy.

[0167] Specific implementation steps include:

[0168] Metrics collection: Obtain instance status periodically (e.g., every second) through Redis INFO commands or third-party monitoring tools (e.g., Prometheus);

[0169] Load calculation: Define the load scoring formula, for example:

[0170] Load score = CPU usage * 0.4 + memory usage * 0.3 + current connection ratio * 0.3

[0171] Classify the load level according to the score (such as low, medium, high);

[0172] The priority scheduling strategy is as follows:

[0173] Read operations (such as GET and HGET): marked as high priority to ensure user experience;

[0174] Write operations (such as SET, HSET): marked as normal priority, allowing appropriate delay;

[0175] The instance load level has different effects on the priority and limits of read and write operations. Specifically:

[0176] At low load levels, read operations have high priority and write operations have medium priority. At this time, both read and write operations can be performed normally without any restrictions.

[0177] At medium load levels, read operations remain prioritized, while write operations are prioritized to low. Write operations are delayed until the load decreases.

[0178] Under high load levels, read operations have the highest priority and write operations will be suspended. At this time, the system only allows read operations and write operations will be queued until the load situation improves.

[0179] In specific implementation, a priority queue can be maintained for each Redis instance to sort tasks according to operation type and load level; a thread pool can also be used to extract tasks from the queue according to priority for execution, and high-priority tasks (such as read operations) are given priority in allocating thread resources.

[0180] The following solutions can be used to link or cooperate with the above modules:

[0181] Based on the original dynamic configuration parsing module (responsible for loading Redis instance connection parameters and connection pool configuration), the parsing and initialization of priority scheduling policy parameters have been added to support load awareness and task scheduling functions.

[0182] Configuration parameter expansion: Add the following policy parameters to the Redis data source configuration (example):

[0183] ds1:

[0184] priority-scheduling:

[0185] read-weight:10#Read operation weight (the higher the value, the higher the priority)

[0186] write-weight:5#write operation weight

[0187] max-queue-size:100# Maximum task queue capacity (to prevent memory overflow);

[0188] read-weight / write-weight: defines the relative priority of read / write operations;

[0189] max-queue-size: limits the number of queued tasks for a single instance to avoid resource exhaustion;

[0190] When parsing the configuration file, the priority policy parameters are associated with the Redis instance information (such as host and port) and stored;

[0191] Use ConcurrentHashMap to cache policy parameters, with the key being the data source name (such as ds1) and the value being the policy object;

[0192] Create a priority task queue for each Redis instance, and implement weight-based sorting (such as a priority queue);

[0193] The existing connection parameter parsing logic remains unchanged, and new policy parameters are managed through independent fields to ensure backward compatibility. Instances without a priority policy are set to use the global policy (e.g., read-weight = 5, write-weight = 3) by default.

[0194] Based on the data source routing function of the original unified operation interface module (binding instances based on the @RedisTemplateName annotation), new load awareness and task scheduling logic are added to achieve dynamic adjustment of operation priorities. The specific steps include:

[0195] The load monitoring module collects the following metrics of the Redis instance in real time:

[0196] CPU usage: obtained through the Redis INFO command;

[0197] Memory usage: monitor the ratio of used_memory to maxmemory;

[0198] Current number of connections: count the number of active connections from the connection pool;

[0199] Queue waiting time: the average waiting time of a task in the queue;

[0200] Load level calculation:

[0201] Calculate the load score according to the preset formula (example):

[0202] Load score = CPU usage * 0.4 + memory usage * 0.3 + connection ratio * 0.3;

[0203] Rating by rating:

[0204] Low load: score <60;

[0205] Medium load: 60≤score<85;

[0206] High load: score ≥85;

[0207] Mark get, hget, etc. as read operations, and set, hset, etc. as write operations;

[0208] Read operations: When the load is low, the operations are executed according to the configured weight; when the load is high, the priority is automatically increased (for example, the weight value is temporarily doubled).

[0209] Write operations: Under high load, lower the priority or suspend enqueuing until the load recovers.

[0210] Each Redis operation is encapsulated as a RedisTask object, which includes the operation type, priority weight, execution logic, etc.

[0211] Dynamically calculate task priority based on the current instance load level and operation type;

[0212] Submit the task to the priority queue of the corresponding instance;

[0213] The background thread pool extracts tasks from the queue according to priority and executes them;

[0214] The business code does not need to be modified and can still be called through redisService.get(key) or redisService.set(key,value);

[0215] The original data source dynamic switching logic (based on ThreadLocal binding) remains unchanged, and priority scheduling is transparent to the business.

[0216] Based on the above technical solution, the system will regularly check the connection status of Redis instances during operation. If a connection abnormality is found in a Redis instance, the system will automatically mark the instance as unavailable and temporarily redirect the business requests using the instance to the backup Redis instance, while recording the abnormality information and redirection operation log.

[0217] Exemplarily, the connection status health check and automatic failover functions are implemented through the following modules;

[0218] Health check module: Periodically checks the connection status of the Redis instance (such as network connectivity and service responsiveness) and maintains the instance health status (available / unavailable).

[0219] Testing methods include:

[0220] Heartbeat detection (PING): Periodically sends a PING command to the Redis instance. If no PONG response is received within the timeout period, it is considered abnormal.

[0221] Command verification: Execute lightweight commands (such as INFO) to verify service availability and response latency.

[0222] The following examples show the detection frequency and threshold:

[0223] health-check:

[0224] interval:5000 #Detection interval (milliseconds)

[0225] timeout:1000 # single detection timeout

[0226] failure-threshold:3#The number of consecutive failures triggers the flag to be unavailable.

[0227] Failover module: When an abnormality is detected in the primary instance, it automatically routes requests to a predefined backup instance to ensure business continuity.

[0228] State management module: records the health status of the instance and supports manual or automatic recovery and re-enabling of the primary instance; for example, a memory cache (such as ConcurrentHashMap) is used to record the instance status, with the key being the data source name and the value being a Boolean type (true indicates health).

[0229] Add standby instance definitions and health check parameters (such as detection frequency and timeout) to the original data source configuration;

[0230] Before executing an operation, check the health status of the target instance and trigger the failover logic if it is unavailable.

[0231] The failover logic is implemented by explicitly defining the primary and backup relationships in the data source configuration. The routing policy is set as follows:

[0232] Priority routing: select the first healthy instance in the order of the standby instance list;

[0233] Load balancing routing: Random or round-robin selection from healthy standby instances.

[0234] The temporary redirection process includes:

[0235] The primary instance ds-primary is detected to be unavailable and its status is updated to unhealthy.

[0236] Select the first healthy backup instance from the backups list (for example, backup instance ds-backup1);

[0237] Route subsequent requests to ds-backup1 until ds-primary recovers.

[0238] The abnormal recovery mechanism is set as follows:

[0239] Automatic recovery detection: For instances marked as unavailable, retry the health check at a lower frequency (such as every 30 seconds). If it succeeds N times in a row (such as 3 times), it is marked as available and switched back to the primary instance.

[0240] Manual intervention: Provides a management interface (such as REST API) to force refresh instance status or trigger active / standby switchover.

[0241] Based on the above technical solution, the dynamic configuration parsing module supports encryption of the configuration file when reading it; before parsing, the configuration file is first decrypted using a preset key to ensure the security of the Redis instance connection information.

[0242] For example, encrypted fields: encrypt sensitive information in the Redis configuration (such as password, host, port), while other non-sensitive fields (such as connection pool parameters) can remain in plain text;

[0243] Select symmetric encryption (AES) for the encryption algorithm: Use the AES-256-GCM algorithm, which balances performance and security and supports data integrity verification after encryption.

[0244] The key is injected through environment variables (REDIS_CONFIG_KEY) or key management services (such as AWS KMS, HashiCorp Vault), and must not be hard-coded in code or configuration files.

[0245] Supports key rotation mechanism, automatically re-encrypting with a new key after the old key is decrypted.

[0246] The configuration file format is adjusted as follows:

[0247] Use !cipher to mark the encrypted field, and the dynamic configuration parsing module will automatically decrypt it after identification, for example:

[0248] redis:

[0249] datasources:

[0250] ds1:

[0251] host:"!cipher{AQIDBAUGBwgJCgsMDQ4PEBES}"

[0252] password:

[0253] "!cipher{EhITFBUWFxgZGhscHR4fICEiIw==}".

[0254] The decryption process of the dynamic configuration parsing module includes:

[0255] Read configuration files: load the original configuration content (YAML / Properties) and identify encrypted fields marked with !cipher{...};

[0256] Key acquisition: Get the key from a predefined security source (such as environment variables, KMS), and throw DecryptionKeyMissingException if not found;

[0257] Decryption operation: After decoding the encrypted field using Base64, decrypt it using AES-256-GCM to obtain the plaintext value;

[0258] Replace plaintext: Replace the decrypted value back into the configuration object. The subsequent process is the same as that for unencrypted configuration.

[0259] Key injection methods include: environment variables, such as REDIS_CONFIG_KEY = your_aes_256_key; startup parameters, such as redis.config.key = your_aes_256_key; Key Management Service (KMS), such as dynamically obtaining temporary keys from AWS KMS.

[0260] The following are specific examples.

[0261] Multi-tenant system: Assign independent Redis instances to different tenants. By adding the @RedisTemplateName annotation to the business method, the data of different tenants can be quickly isolated to ensure that the data between tenants does not interfere with each other.

[0262] Hybrid cloud architecture: In the Redis instance scenario across cloud vendors, the dynamic configuration parsing module is used to load the Redis instance configurations of different cloud vendors. Annotation-driven switching is used to achieve dynamic routing of Redis instances across cloud vendors, support disaster recovery switching, and improve system reliability and availability.

[0263] AB testing environment: Direct experimental traffic to a specific Redis instance. By adding different data source annotations to the business methods of different experimental groups, data bucket statistics are implemented to provide accurate data support for AB testing.

[0264] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0265] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by those skilled in the art based on the contents disclosed in the present invention should be included in the protection scope recorded in the claims.

Claims

1. A Redis multi-data source dynamic switching method, characterized in that: The steps include: During the system startup phase, the dynamic configuration parsing module reads the external configuration file; The dynamic configuration parsing module automatically parses the configuration file content according to the grammatical rules to obtain the connection parameters of each Redis instance and the connection pool configuration parameters; Build connection pool configuration data based on connection parameters and connection pool configuration parameters; During the execution of business code, when a method with a custom annotation is called, the annotation-driven interception module is triggered; The annotation-driven interception module first extracts the data source name from the annotation, and then searches the registry for the Redis instance with the corresponding name. After finding the corresponding Redis instance, the annotation-driven interception module binds the found instance to the current thread context with the help of the thread local variable ThreadLocal; When performing Redis data operations, the corresponding Redis operation methods are called through the unified operation interface module; When the Redis data operation is completed, the unified operation interface module clears the current thread context information; During the entire data processing process, if an abnormal situation occurs, the system will throw a standardized exception and record detailed error information.

2. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: When building the connection pool configuration data, the dynamic configuration parsing module will check the validity of the parameters. If the parameters are invalid, the system will throw a corresponding exception and prompt the user to check the configuration file.

3. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: The configuration file includes an immediate use flag; Based on the value of the immediate use flag, for instances that need to be used immediately, the dynamic configuration parsing module will immediately create and initialize the corresponding Redis connection pool instance based on the parsed connection parameters and connection pool configuration parameters; For instances that are not marked for immediate use, the system only records their configuration information and performs initialization operations when the instance is actually needed for business, thus achieving lazy loading.

4. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: The custom annotation specifies that the annotation can only be used on methods through the meta-annotation @Target(ElementType.METHOD), and specifies that the annotation is valid at runtime through @Retention(RetentionPolicy.RUNTIME); The custom annotation defines a value attribute to specify the data source name.

5. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: Create an AOP aspect to intercept methods marked with custom annotations.

6. A Redis multi-data source dynamic switching method as claimed in claim 5, characterized in that: The specific steps for AOP aspects to intercept methods marked with custom annotations include: Define the aspect class and use the @Around annotation to specify the intercepted method expression; In the aspect's around notification method, first get the custom annotation on the target method and get the data source name; According to the data source name, obtain the corresponding Redis connection pool or RedisTemplate instance from the registration center; Use thread-local variables to bind the obtained Redis instance to the current thread for use in subsequent business operations.

7. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: Before calling the Redis operation method, the unified operation interface module verifies the permissions of the current thread to ensure that the thread has the operation permissions for the target Redis instance; if the permissions are insufficient, the system throws an insufficient permissions exception and records the relevant access information.

8. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: When performing Redis data operations, the unified operation interface module will dynamically adjust the priority of the operation based on the operation type and the load of the target Redis instance. For high-load Redis instances, read operations are executed before write operations to ensure overall system performance.

9. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: During operation, the system will regularly check the connection status of the Redis instance. If a Redis instance connection anomaly is found, the system will automatically mark the instance as unavailable and temporarily redirect business requests using the instance to the backup Redis instance. At the same time, the system will record the exception information and redirection operation log.

10. A Redis multi-data source dynamic switching method according to claim 1, characterized in that: The dynamic configuration parsing module supports encryption of the configuration file when reading it; before parsing, the configuration file is first decrypted using a preset key to ensure the security of the Redis instance connection information.

Images