Redis chaos fault test method, storage medium and equipment
By binding Redis to a proxy layer and simulating faults using a fault strategy engine, the problem of existing tools interfering with the production environment is solved, achieving efficient and accurate Redis chaotic fault testing, suitable for black-box testing and load testing scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TELECOM DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-24
AI Technical Summary
Existing Redis chaos fault testing tools are prone to interfering with production environments, have high testing complexity, low automation, and difficulty in accurately reproducing core cache risk scenarios, resulting in insufficient testing efficiency and accuracy.
By binding Redis through the proxy layer, application data is collected in real time, fault policies are configured, and faults are simulated using a fine-grained matching and random fault injection strategy engine to perform end-to-end monitoring and evaluation, and generate evaluation reports.
It enables fault simulation without modifying infrastructure or application processes, reducing test interference, improving test efficiency and accuracy, and comprehensively verifying fault tolerance and data consistency. It is suitable for black-box testing and load testing scenarios.
Smart Images
Figure CN121919081A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of software testing technology, specifically relating to a Redis chaos fault testing method, storage medium, and device. Background Technology
[0002] In distributed systems, Redis, as a high-performance in-memory database, is widely used in scenarios such as caching, message queues, and distributed locks. As business scales up, the reliability and fault tolerance of Redis clusters become crucial for ensuring the stability of distributed systems. Traditional testing methods primarily verify the functionality and performance of distributed systems under normal conditions, but often fall short in testing for abnormal scenarios such as node failures, network partitions, insufficient memory, and command timeouts.
[0003] Currently, chaos fault testing tools for Redis can be mainly divided into the following two categories: (1) Infrastructure layer injection tools: Fault injection is achieved by operating the underlying hardware, operating system or container platform. They use system calls, kernel modules or cloud platform APIs to intervene in basic resources such as computing, network, and storage. Typical examples include Chaos Mesh and ChaosBlade-exec-os. The advantage of such tools is that the fault scenarios are close to the real faults of the infrastructure, but they are highly invasive and may interfere with other services in the production environment. (2) Application layer injection tools: By modifying the internal logic or bytecode of the application process, fault logic is inserted without directly operating the Redis server. Typical examples include Gremlin ALFI and ChaosBlade-exec-jvm. Their advantage is that they support context-aware fault simulation, but they need to modify the application deployment process and the language support is limited to the JVM ecosystem.
[0004] In summary, existing chaotic fault tests for Redis may interfere with normal production environments, thereby increasing the complexity and risk of testing. At the same time, the automation level of the entire process of fault configuration, execution and analysis is low, requiring a lot of manual intervention and resulting in low testing efficiency. In addition, the simulation capability for core risk scenarios of Redis caching, such as cache breakdown and avalanche, is weak, making it impossible to accurately reproduce such business-level faults and making it difficult to verify the effectiveness of the high availability design of the cache layer. Summary of the Invention
[0005] To address the problems existing in the prior art, this invention provides a Redis chaotic fault testing method, storage medium, and device, which can comprehensively verify Redis's fault tolerance, data consistency, and stability, reduce testing costs, and improve testing efficiency and accuracy.
[0006] To achieve the above technical objectives, the present invention adopts the following technical solution: A method for testing chaotic faults in Redis includes the following steps: Step S1: Obtain the resource information of the Redis to be tested, and bind the Redis to be tested to the proxy layer based on the resource information; Step S2: Configure fault policy; Step S3: The proxy layer collects real-time usage data of the Redis application to be tested, extracts the characteristics of Redis commands from the usage data, and injects the characteristics of Redis commands into the fault policy to obtain simulated faults; Step S4: Set up the fault injection strategy engine. The proxy layer injects simulated faults according to the decisions of the fault injection strategy engine, and performs full-link monitoring and evaluation during the simulated fault injection process to generate an evaluation report.
[0007] Furthermore, the configuration content of the fault policy includes: fault type, fault parameter name, fault parameter rule, fault execution method, and fault effective time.
[0008] Furthermore, the specific process of step S3 is as follows: The proxy layer parses the communication data packets between the application and the Redis under test in real time, extracts the characteristics of the Redis commands, including: Redis command type, key name, parameter value, request frequency and hot key, and injects the Redis command characteristics into the fault parameter rules of the fault policy.
[0009] Furthermore, it also includes: parsing the business identifier in the application request header through the proxy layer, associating the cross-Redis call chain based on the business identifier, and binding the business interface of the call chain with the Redis command.
[0010] Furthermore, the fault injection strategy engine includes: a fine-grained matching fault injection strategy engine and a random fault injection strategy engine; The fine-grained matching fault injection strategy engine is used to support decision-making based on any one or more of the following conditions: fault type, fault parameter name, fault parameter rules, fault execution method, business dimension, and fault effective time. The random fault injection strategy engine is used to support one or more of probabilistic triggering decisions, multi-fault hybrid decisions, and stress-aware adjustment decisions.
[0011] Furthermore, the simulated faults include: delay faults, error faults, cache invalidation faults, and network interruption faults.
[0012] Furthermore, it also includes configuring the duration or termination conditions of the simulated failure, and performing a Redis state reset after the simulated failure terminates the injection.
[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that causes a computer to execute the Redis chaos fault testing method described above.
[0014] Furthermore, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the Redis chaos fault testing method.
[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The chaotic fault testing method of Redis database in this invention realizes fault simulation through the proxy layer between Redis and application, without modifying the infrastructure or application process, thus reducing the interference of fault testing on business system. (2) The Redis chaotic fault testing method of the present invention sets up a fault injection strategy engine. The proxy layer injects simulated faults according to the decision of the fault injection strategy engine, realizing dual-mode injection of fine matching and random faults. Fine matching can help black-box testing verify the performance bottleneck of business process one by one, achieving some "white-box" testing effects at extremely low cost. Random faults can dynamically adjust the probability of random faults to simulate the scenario of deterministic faults and occasional faults coexisting in the production environment, improving the realism of fault testing.
[0016] The Redis chaotic fault testing method of this invention can more comprehensively verify the fault tolerance, data consistency and stability of business systems, especially suitable for mainstream scenarios such as black-box testing and load testing, reducing testing costs and improving fault testing efficiency and accuracy. Attached Figure Description
[0017] Figure 1 This is a flowchart of the Redis chaotic fault testing method of the present invention; Figure 2 This is a schematic diagram illustrating the configuration of the fault strategy in this invention. Detailed Implementation
[0018] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.
[0019] like Figure 1The flowchart below illustrates the Redis chaotic fault testing method of this invention. This method includes the following steps: Step S1: Obtain the resource information of the Redis instance to be tested, including its address and port. Based on the resource information, bind the Redis instance to be tested to a proxy layer. The proxy layer acts as an intermediary for communication between the application and Redis, handling subsequent traffic interception and fault injection. Fault simulation is achieved through the proxy layer without modifying the infrastructure or application process, reducing the interference of fault testing on the business system. In this invention, the proxy layer adopts a non-blocking communication architecture design, ensuring stable operation even under high-concurrency stress testing scenarios, enabling synchronous chaotic testing and stress testing, and comprehensively verifying the fault tolerance capability of the business system under high load.
[0020] Step S2: Configure the fault policy. Specifically, you can select preset fault policies from the experience base, such as cache breakdown and network latency, or you can customize fault parameters to determine the fault policy. Figure 2 The configuration content of the fault strategy in this invention includes: fault type, fault parameter name, fault parameter rule, fault execution method and fault effective time.
[0021] Step S3: The proxy layer collects real-time usage data of the application and the Redis under test, extracts the characteristics of Redis commands from the usage data, and injects the characteristics of Redis commands into the fault policy to obtain simulated faults, thereby providing a basis for accurate fault injection. Specifically, the proxy layer parses the communication data packets between the application and the Redis under test in real time, extracts the characteristics of Redis commands, including: Redis command type, key name, parameter value and request frequency, identifies hot keys through the sliding window algorithm, and injects the Redis command characteristics into the fault parameter rules of the fault policy.
[0022] Step S4: Set up the fault injection strategy engine. The proxy layer injects simulated faults according to the decisions of the fault injection strategy engine, and performs full-link monitoring and evaluation during the simulated fault injection process to generate an evaluation report.
[0023] In one technical solution of the present invention, the method further includes: parsing the business identifier in the application request header through the proxy layer, associating the cross-Redis call chain according to the business identifier, binding the business interface of the call chain with the Redis command, realizing the management of business process and Redis command, so that the result of fault injection can be directly mapped to the specific business, avoiding the problem of unclear fault impact scope in black-box testing, and improving testing efficiency.
[0024] In one technical solution of the present invention, the fault injection strategy engine includes: a fine-matching fault injection strategy engine and a random fault injection strategy engine, which not only supports fine-matching faults for core business, but also can superimpose random faults to simulate the scenario where deterministic faults and occasional faults coexist in the production environment, so as to make the fault test coverage more comprehensive.
[0025] The fine-grained matching fault injection strategy engine supports decision-making based on any one or more of the following conditions: fault type, fault parameter name, fault parameter rules, fault execution method, business dimension, and fault effective time. It adopts an incremental update mechanism, which makes decision modifications effective within 100ms without restarting the proxy layer. At the same time, the decisions can be reused, which facilitates problem localization and regression testing. Fine-grained matching can realize targeted fault simulation, helping black-box testing to verify the performance bottlenecks of business processes one by one without delving into the internal code of the business system, achieving some of the effects of white-box testing at a very low cost.
[0026] The random fault injection strategy engine supports one or more of the following: probabilistic triggering decision, multi-fault hybrid decision, and stress-aware adjustment decision. Probabilistic triggering decision supports configuring fault trigger probabilities from 0.1% to 100%, or implementing random triggering based on a Poisson distribution to simulate occasional faults. Multi-fault hybrid decision can configure random combinations of various fault types and supports setting parameter ranges for each type of fault. Stress-aware adjustment decision dynamically adjusts fault probabilities through a built-in QPS stress-aware algorithm to simulate scenarios where fault probability increases under high load. The random fault injection strategy engine can dynamically adjust random fault probabilities to simulate scenarios where deterministic and occasional faults coexist in a production environment, improving the realism of fault testing.
[0027] In one technical solution of the present invention, simulated faults include: delay faults, error faults, cache invalidation faults, and network interruption faults. Among them, delay faults include precise control of delays through scheduled tasks; error faults include constructing standard Redis error responses; cache invalidation faults include actively deleting specified hot keys or setting expiration times to precisely control the expiration time of hot keys, synchronously generating high-concurrency request streams, verifying the fault tolerance mechanisms such as rate limiting and degradation of the business system at the moment of hot data invalidation, supporting the simultaneous invalidation configuration of batch keys, simulating the scenario of concentrated cache expiration; it can simulate the overall unavailability of the cache service, test the stress resistance of Redis and the system degradation strategy; network interruption faults include simulating TCP connection disconnection or response timeout.
[0028] In one technical solution of the present invention, the specific process of performing full-link monitoring and evaluation during simulated fault injection is as follows: real-time collection of database metrics, application metrics, and business metrics, wherein the database metrics include: response time, throughput, number of connections, and replication latency; the application metrics include: response time quantiles, error rate, and transaction success rate; and the business metrics include: order processing volume, payment success rate, and cache hit rate; the fault tolerance capability of the business system is evaluated based on the collected metrics, and key vulnerabilities are identified; by comparing the changes in metrics before and after the fault, a test report is generated that includes the fault scenario, scope of impact, recovery time, and improvement suggestions.
[0029] In one technical solution of the present invention, the effective rules, scenarios and analysis results of this fault simulation are organized and new experience is generated and added to the experience base so as to provide a reference for subsequent Redis chaotic fault simulation, thereby realizing the accumulation and reuse of experience.
[0030] In one technical solution of the present invention, the method further includes: configuring the duration or termination condition of the simulated fault, and after the simulated fault terminates the injection, performing a Redis state reset, such as deleting temporary keys, restoring the tampered configuration and rebuilding the cache, and performing a health check after the reset to confirm that Redis has returned to normal; at the same time, an HTTP interface and command line tool are provided to support one-click stopping of all fault injections, with a response time ≤100ms.
[0031] The Redis chaotic fault testing method of this invention can more comprehensively verify the fault tolerance, data consistency and stability of business systems. It is especially suitable for mainstream scenarios such as black-box testing and load testing, reducing testing costs and improving fault testing efficiency and accuracy.
[0032] Example 1: Cache breakdown test in e-commerce flash sale scenario Scene description: An e-commerce platform is holding a flash sale. The product inventory information (key: seckill:goods:1001) is a hot key, with peak QPS exceeding 5000 during the event. It is necessary to verify whether the system can prevent instantaneous database overload through mechanisms such as rate limiting and mutex locks when the hot key expires, while simultaneously ensuring a smooth flash sale experience for normal users.
[0033] Using the Redis chaotic fault testing method of this invention, the address of the proxy layer is configured as the Redis connection address of the e-commerce application. Through hotspot identification, seckill:goods:1001 is confirmed as a hot key with more than 3,000 accesses per minute.
[0034] Fault policy configuration: Fine-grained matching: key mode is seckill:goods:1001, fault type is "cache invalidation", and trigger time is set to 5 minutes after the start of the flash sale; Random Fault: After a fault is triggered, a "50-100ms delay" fault with a 30% probability is superimposed to simulate network fluctuations under high concurrency; Load testing linkage: Simulate a 5000 QPS flash sale request flow using JMeter, continuously sending inventory query and deduction requests to the application.
[0035] Fault execution and monitoring: Once the trigger time is reached, the system will automatically perform a cache invalidation operation; The monitoring and evaluation module allows for real-time observation of metrics such as database QPS, application response time, and error rate.
[0036] Restore verification: After the failure lasted for 10 minutes, the automated recovery module automatically terminated the failure, performed a Redis state reset, and confirmed that the service had returned to normal through a health check.
[0037] Expected results: When the hot key expired, the database QPS did not exceed the threshold, and the application P99 response time increased from the normal 50ms to 150ms but did not time out, with an error rate of less than 0.5%, indicating that the rate limiting mechanism was successfully effective. By accurately identifying hot keys and combining the combined fault of "cache expiration + random latency," the cache breakdown risk in a flash sale scenario was realistically reproduced; non-intrusive deployment avoided affecting the normal preparation of the event, and high-performance proxy ensured the authenticity and reliability of the load test data.
[0038] Example 2: Cache Avalanche Test in Financial Payment Scenarios Scene description: The financial platform's payment order cache (key prefix: pay:order:*) contains over 100,000 keys, all with the same expiration time of 2 AM. It is necessary to verify whether the system can prevent database crashes and ensure payment continuity when a large number of keys expire simultaneously through strategies such as service degradation and cache preheating.
[0039] The Redis chaotic fault testing method of this invention is adopted, and the proxy layer connects to the Redis Cluster of the financial platform; through the business process association function, the "payment order query" business scenario is divided and all pay:order:* keys are associated.
[0040] Fault policy configuration: Fine-grained matching: The business scenario is "payment order query", the fault type is "batch cache failure", select 100,000 pay:order:* keys, and set the trigger time to 1:59 AM; Pressure sensing adjustment: Enable "Double the probability of failure when QPS>2000" to simulate the risk of avalanche under high load; Business traffic simulation: Real payment order query traffic is injected into the test environment by replicating the shadow traffic of the production environment. The normal QPS is about 1500.
[0041] Fault execution and monitoring: At 1:59 AM, a batch cache invalidation was triggered. The monitoring and evaluation module tracked metrics such as database CPU utilization, cache hit rate, and application degradation interface call volume in real time.
[0042] Restore verification: Thirty minutes after the fault is triggered, the automated recovery is manually triggered. The system performs cache preheating, batch loads hot order data, generates a test report, and analyzes the effectiveness of the degradation strategy.
[0043] Expected results: After a batch of keys expired, the cache hit rate dropped from 95% to 30%, but the application automatically triggered a degradation strategy, returning basic order information. Database CPU utilization remained below 60%, and the payment query interface availability was 100%. This capability supports the simulation of batch key expiration, addressing the pain point of traditional tools being unable to reproduce large-scale cache avalanches. Combined with business scenario segmentation and stress perception adjustment, it enables avalanche scenario testing that more closely resembles production settings, providing strong validation for financial-grade high availability.
[0044] Example 3: Combining Random Fault Testing and Load Testing in Social Application Scenarios Scene description: The social application's user dynamic cache (key: user:feed:*) is called more than 100 million times per day. It is necessary to verify the impact of occasional random failures such as Redis command delays and errors on the user's dynamic refresh function under daily high concurrency (QPS 3000+) scenarios, and to evaluate the system's fault tolerance resilience.
[0045] Using the Redis chaotic fault testing method of this invention, the proxy service connects the application to the Redis master-slave cluster. Through command feature extraction, it is confirmed that the main operation commands of user:feed:* are LRANGE and LPUSH.
[0046] Fault policy configuration: Random Faults: The command type is LRANGE, the fault combination is "20% probability of delay (100-300ms), 5% probability of error (returns "-ERR timeout")", and the random seed is set to "20250101" to ensure reproducibility; Time dimension: The fault is effective during the weekday period from 10:00 to 12:00 (peak user activity). Load test execution: Simulate 3000 QPS of dynamic user refresh requests using a load testing tool for 2 hours.
[0047] Fault execution and monitoring: The monitoring and evaluation module collects indicators such as application response time, number of retries, and user operation failure rate in real time, and generates "fault-performance" correlation analysis charts.
[0048] Restore verification: After the test, key failure scenarios were reproduced using random seeds to verify the correctness of the application's retry mechanism and error handling logic.
[0049] Expected results: During random fault injection, the application's average response time increased from 80ms to 120ms, the user operation failure rate was less than 0.3%, and the retry mechanism successfully converted 90% of delayed requests into successful responses. This deep integration of random faults with stress testing solves the disconnect between traditional tools that focus solely on either pure stress testing or pure chaos; by using controllable random seeds to reproduce fault scenarios, the development team can accurately locate and optimize fault-tolerant logic, improving the stability of social applications in complex online environments.
[0050] In one technical solution of the present invention, a computer-readable storage medium is also provided, which stores a computer program that causes a computer to execute the Redis chaos fault testing method of the present invention.
[0051] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the Redis chaos fault testing method of the present invention.
[0052] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, and portable compact disc read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0053] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0054] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A method for testing chaotic faults in Redis, characterized in that, Includes the following steps: Step S1: Obtain the resource information of the Redis to be tested, and bind the Redis to be tested to the proxy layer based on the resource information; Step S2: Configure fault policy; Step S3: The proxy layer collects real-time usage data of the Redis application to be tested, extracts the characteristics of Redis commands from the usage data, and injects the characteristics of Redis commands into the fault policy to obtain simulated faults; Step S4: Set up the fault injection strategy engine. The proxy layer injects simulated faults according to the decisions of the fault injection strategy engine, and performs full-link monitoring and evaluation during the simulated fault injection process to generate an evaluation report.
2. The Redis chaotic fault testing method according to claim 1, characterized in that, The configuration content of the fault policy includes: fault type, fault parameter name, fault parameter rule, fault execution method, and fault effective time.
3. The Redis chaotic fault testing method according to claim 1, characterized in that, The specific process of step S3 is as follows: The proxy layer parses the communication data packets between the application and the Redis under test in real time, extracts the characteristics of the Redis commands, including: Redis command type, key name, parameter value, request frequency and hot key, and injects the Redis command characteristics into the fault parameter rules of the fault policy.
4. The Redis chaotic fault testing method according to claim 3, characterized in that, Also includes: The proxy layer parses the business identifier in the application request header, associates the cross-Redis call chain based on the business identifier, and binds the business interface of the call chain with the Redis command.
5. A Redis chaotic fault testing method according to claim 1, characterized in that, The fault injection strategy engine includes: a fine-match fault injection strategy engine and a random fault injection strategy engine; The fine-grained matching fault injection strategy engine is used to support decision-making based on any one or more of the following conditions: fault type, fault parameter name, fault parameter rules, fault execution method, business dimension, and fault effective time. The random fault injection strategy engine is used to support one or more of probabilistic triggering decisions, multi-fault hybrid decisions, and stress-aware adjustment decisions.
6. A Redis chaotic fault testing method according to claim 1, characterized in that, The simulated faults include: delay faults, error faults, cache invalidation faults, and network interruption faults.
7. A Redis chaotic fault testing method according to claim 1, characterized in that, Also includes: Configure the duration or termination conditions for the simulated failure, and perform a Redis state reset after the simulated failure terminates the injection.
8. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the Redis chaos fault testing method as described in any one of claims 1-7.
9. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the Redis chaotic fault testing method as described in any one of claims 1-7.