Chaotic test dynamic operation method

Abstract

The invention provides a chaotic test dynamic operation method. Service nodes and service processes are mapped and associated through a service weight atlas construction module, and the problem of service priority missing is solved based on a multi-dimensional service attribute quantification weight value and a fault conduction coefficient; through a dynamic fault arrangement module, differentiated fault risk levels are matched according to weight values, the fault strength is dynamically adjusted in combination with conduction coefficients, low-load time periods are selected for execution, and core services are prevented from being impacted; through an intelligent loss stopping module, a second-level automatic loss stopping and recovery process is triggered based on a dual-threshold monitoring core business index and an emergency threshold, and response time is compressed to a second level. And finally, quantitative evaluation of the business influence is realized, the test risk is remarkably reduced, and the controllability is improved.

Description

Technical Field

[0001] This application relates to the field of microservice testing, and in particular to a dynamic method for running chaos tests. Background Technology

[0002] With the deep application of distributed systems (such as microservices and cloud-native architectures) in various fields, system complexity is increasing exponentially, and potential failure points (such as service downtime, network latency, and resource exhaustion) are difficult to cover through traditional testing. Chaos testing, as a core technology to ensure system stability, actively injects faults to simulate real abnormal scenarios and verifies the system's fault tolerance and fault recovery capabilities, and has become a standard part of distributed system operation and maintenance.

[0003] Currently, the core requirements of chaos testing focus on "accurate fault injection" and "controllable risk." It aims to expose system vulnerabilities through faults while preventing actual losses to core business operations (such as e-commerce payments and financial transactions) due to test interruptions. However, many feasible chaos testing solutions focus on technical-layer fault coverage without deep coupling with business priorities. This makes core business operations vulnerable to impact during testing, and fault mitigation relies on manual intervention, failing to meet the demands of enterprise-level systems for both stability and business continuity. In summary, current chaos testing methods suffer from poor fault injection accuracy, passive mitigation mechanisms, and difficulty in quantifying risks. Therefore, a dynamic chaos testing methodology is needed to improve the effectiveness of chaos testing while ensuring business stability and controllability. Summary of the Invention

[0004] The purpose of this application is to address at least one of the aforementioned technical deficiencies, particularly the lack of stability and effectiveness of chaos testing in the prior art, which has a significant impact on business operations.

[0005] Firstly, this application provides a method for dynamically running a chaos test, the method comprising:

[0006] Collect mapping data of service call relationships and business processes corresponding to each service node, determine the business weight value of each service node by weighting according to multiple preset evaluation dimensions, and determine the fault transmission coefficient between each service node. Based on the mapping data, the business weight value and the fault transmission coefficient, construct a business weight map corresponding to each service node.

[0007] Based on the business weight value of each service node, the fault risk level is matched and at least one fault event is selected. Then, a fault injection operation corresponding to each fault event is performed on each service node.

[0008] During fault operation, core business indicators are monitored in real time, and the fault operation status is adjusted according to the changes in each of the core business indicators.

[0009] As an optional implementation, the evaluation dimensions include business importance parameters, impact scope parameters, and fault recovery cost parameters. The step of determining the business weight value of each service node based on a weighted average of multiple preset evaluation dimensions includes:

[0010] Based on the importance of the business operations supported by each service node, the business importance parameter is determined; based on the user coverage affected by each service node, the impact range parameter is determined; and based on the resource consumption for fault recovery of each service node, the fault recovery cost parameter is determined.

[0011] Assign corresponding weight coefficients to the business importance parameter, the scope of influence parameter, and the fault recovery cost parameter, and calculate the business weight value of each service node using weighted average.

[0012] As an optional implementation, determining the fault propagation coefficient between each of the service nodes includes:

[0013] Based on the service call relationship and the business process, determine the dependency type between each service node;

[0014] The dependency types include strong dependency, weak dependency, and no dependency.

[0015] Based on the dependency type, the range of the fault propagation coefficient is determined, and based on the service functions of each service node and the corresponding historical operation data, the value of the fault propagation coefficient is determined within the corresponding range.

[0016] As an optional implementation, the step of matching fault risk levels and selecting at least one fault event based on the service weight values ​​of each service node includes:

[0017] Based on the negative correlation between the business weight value of each service node and the fault risk level, a target fault risk level corresponding to each service node is determined, and at least one fault event of fault type is selected from the target fault risk level.

[0018] Furthermore, if the fault transmission coefficient from the target service node to the core service node exceeds a set value, the fault intensity parameter corresponding to each fault event of the target service node is adjusted.

[0019] As an optional implementation, the step of performing fault injection operations corresponding to each of the fault events on each of the service nodes includes:

[0020] Determine the business load distribution based on historical business traffic data;

[0021] Based on the aforementioned business load distribution and each of the aforementioned fault events, the fault execution period is determined;

[0022] Based on the fault execution period, perform fault injection operations corresponding to each fault type on each of the service nodes.

[0023] As an optional implementation, adjusting the fault operation status according to changes in the core business indicators includes:

[0024] When any of the core business metrics reaches the first threshold, the fault is terminated to restore normal service status.

[0025] When any of the core business indicators reaches the second threshold, the system continues to operate under fault conditions but triggers the generation of an alarm signal.

[0026] After the fault is terminated, a corresponding fault log is generated to analyze the fault operation and optimization plan.

[0027] Secondly, this application provides a dynamic operation device for chaos testing, comprising:

[0028] The processing module is used to collect mapping data of service call relationships and business processes corresponding to each service node, determine the business weight value of each service node by weighting according to multiple preset evaluation dimensions, determine the fault transmission coefficient between each service node, and construct a business weight map corresponding to each service node based on the mapping data, the business weight value and the fault transmission coefficient.

[0029] The processing module is further configured to match the fault risk level and select at least one fault event according to the business weight value of each service node, and perform a fault injection operation corresponding to each fault event on each service node.

[0030] The processing module is also used to monitor core business indicators in real time during fault operation and adjust the fault operation status according to the changes in each core business indicator.

[0031] As an optional implementation, the evaluation dimensions include business importance parameters, impact scope parameters, and fault recovery cost parameters. The specific method by which the processing module determines the business weight value of each service node based on multiple preset evaluation dimensions includes:

[0032] Based on the importance of the business operations supported by each service node, the business importance parameter is determined; based on the user coverage affected by each service node, the impact range parameter is determined; and based on the resource consumption for fault recovery of each service node, the fault recovery cost parameter is determined.

[0033] Assign corresponding weight coefficients to the business importance parameter, the scope of influence parameter, and the fault recovery cost parameter, and calculate the business weight value of each service node using weighted average.

[0034] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the method described in the first aspect.

[0035] Fourthly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method described in the first aspect.

[0036] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0037] Based on any of the above embodiments, this application uses a business weight graph construction module to map and associate service nodes with business processes, quantifies weight values ​​and fault propagation coefficients based on multi-dimensional business attributes, and solves the problem of missing business priorities. Through a dynamic fault orchestration module, it matches differentiated fault risk levels according to weight values, dynamically adjusts fault intensity based on propagation coefficients, and selects low-load periods for execution to avoid impacting core businesses. Through an intelligent loss mitigation module, it monitors core business indicators based on dual thresholds, and triggers second-level automatic loss mitigation and recovery processes based on emergency thresholds, compressing response time to the second level. Ultimately, it achieves quantitative assessment of business impact, significantly reducing testing risks and improving controllability. The core technical effect stems from the closed-loop collaboration of three innovative modules. First, the business weight graph construction module collects the mapping relationship between service call chains and business processes, defines three evaluation dimensions—business revenue share, user coverage, and fault recovery cost—and calculates the weight value of each service node, quantifying business priorities into actionable indicators. Simultaneously, it dynamically calculates fault propagation coefficients based on dependency types, accurately predicting fault propagation paths and solving the problem of missing associations between business entities and technical entities in existing technologies. Secondly, the dynamic fault orchestration module automatically assigns low-risk fault events to high-weight services and high-risk fault events to low-weight services based on the negative correlation mapping rule between weight value ranges and fault risk levels. It also dynamically reduces fault intensity parameters when the transmission coefficient exceeds a threshold, preventing core business from being impacted by related faults. Furthermore, it identifies faults executing during low-load periods using historical business traffic data, ensuring business continuity and completely resolving the problem of blind fault injection. Finally, the intelligent loss mitigation module implements tiered monitoring and response by configuring two-level thresholds for core business indicators: warning thresholds trigger alarm notifications, and emergency thresholds trigger fully automated second-level loss mitigation processes, including fault termination, service recovery, and log recording, reducing manual response time from minutes to seconds. After loss mitigation, it automatically generates a business impact report, quantifying the fault duration, order loss, and number of affected users, making test risks assessable and optimizable. The overall solution upgrades chaos testing from technical verification to a business stability assurance tool, balancing security and practicality. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a dynamic running method for chaos testing provided in one embodiment of this application.

[0040] Figure 2A schematic diagram of the system architecture corresponding to the dynamic running method for chaos testing provided in one embodiment of this application;

[0041] Figure 3 This is a schematic diagram of a visualized business weight graph generated in a dynamic running method for chaos testing provided in one embodiment of this application.

[0042] Figure 4 A flowchart illustrating the process of generating and executing a fault orchestration plan for a dynamic running method of chaos testing provided in one embodiment of this application;

[0043] Figure 5 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] With the deep application of distributed systems (such as microservices and cloud-native architectures) in e-commerce, finance, government affairs, and other fields, system complexity is increasing exponentially, and potential failure points (such as service downtime, network latency, and resource exhaustion) are difficult to cover through traditional testing. Chaos testing, as a core technology to ensure system stability, actively injects faults to simulate real abnormal scenarios and verifies the system's fault tolerance and fault recovery capabilities, and has become a standard part of distributed system operation and maintenance.

[0046] Currently, the core requirements of chaos testing focus on "accurate fault injection" and "controllable risk." It aims to expose system vulnerabilities through faults while preventing actual losses to core business operations (such as e-commerce payments and financial transactions) due to test interruptions. However, existing chaos testing solutions mostly focus on "technical layer fault coverage" and are not deeply coupled with business priorities. This makes core business operations vulnerable to impact during testing, and fault mitigation relies on manual intervention, failing to meet the enterprise-level system's requirement of "balancing stability and business continuity."

[0047] Currently, some feasible chaos testing solutions, along with their technical approaches and limitations, are illustrated below:

[0048] Option 1: General-purpose chaos testing tools (represented by ChaosBlade and Gremlin)

[0049] Technical principle: Fault injection is achieved based on "fault template library + manual configuration". Users need to manually specify the target service (such as "order service instance 1"), fault type (such as "instance downtime" or "CPU utilization of 90%)" and duration. The tool performs fault injection on the target node through API or Agent, while collecting basic monitoring indicators (such as CPU, memory, and interface response codes).

[0050] Application scenarios: Suitable for stability testing of non-core business (such as log service, data backup service), or internal system testing without differences in business priority.

[0051] Option 2: Chaos test orchestration scheme based on architecture diagram (represented by ChaosMesh and HaloChaos)

[0052] Technical principle: By parsing the architecture diagram of the distributed system (such as Kubernetes resource topology, service call relationship diagram), it automatically matches "architecture node - fault template" (such as "database node → injection connection timeout fault"), and supports batch orchestration of faults by "architecture layer" (such as application layer, data layer), but the fault priority within the layer still needs to be manually defined.

[0053] Application scenarios: Suitable for systems with a clear architecture and strong binding between business modules and technical layers (such as standardized government approval systems), but unable to distinguish the importance of different businesses within the same layer (such as the difference between "payment database" and "log database" in the data layer).

[0054] Therefore, the core limitations of these feasible implementation methods are concentrated in the following three points, which fail to meet the chaos testing requirements of enterprise-level core businesses:

[0055] 1. Blind Fault Injection: Failure to correlate with business priorities leaves core businesses vulnerable. Existing solutions select fault targets solely based on technical attributes (such as service type and architecture level), without considering differences in business priorities. For example, injecting the same level of "instance downtime" fault into "payment service" (a core business affecting revenue) and "log service" (a non-core business affecting only troubleshooting efficiency) could lead to payment process interruptions and actual economic losses.

[0056] 2. Passive loss mitigation mechanism: Relies on manual monitoring, which can lead to the amplification of the impact of failures. Existing loss mitigation solutions require real-time monitoring of key metrics (such as interface success rate and business completion rate). When an anomaly is detected in the core business, the "termination of failure" operation must be manually executed. The response time from "anomaly detection → manual intervention → loss mitigation execution" usually exceeds 5 minutes, which can easily lead to the spread of the impact of failures (such as in e-commerce flash sale scenarios, where thousands of failed orders may be generated within 5 minutes).

[0057] 3. Risks are not quantifiable: Lack of business impact assessment makes it difficult to control the testing scope. Existing solutions cannot quantify the impact of failures on business, and can only judge the system status through technical indicators (such as CPU utilization and interface latency). For example, after injecting a "network latency" failure, it is impossible to predict whether the failure will lead to a "1% decrease in payment success rate" or a "10% decrease," resulting in a testing scope that is either too narrow (failing to expose potential risks) or too broad (leading to business risks).

[0058] This application aims to provide a chaos testing system that deeply integrates business and technology. It enables dynamic fault orchestration driven by business weights, assigning weights to service nodes based on business importance and automatically matching differentiated fault types and intensities to prevent core business operations from being impacted by high-level faults. Simultaneously, it achieves intelligent loss mitigation through monitoring linkage, collecting core business metrics in real time and setting thresholds to trigger automatic loss mitigation, reducing fault response time from minutes to seconds and minimizing the scope of fault impact. Furthermore, it enables quantitative assessment of business impact, constructing a "fault-business impact" correlation model to predict risk levels before testing and output quantitative reports after testing, ensuring the testing process is "controllable, assessable, and low-risk."

[0059] In summary, the technical concept of this application lies in the following: First, through a business weight graph construction module, service nodes are mapped and associated with business processes. This module quantifies weight values ​​and fault propagation coefficients based on multi-dimensional business attributes, addressing the issue of missing business priorities. Second, through a dynamic fault orchestration module, differentiated fault risk levels are matched based on weight values, and fault intensity is dynamically adjusted in conjunction with the propagation coefficient, with execution occurring during low-load periods to avoid impacting core business operations. Third, through an intelligent loss mitigation module, core business indicators are monitored based on dual thresholds, with emergency thresholds triggering second-level automatic loss mitigation and recovery processes, compressing response time to the second level. Ultimately, this achieves quantitative assessment of business impact, significantly reducing testing risks and improving controllability. The core technical effect stems from the closed-loop collaboration of these three innovative modules. First, the business weight graph construction module collects the mapping relationship between service call chains and business processes, defines three evaluation dimensions—business revenue share, user coverage, and fault recovery cost—and calculates the weighted business weight value of each service node, quantifying business priorities into actionable indicators. Simultaneously, it dynamically calculates fault propagation coefficients based on dependency types, accurately predicting fault propagation paths and addressing the problem of missing associations between business entities and technical entities in existing technologies. Secondly, the dynamic fault orchestration module automatically assigns low-risk fault events to high-weight services and high-risk fault events to low-weight services based on the negative correlation mapping rule between weight value ranges and fault risk levels. It also dynamically reduces fault intensity parameters when the transmission coefficient exceeds a threshold, preventing core business from being impacted by related faults. Furthermore, it identifies faults executing during low-load periods using historical business traffic data, ensuring business continuity and completely resolving the problem of blind fault injection. Finally, the intelligent loss mitigation module implements tiered monitoring and response by configuring two-level thresholds for core business indicators: warning thresholds trigger alarm notifications, and emergency thresholds trigger fully automated second-level loss mitigation processes, including fault termination, service recovery, and log recording, reducing manual response time from minutes to seconds. After loss mitigation, it automatically generates a business impact report, quantifying the fault duration, order loss, and number of affected users, making test risks assessable and optimizable. The overall solution upgrades chaos testing from technical verification to a business stability assurance tool, balancing security and practicality.

[0060] The methods provided in this application will be described in detail below based on the corresponding implementation methods in some practical application scenarios.

[0061] Figure 1 This is a flowchart illustrating a dynamic running method for chaos testing provided in one embodiment of this application, as shown below. Figure 1 As shown, this application provides a method for dynamically running a chaos test, the method comprising:

[0062] S101. Collect mapping data of service call relationships and business processes corresponding to each service node, determine the business weight value of each service node by weighting according to multiple preset evaluation dimensions, and determine the fault transmission coefficient between each service node. Based on the mapping data, the business weight value and the fault transmission coefficient, construct a business weight map corresponding to each service node.

[0063] S102. Based on the business weight value of each service node, match the fault risk level and select at least one fault event, and perform a fault injection operation corresponding to each fault event on each service node.

[0064] S103. During fault operation, monitor core business indicators in real time and adjust the fault operation status according to the changes in each of the core business indicators.

[0065] The implementation method provided in this application is achieved through three core modules: a business weight graph construction module, a dynamic fault orchestration module, and an intelligent loss prevention module. These modules work together to form a fully closed-loop chaotic testing process, from "defining business weights → dynamically executing faults → automatically stopping losses due to anomalies." The following, with reference to the schematic diagram, elaborates on the implementation logic, key steps, and specific examples of each module.

[0066] Please see Figure 2 , Figure 2 This is a schematic diagram of the system architecture corresponding to the dynamic running method for chaos testing provided in one embodiment of this application, as shown below. Figure 2 As shown, the core consists of a "data acquisition layer," a "core processing layer," an "execution layer," and a "visualization layer," with the functions of each layer as follows:

[0067] Data Acquisition Layer: Service dependencies are collected through call chain tools (such as SkyWalking), business metrics (such as payment success rate and order completion rate) are collected through business monitoring platforms (such as Prometheus + Grafana), and business weight rules are obtained through the configuration center.

[0068] Core processing layer: It contains three core modules (business weight graph construction, dynamic fault orchestration, and intelligent loss prevention) and is the logical core of the system.

[0069] Execution layer: Fault injection is performed through a Chaos Agent (such as ChaosBlade Agent), and alarms and loss-stopping operations are triggered by API calls to the monitoring platform.

[0070] Visualization layer: Displays business weight graph, fault execution status, and business impact report, and supports users in configuring weight rules and stop-loss thresholds.

[0071] In the following implementations, the parts involving practical application scenarios will be explained in conjunction with the module architecture, based on some feasible implementation methods or values ​​in the e-commerce field, and some specific steps in the method will be described.

[0072] This implementation constructs a business weight graph, mapping and associating service nodes with business processes. It calculates business weight values ​​based on multiple evaluation dimensions, including revenue share, user coverage, and fault recovery costs, quantifying differences in business priority and resolving the disconnect between fault injection target selection and business importance. By evaluating the fault propagation coefficient between service nodes, it accurately predicts the extent of fault spread in the dependency chain, providing a basis for dynamic orchestration. Based on business weight values, it automatically matches differentiated fault risk levels, ensuring that high-weight services are only injected with low-risk faults, avoiding impact on core businesses. Combined with a dual-threshold mechanism for real-time monitoring of core business metrics, it automatically terminates faults and restores services when metrics reach emergency thresholds, achieving second-level response and significantly reducing the scope of fault impact. This enhances the safety and controllability of chaos testing while ensuring business continuity.

[0073] As an optional implementation, the evaluation dimensions include business importance parameters, impact scope parameters, and fault recovery cost parameters. The step of determining the business weight value of each service node based on a weighted average of multiple preset evaluation dimensions includes:

[0074] Based on the importance of the business operations supported by each service node, the business importance parameter is determined; based on the user coverage affected by each service node, the impact range parameter is determined; and based on the resource consumption for fault recovery of each service node, the fault recovery cost parameter is determined.

[0075] Assign corresponding weight coefficients to the business importance parameter, the scope of influence parameter, and the fault recovery cost parameter, and calculate the business weight value of each service node using weighted average.

[0076] This implementation method defines three evaluation dimensions: business importance parameter, impact scope parameter, and fault recovery cost parameter. It quantifies business operational value, user impact scale, and fault recovery resource consumption into calculable indicators, and then weights these indicators based on preset weight coefficients to form a comprehensive business weight value. This design covers key aspects of business value, ensuring that the weight calculation fully reflects the actual business priority of service nodes and avoids evaluation biases caused by relying solely on technical attributes. This provides a reliable basis for accurate matching of subsequent fault risk levels, improving the targeting and security of fault injection.

[0077] As an optional implementation, determining the fault propagation coefficient between each of the service nodes includes:

[0078] Based on the service call relationship and the business process, determine the dependency type between each service node;

[0079] The dependency types include strong dependency, weak dependency, and no dependency.

[0080] Based on the dependency type, the range of the fault propagation coefficient is determined, and based on the service functions of each service node and the corresponding historical operation data, the value of the fault propagation coefficient is determined within the corresponding range.

[0081] This implementation analyzes service call relationships and business processes, classifying inter-node dependencies into three categories: strong dependencies, weak dependencies, and no dependencies. Based on these dependency types, it sets a range for the fault propagation coefficient. By combining historical operational data within this range, specific coefficient values ​​are determined, quantifying the degree of fault propagation in the service chain. This mechanism solves the problem of dynamically assessing the impact chain of faults, ensuring that potential impacts on downstream core services are predictable, providing data support for dynamically adjusting fault intensity, and enhancing the risk controllability of the testing process.

[0082] Please see Figure 3 , Figure 3 This is a schematic diagram of a visualized business weight graph generated in a dynamic running method for chaos testing according to one embodiment of this application. Each element in the visualized business weight graph is related to the aforementioned implementation methods, including the business weight score of each service node and the corresponding fault propagation coefficient between service nodes. For example... Figure 3 This demonstrates the strong dependency between order placement and payment services, as well as the highest business weight of payment services in the e-commerce sector.

[0083] In specific application scenarios, a visual business weight graph is constructed through the business weight graph construction module. Based on "service dependency relationship + business attribute", a visual graph containing "service node, business weight value and fault propagation coefficient" is automatically constructed to provide a decision basis for subsequent fault orchestration.

[0084] Step 1: Map the service call chain to the business process:

[0085] Implementation logic: Real-time call relationships of the distributed system are collected through call chain tools (such as SkyWalking) to generate "service call topology" (such as "user service → order service → payment service → reconciliation service"); at the same time, through the business configuration interface, users associate "service nodes → business processes" (such as associating "order service + payment service" with "core transaction process", and associating "log service" with "auxiliary operation and maintenance process").

[0086] E-commerce example: The collected service call chain is "User Service → Order Service → Inventory Service → Payment Service → Reconciliation Service → Log Service", where "Order → Inventory → Payment → Reconciliation" is associated with the "Core Transaction Process", and "Log Service" is associated with the "Auxiliary Process".

[0087] Step 2: Define the business weight evaluation dimensions and calculation rules:

[0088] Implementation logic: The system has 3 preset core evaluation dimensions, and users can configure the weight ratio of each dimension (the default sum is 100%).

[0089] Business importance parameter, business revenue share (weight 50%, score range 0-10): the proportion of revenue generated by the business supported by this service to the total revenue (e.g., if payment service supports 100% of transaction revenue, then it is 10 points).

[0090] Impact range parameter (weight 30%): the percentage of users who depend on this service (e.g., if the payment service affects 100% of users, the weight is 10 points).

[0091] Fault recovery cost parameters (weight 20%): The time and manpower costs required to restore business after a service failure. Please refer to Table 1 below to determine how to determine the fault recovery cost parameters.

[0092] Table 1

[0093]

[0094] Business weight value calculation formula: Business weight value = (revenue percentage × 50%) + (user scope × 30%) + (recovery cost score × 20%), with a value range of 0-10 points (10 points is the highest priority).

[0095] Step 3: Calculate the business weight value of each service node:

[0096] E-commerce example: Based on the above calculation method, the weight calculation results of each service are shown in Table 2:

[0097] Table 2

[0098]

[0099] Step 4: Evaluate the fault transmission coefficient:

[0100] Implementation logic: The fault propagation coefficient refers to "the proportion of business impact on downstream dependent services after a service failure," calculated based on the "strong / weak dependency relationship" of service calls.

[0101] Strong dependency (e.g., "order service → inventory service": ordering requires checking inventory): transmission coefficient = 90%-100%;

[0102] Weak dependencies (e.g., "Payment service → Log service": asynchronous log writing after successful payment): Transmission coefficient = 10%-30%;

[0103] No dependencies (e.g., "log service → payment service"): Transmission coefficient = 0%.

[0104] E-commerce example: The transmission coefficient of the payment service (10 points) to the downstream reconciliation service is 90% (payment failure directly leads to reconciliation anomalies), and the transmission coefficient to the log service is 20% (payment failure only affects some log records).

[0105] Finally, by combining the information from the aforementioned aspects, the following was integrated to form: Figure 3 The diagram shown.

[0106] As an optional implementation, the step of matching fault risk levels and selecting at least one fault event based on the service weight values ​​of each service node includes:

[0107] Based on the negative correlation between the business weight value of each service node and the fault risk level, a target fault risk level corresponding to each service node is determined, and at least one fault event of fault type is selected from the target fault risk level.

[0108] Furthermore, if the fault transmission coefficient from the target service node to the core service node exceeds a set value, the fault intensity parameter corresponding to each fault event of the target service node is adjusted.

[0109] This implementation method automatically matches low-risk fault events to high-weight services and high-risk fault events to low-weight services based on a negative correlation mapping rule between business weight values ​​and fault risk levels, fundamentally preventing core businesses from being impacted by high-level faults. Simultaneously, when the fault propagation coefficient of a target service node to the core service exceeds a threshold, its fault intensity parameter is dynamically reduced, further blocking the propagation path of the fault to the core business. This collaborative logic solves the problem of blind fault injection and significantly reduces the risk of core business interruption.

[0110] As an optional implementation, the step of performing fault injection operations corresponding to each of the fault events on each of the service nodes includes:

[0111] Determine the business load distribution based on historical business traffic data;

[0112] Based on the aforementioned business load distribution and each of the aforementioned fault events, the fault execution period is determined;

[0113] Based on the fault execution period, perform fault injection operations corresponding to each fault type on each of the service nodes.

[0114] This implementation method identifies periods of low system load by analyzing historical business traffic data and dynamically plans the timing of fault execution based on this. High-risk faults are avoided during peak business hours, and core service faults are avoided during critical business activity periods, ensuring that business continuity is not significantly affected during fault injection. This strategy, combined with fault level selection driven by business weights, forms a dual protection mechanism, further optimizing the impact of the testing process on actual business operations.

[0115] Please see Figure 4 , Figure 4 This is a flowchart illustrating the process of generating and executing a fault orchestration plan using a dynamic chaotic testing method provided in one embodiment of this application. This process is implemented through a dynamic fault orchestration module, which automatically matches "fault type, fault intensity, and execution timing" based on a business weight graph to achieve differentiated fault orchestration. Its core logic is "high-weight services for low-risk faults, and low-weight services for high-risk faults."

[0116] Specific steps may include:

[0117] Step 1: Preset fault type and weight matching rules:

[0118] Implementation logic: The system has four built-in fault types, classified into L1 (low risk), L2 (medium risk), L3 (high risk), and L4 (extremely high risk) according to their "risk level" from low to high. It also has a preset "weight value - fault risk level" matching rule (which can be customized by the user). See Table 3 for details.

[0119] Table 3

[0120]

[0121] Step 2: Automatically select the fault target and fault type:

[0122] E-commerce example:

[0123] For the "Payment Service" (core) with a weight of 10 points: match L1 level faults and select "network latency 100ms";

[0124] For the "Log Service" (non-core) with a weight of 0.8: match L4 level faults and select "Instance Downtime";

[0125] For the "Reconciliation Service" (General) with a weight of 6.5: Match L2 level faults and select "Memory Usage 50%".

[0126] Step 3: Dynamically adjust fault intensity and execution timing:

[0127] Fault Intensity Adjustment: The intensity is dynamically adjusted based on the fault propagation coefficient. If the propagation coefficient of service A to core service B is ≥80%, the fault intensity of service A is reduced by 30%. For example, if the propagation coefficient of "reconciliation service" to "payment service" is 90%, then its L2 level fault "memory utilization rate 50%" is adjusted to "memory utilization rate 35%".

[0128] Execution timing selection: Avoid peak business periods and automatically select "low load periods" to execute faults (e.g., e-commerce systems select 2-4 am to execute high-risk faults, and core service faults select non-flash sale periods).

[0129] This allows for the generation of fault orchestration plans and the execution of fault injection operations according to those plans.

[0130] As an optional implementation, adjusting the fault operation status according to changes in the core business indicators includes:

[0131] When any of the core business metrics reaches the first threshold, the fault is terminated to restore normal service status.

[0132] When any of the core business indicators reaches the second threshold, the system continues to operate under fault conditions but triggers the generation of an alarm signal.

[0133] After the fault is terminated, a corresponding fault log is generated to analyze the fault operation and optimization plan.

[0134] This implementation method achieves tiered responses to core business metrics by setting two-tiered monitoring thresholds: a warning threshold triggers an alarm notification but does not terminate the test, while an emergency threshold triggers a fully automated, second-level loss mitigation process, including fault termination, service restoration, and log recording. This mechanism reduces manual response time from minutes to seconds, significantly shortening the fault impact cycle. After loss mitigation, a business impact report is automatically generated, quantifying the business losses and affected scope caused by the fault, providing data for subsequent test optimization, and improving the assessability of the testing process.

[0135] During a faulty operation, an intelligent loss mitigation mechanism needs to be designed to prevent normal business processes from being affected. This can be achieved through an intelligent loss mitigation module, which uses the logic of "real-time monitoring of business indicators → threshold judgment → automatic loss mitigation" to achieve a second-level response to the fault and prevent the abnormal escalation of core business operations. The implementation steps are illustrated below using an "e-commerce payment service" example:

[0136] Step 1: Configure core business metrics and stop-loss thresholds:

[0137] Implementation logic: Users configure "core business metrics" (strongly correlated with business value) and "stop-loss thresholds" (including warning thresholds and emergency thresholds) for each service node. The system collects metric data in real time through a monitoring platform (such as Prometheus) (collection frequency 1 second / time).

[0138] E-commerce payment service example: The core metrics and thresholds configured are shown in Table 4.

[0139] Table 4

[0140]

[0141] Step 2: Real-time indicator monitoring and anomaly detection:

[0142] Implementation logic: The system retrieves metric data in real time through the monitoring API, compares it with thresholds, and classifies it into three states: "Normal," "Warning," and "Urgent."

[0143] Normal: The indicators are within the normal range; continue processing the fault.

[0144] Warning: When an indicator triggers a warning threshold, an alarm notification (such as SMS or DingTalk) is sent, but the fault is not terminated.

[0145] Emergency: The indicator triggers the emergency threshold, and a stop-loss operation is executed immediately.

[0146] Step 3: Automatic stop-loss execution and fault rollback:

[0147] Execution logic: When the emergency threshold is triggered, the system implements stop-loss through the following 3 steps:

[0148] Send an "emergency stop loss command" to the Chaos Agent to terminate the current fault (such as shutting down network latency injection for payment services).

[0149] Call the service recovery API (such as restarting the payment service instance or cleaning up abnormal connections) to ensure that the service is restored to normal.

[0150] Record stop-loss logs (including fault type, trigger threshold, stop-loss time, and recovery time) for subsequent business impact analysis.

[0151] E-commerce example: When the payment service experiences a "100ms network latency" failure, monitoring shows that "the payment success rate drops from 99.98% to 99.88% (< the emergency threshold of 99.90%)", and the system immediately executes:

[0152] Within 1 second: Terminate network latency issues;

[0153] Within 3 seconds: Invoke the payment service to restart the API and clean up abnormal connections;

[0154] Within 5 seconds: Payment success rate recovers to 99.97%, loss stoppage complete.

[0155] Step 4: Business Impact Analysis After Stop-Loss:

[0156] Implementation logic: After the loss is stopped, the system automatically generates a "Fault-Business Impact" report, which includes information such as "fault duration, abnormal index magnitude, number of affected users, and estimated loss".

[0157] E-commerce example: A payment service outage lasted for 8 seconds, resulting in the failure of 12 order payments (a loss of approximately 2,400 yuan) and affecting 12 users. The report recommends that "the network latency threshold for the payment service can be adjusted to 80ms in subsequent tests".

[0158] Therefore, the key points of this application are as follows:

[0159] The automatic construction method of the business weight graph protects the "weight calculation method based on revenue share, user scope, and recovery cost" and the full-process implementation of "service call chain and business process mapping → weight value calculation → fault propagation coefficient evaluation", solving the problem that existing technologies "cannot quantify business priority".

[0160] The dynamic fault orchestration strategy based on business weights protects the collaborative logic of "weight value-fault risk level matching rules", "fault intensity dynamically adjusted with transmission coefficient", and "automatic selection of low load timing", solving the problems of "blind fault injection and vulnerability of core business" in existing technologies.

[0161] The intelligent loss prevention mechanism with monitoring linkage protects the "core business indicators - loss prevention threshold association configuration" and the "second-level loss prevention process after emergency threshold triggering (termination of fault → service recovery → log recording)," solving the problem of "loss prevention relying on manual intervention and slow response" in existing technologies.

[0162] In real-world scenarios, the solution presented in this application has been tested and has achieved the following significant effects:

[0163] Security: Risk of impact on core business reduced by over 90%;

[0164] Existing architecture orchestration schemes inject the same level of faults into the "payment service" and "log service", resulting in a high risk of core business interruption. This invention uses differentiated faults by matching business weights (e.g., injecting only low-risk delays into the payment service and high-risk downtime into the log service). According to actual testing in e-commerce systems, the anomaly rate of core business caused by testing has been reduced from "5%" to below "0.5%", reducing the impact risk by 90%.

[0165] Controllability: Fault response time has been reduced from "minutes" to "seconds";

[0166] Existing solutions require manual monitoring and stop-loss, with a response time of ≥5 minutes. This invention, through "real-time indicator collection → automatic threshold judgment → second-level stop-loss execution," reduces the stop-loss response time of e-commerce payment services to only 5 seconds, reduces the scope of fault impact (such as the number of failed orders) by 95%, and avoids the expansion of losses.

[0167] Practicality: Business impact is quantifiable, and the testing scope is precise and controllable;

[0168] Existing solutions can only output technical indicator reports (such as CPU utilization and interface latency), and cannot correlate them with business losses. This invention can quantify "revenue loss caused by failure and number of affected users", predict the risk level before testing (such as "a 100ms delay in payment service is expected to result in a loss of 2,000 yuan"), and output business optimization suggestions after testing. This upgrades chaos testing from "technical verification" to "business stability assurance tool", which is more in line with the actual needs of enterprises.

[0169] This application also provides a dynamic running device for chaos testing, comprising:

[0170] The processing module is used to collect mapping data of service call relationships and business processes corresponding to each service node, determine the business weight value of each service node by weighting according to multiple preset evaluation dimensions, determine the fault transmission coefficient between each service node, and construct a business weight map corresponding to each service node based on the mapping data, the business weight value and the fault transmission coefficient.

[0171] The processing module is further configured to match the fault risk level and select at least one fault event according to the business weight value of each service node, and perform a fault injection operation corresponding to each fault event on each service node.

[0172] The processing module is also used to monitor core business indicators in real time during fault operation and adjust the fault operation status according to the changes in each core business indicator.

[0173] This implementation constructs a business weight graph, mapping and associating service nodes with business processes. It calculates business weight values ​​based on multiple evaluation dimensions, including revenue share, user coverage, and fault recovery costs, quantifying differences in business priority and resolving the disconnect between fault injection target selection and business importance. By evaluating the fault propagation coefficient between service nodes, it accurately predicts the extent of fault spread in the dependency chain, providing a basis for dynamic orchestration. Based on business weight values, it automatically matches differentiated fault risk levels, ensuring that high-weight services are only injected with low-risk faults, avoiding impact on core businesses. Combined with a dual-threshold mechanism for real-time monitoring of core business metrics, it automatically terminates faults and restores services when metrics reach emergency thresholds, achieving second-level response and significantly reducing the scope of fault impact. This enhances the safety and controllability of chaos testing while ensuring business continuity.

[0174] As an optional implementation, the evaluation dimensions include business importance parameters, impact scope parameters, and fault recovery cost parameters. The specific method by which the processing module determines the business weight value of each service node based on multiple preset evaluation dimensions includes:

[0175] Based on the importance of the business operations supported by each service node, the business importance parameter is determined; based on the user coverage affected by each service node, the impact range parameter is determined; and based on the resource consumption for fault recovery of each service node, the fault recovery cost parameter is determined.

[0176] Assign corresponding weight coefficients to the business importance parameter, the scope of influence parameter, and the fault recovery cost parameter, and calculate the business weight value of each service node using weighted average.

[0177] This implementation method defines three evaluation dimensions: business importance parameter, impact scope parameter, and fault recovery cost parameter. It quantifies business operational value, user impact scale, and fault recovery resource consumption into calculable indicators, and then weights these indicators based on preset weight coefficients to form a comprehensive business weight value. This design covers key aspects of business value, ensuring that the weight calculation fully reflects the actual business priority of service nodes and avoids evaluation biases caused by relying solely on technical attributes. This provides a reliable basis for accurate matching of subsequent fault risk levels, improving the targeting and security of fault injection.

[0178] As an optional implementation, the processing module determines the specific method for determining the fault propagation coefficient between each of the service nodes, including:

[0179] Based on the service call relationship and the business process, determine the dependency type between each service node;

[0180] The dependency types include strong dependency, weak dependency, and no dependency.

[0181] Based on the dependency type, the range of the fault propagation coefficient is determined, and based on the service functions of each service node and the corresponding historical operation data, the value of the fault propagation coefficient is determined within the corresponding range.

[0182] This implementation analyzes service call relationships and business processes, classifying inter-node dependencies into three categories: strong dependencies, weak dependencies, and no dependencies. Based on these dependency types, it sets a range for the fault propagation coefficient. By combining historical operational data within this range, specific coefficient values ​​are determined, quantifying the degree of fault propagation in the service chain. This mechanism solves the problem of dynamically assessing the impact chain of faults, ensuring that potential impacts on downstream core services are predictable, providing data support for dynamically adjusting fault intensity, and enhancing the risk controllability of the testing process.

[0183] As an optional implementation, the processing module matches the fault risk level and selects at least one specific method for a fault event based on the business weight value of each service node, including:

[0184] Based on the negative correlation between the business weight value of each service node and the fault risk level, a target fault risk level corresponding to each service node is determined, and at least one fault event of fault type is selected from the target fault risk level.

[0185] Furthermore, if the fault transmission coefficient from the target service node to the core service node exceeds a set value, the fault intensity parameter corresponding to each fault event of the target service node is adjusted.

[0186] This implementation method automatically matches low-risk fault events to high-weight services and high-risk fault events to low-weight services based on a negative correlation mapping rule between business weight values ​​and fault risk levels, fundamentally preventing core businesses from being impacted by high-level faults. Simultaneously, when the fault propagation coefficient of a target service node to the core service exceeds a threshold, its fault intensity parameter is dynamically reduced, further blocking the propagation path of the fault to the core business. This collaborative logic solves the problem of blind fault injection and significantly reduces the risk of core business interruption.

[0187] As an optional implementation, the specific method by which the processing module performs fault injection operations corresponding to each fault event on each of the service nodes includes:

[0188] Determine the business load distribution based on historical business traffic data;

[0189] Based on the aforementioned business load distribution and each of the aforementioned fault events, the fault execution period is determined;

[0190] Based on the fault execution period, perform fault injection operations corresponding to each fault type on each of the service nodes.

[0191] This implementation method identifies periods of low system load by analyzing historical business traffic data and dynamically plans the timing of fault execution based on this. High-risk faults are avoided during peak business hours, and core service faults are avoided during critical business activity periods, ensuring that business continuity is not significantly affected during fault injection. This strategy, combined with fault level selection driven by business weights, forms a dual protection mechanism, further optimizing the impact of the testing process on actual business operations.

[0192] As an optional implementation, the specific method by which the processing module adjusts the fault operation status according to changes in each of the core business indicators includes:

[0193] When any of the core business metrics reaches the first threshold, the fault is terminated to restore normal service status.

[0194] When any of the core business indicators reaches the second threshold, the system continues to operate under fault conditions but triggers the generation of an alarm signal.

[0195] After the fault is terminated, a corresponding fault log is generated to analyze the fault operation and optimization plan.

[0196] This implementation method achieves tiered responses to core business metrics by setting two-tiered monitoring thresholds: a warning threshold triggers an alarm notification but does not terminate the test, while an emergency threshold triggers a fully automated, second-level loss mitigation process, including fault termination, service restoration, and log recording. This mechanism reduces manual response time from minutes to seconds, significantly shortening the fault impact cycle. After loss mitigation, a business impact report is automatically generated, quantifying the business losses and affected scope caused by the fault, providing data for subsequent test optimization, and improving the assessability of the testing process.

[0197] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0198] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the methods of any of the embodiments described above.

[0199] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0200] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0201] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the method provided in any embodiment.

[0202] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0203] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0204] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A chaotic test dynamic operation method, characterized in that, The method comprises the following steps: Collecting mapping data of service call relationships and business processes corresponding to each service node, determining business weight values of each service node according to preset evaluation dimensions, determining failure transmission coefficients between each service node, and constructing a business weight graph corresponding to each service node based on the mapping data, the business weight values and the failure transmission coefficients; According to the business weight values of each service node, matching the failure risk level and selecting at least one failure event, and performing a failure injection operation corresponding to each failure event on each service node; During the failure operation process, real-time monitoring of core business indicators, and adjusting the failure operation state according to the change of each core business indicator.

2. The method of claim 1, wherein, The evaluation dimensions include business importance parameters, influence range parameters and failure recovery cost parameters, and the business weight values of each service node are determined according to the preset evaluation dimensions, which includes: According to the business operation importance degree of each service node supporting business, the business importance parameter is determined, according to the user coverage range influenced by each service node, the influence range parameter is determined, and according to the resource consumption of failure recovery of each service node, the failure recovery cost parameter is determined; The business importance parameter, the influence range parameter and the failure recovery cost parameter are respectively assigned corresponding weight coefficients, and the business weight values of each service node are weighted calculated.

3. The method of claim 1, wherein, The determination of the failure transmission coefficient between each service node includes: According to the service call relationship and the business process, the dependence type between each service node is determined; Wherein, the dependence type includes strong dependence relationship, weak dependence relationship and no dependence relationship; According to the dependence type, the range of failure transmission coefficient is determined, and according to the service function of each service node and the corresponding historical operation data, the value of the failure transmission coefficient is determined in the corresponding range.

4. The method of claim 1, wherein, According to the business weight values of each service node, matching the failure risk level and selecting at least one failure event, which includes: According to the negative correlation between the business weight values of each service node and the failure risk level, the target failure risk level corresponding to each service node is determined, and at least one failure event of failure type is selected in the target failure risk level; And, if the failure transmission coefficient of the target service node to the core service node exceeds the set value, adjust the failure intensity parameter corresponding to each failure event corresponding to the target service node.

5. The method of claim 1, wherein, The failure injection operation corresponding to each failure event on each service node includes: According to the historical business traffic data, the business load distribution is determined; According to the business load distribution and each failure event, the failure execution period is determined; According to the failure execution period, the failure injection operation corresponding to each failure type is performed on each service node.

6. The method according to any one of claims 1 to 5, characterized in that, According to the change of each core business indicator, adjusting the failure operation state, including: When any core business indicator reaches the first threshold, terminate the failure to restore the normal service state; When any of the core business indicators reaches a second threshold value, continue the fault operation but trigger the generation of an alarm signal; After the fault is terminated, generate a corresponding fault log to analyze the fault operation situation and optimization scheme.

7. A chaotic test dynamic running device, characterized in that, Comprise: The processing module is used for collecting mapping data corresponding to service invocation relationship and business process of each service node, determining business weight value of each service node according to preset multiple evaluation dimensions weighting, determining fault transmission coefficient between each service node, and constructing business weight atlas corresponding to each service node based on the mapping data, the business weight value and the fault transmission coefficient; The processing module is also used for matching fault risk level and selecting at least one fault event according to the business weight value of each service node, and performing fault injection operation corresponding to each fault event on each service node; The processing module is also used for monitoring core business indicators in real time during the fault operation, and adjusting the fault operation state according to the change of each core business indicator.

8. The apparatus of claim 7, wherein, The evaluation dimensions include business importance parameter, influence range parameter and fault recovery cost parameter, and the specific way of the processing module for determining the business weight value of each service node according to the preset multiple evaluation dimensions weighting comprises: According to the business operation importance degree of each service node supporting business, the business importance parameter is determined, according to the user coverage range influenced by each service node, the influence range parameter is determined, and according to the resource consumption condition of fault recovery of each service node, the fault recovery cost parameter is determined; The business importance parameter, the influence range parameter and the fault recovery cost parameter are respectively assigned corresponding weight coefficients, and the business weight value of each service node is weighted calculated.

9. A computer device, comprising: One or more processors, and a memory, the memory has computer readable instructions stored therein, the computer readable instructions are executed by the one or more processors, and the steps of the method of any one of claims 1-6 are executed.

10. A storage medium, characterized by The storage medium has computer readable instructions stored therein, the computer readable instructions are executed by one or more processors, and the steps of the method of any one of claims 1-6 are executed.

Images