Automatic fault injection and recovery test method and system based on cloud native

Abstract

The invention belongs to the technical field of cloud computing and software testing, and particularly discloses an automatic fault injection and recovery testing method based on cloud native, which comprises the following steps: acquiring test basic data of a to-be-tested cloud native system; constructing a fault test execution model according to the test basic data; according to the fault test execution model, executing an automatic fault injection operation; generating an abnormal alarm log according to the system running state after the fault injection operation, and synchronously recording a fault injection operation log; according to the abnormal alarm log, executing automatic fault recovery operation; and generating a fault test report according to the fault injection operation log, the system monitoring data and the recovery operation process log. The invention aims to solve the problems of low automation degree, narrow scene coverage, weak monitoring evaluation and passive recovery in cloud native system fault test in the prior art, and provides a comprehensive, automatic, monitorable and evaluable fault injection and recovery test scheme.

Description

Technical Field

[0001] This invention relates to the fields of cloud computing and software testing technology, and in particular to a cloud-native automated fault injection and recovery testing method and system. Background Technology

[0002] With the rapid development of cloud computing technology and the widespread adoption of microservice architecture, system architectures are exhibiting characteristics of being distributed, loosely coupled, and multi-node, leading to an exponential increase in system complexity. Against this backdrop, systems face increasingly diverse failure scenarios in production environments, such as network latency, network outages, service instance crashes, database deadlocks, and configuration errors, placing extremely high demands on system reliability and resilience.

[0003] Traditional testing methods have limitations and cannot meet the testing requirements of cloud-native systems: they mainly rely on manual fault injection and recovery operations, which are time-consuming and error-prone; they can only simulate single fault types, such as single-machine service downtime, and cannot cover multi-dimensional complex faults at the network layer, hardware layer, and application layer; they lack real-time system status monitoring during fault injection, making it impossible to capture the impact of faults on performance indicators in a timely manner, and there is no standardized result analysis and reporting after testing; after a fault occurs, manual intervention is required to formulate recovery strategies, and the automatic triggering and execution of recovery operations cannot be achieved, prolonging the duration of system failures. Therefore, there is an urgent need for a testing method that can comprehensively simulate fault scenarios, automatically execute fault injection and recovery, monitor in real time, and conduct standardized evaluations, in order to expose potential system risks in advance and improve the fault tolerance and rapid recovery capabilities of cloud-native systems. Summary of the Invention

[0004] The main objective of this invention is to provide a cloud-native-based automated fault injection and recovery testing method and system, which aims to solve the technical problems mentioned in the background.

[0005] This invention proposes a cloud-native-based automated fault injection and recovery testing method, comprising: Obtain basic test data for the cloud-native system to be tested, including system resource metadata, service deployment data, and user fault testing requirements data; Based on the aforementioned test data, a fault test execution model is constructed, which includes fault parameter mapping rules, fault injection scheduling rules, and fault resource association graphs. Based on the fault test execution model, perform automated fault injection operations and record fault injection operation logs; Based on the system operating status after the fault injection operation, collect and analyze system monitoring data, generate an anomaly alarm log containing abnormal indicator information, anomaly occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously record the fault injection operation log. Based on the aforementioned abnormal alarm log, perform automated fault recovery operations; A fault test report is generated and archived based on the fault injection operation log, the system monitoring data, and the recovery operation process log.

[0006] The present invention is further configured such that the step of obtaining the test basic data of the cloud-native system to be tested includes: Obtain system resource metadata through cloud-native platform interfaces; Obtain service deployment data through the service registration component; The system receives user fault test requirement data through a visual configuration interface. The fault test requirement data includes fault execution sequence and test priority, and target fault type. The system resource metadata, service deployment data, and user fault test requirement data are subjected to consistency verification. The correspondence between service instances and container instances, the scope of fault impact, and the matching of system resources are checked. If inconsistencies are found, a prompt message is generated and guidance is provided for correction. The verified system resource metadata, service deployment data, and user fault test requirement data are stored in the corresponding directories of the system configuration center, and a unique identifier ID is generated for each type of data.

[0007] The present invention is further configured such that, based on the aforementioned test baseline data, the step of constructing a fault test execution model including fault parameter mapping rules, fault injection scheduling rules, and a fault resource association graph includes: Based on the relationship between system resource metadata and service deployment data, establish a mapping rule between fault types and fault parameters of cloud-native fault injection tools; Based on the fault execution sequence and test priority of the user fault test requirement data, and in conjunction with the fault parameter mapping rules, a fault injection scheduling rule is formulated. Based on the system resource metadata and the service deployment data, construct a fault resource association map between the fault impact range and physical resources; The validity of the fault parameter mapping rule, the fault injection scheduling rule, and the fault resource association graph is verified. The verified fault parameter mapping rules, fault injection scheduling rules, and fault resource association graph are integrated into a fault test execution model.

[0008] The present invention is further configured such that, according to the fault test execution model, the steps of performing automated fault injection operations include: Based on the fault parameter mapping rules and the target fault type in the user's fault testing requirements, the corresponding cloud-native fault injection tool and calling parameters are matched to generate specific fault injection instructions. Based on the fault resource association graph, candidate target resources are determined, their current running status is queried and invalid resources are eliminated to determine the target resource set for fault injection. According to the fault injection scheduling rules, the task triggering conditions are monitored through the timed task framework, the API interface of the fault injection tool is called in order of priority, and the fault injection operation is performed on the target resource set according to the fault injection instructions. Obtain the execution result of the fault injection operation. If the execution is successful, record the fault identifier, target resource, start time and execution status to the fault injection operation log. If the execution fails, record the reason for failure and trigger the retry mechanism. If the retry fails after a preset number of times, generate an alarm message. After the fault injection operation is executed, the status changes of the target resource are monitored in real time to verify whether the fault has taken effect successfully. If it has not taken effect, the cause is analyzed and the parameters are adjusted and the operation is re-executed until the fault takes effect or the operation limit is reached.

[0009] The present invention is further configured to, based on the fault test execution model, collect and analyze system monitoring data, generate anomaly alarm logs containing abnormal indicator information, anomaly occurrence time, associated fault identifiers, fault types, and associated resource identifiers, and simultaneously record fault injection operation logs, including the following steps: Based on the hierarchical structure of the cloud-native system to be tested, deploy corresponding monitoring agents in the infrastructure layer, container runtime layer, and application service layer, and configure the metric collection range, frequency, and transmission method for each layer. Obtain historical system indicator data before fault injection, perform data cleaning and aggregation, use statistical analysis methods to calculate the baseline threshold range of each indicator, and generate system indicator baseline data; After the fault injection operation begins, real-time monitoring metrics of the cloud-native system to be tested are collected and transmitted to the data processing center for aggregation and verification via a message queue. The real-time monitoring indicators are compared and analyzed with the baseline data of the system indicators to identify abnormal indicators that exceed the baseline threshold range. The scope of the abnormal impact is located by combining the fault resource association map and an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier is generated. Perform in-depth analysis on abnormal alarm logs and associated real-time monitoring metrics, calculate the deviation of abnormal metrics, and mark the scope of impact on downstream services. Record the abnormal metrics, deviation, and scope of impact in the fault injection operation log.

[0010] The present invention is further configured such that the step of performing automated fault recovery operation based on the abnormal alarm log includes: Construct a predefined fault recovery strategy library, which stores the correspondence between fault types and recovery strategies, as well as the execution priority rules and triggering conditions of the recovery strategies; Extract the fault type, fault duration, and associated resource identifier from the abnormal alarm log; match the recovery strategy that meets the triggering conditions in the fault recovery strategy library; and select the optimal recovery strategy according to the execution priority rule. The optimal recovery strategy is used to call the cloud-native platform interface to perform recovery operations, including service instance restart, traffic routing switching, configuration version rollback, and resource quota adjustment. Monitor the execution status of recovery operations in real time and record the recovery operation process log; Verify the recovery effect of the recovery operation. If the recovery fails, a secondary recovery mechanism is triggered. After multiple recovery failures, a manual intervention work order is generated.

[0011] The present invention is further configured such that the step of generating and archiving a fault test report based on the fault injection operation log, system monitoring data, and recovery operation process log includes: The fault injection operation log, the system monitoring data, and the recovery operation process log are linked together using a unique fault identifier to form a fault test data chain. Calculate the performance impact during the failure, failure detection time, recovery execution time, and total recovery time, and generate a system resilience score based on the scope of the failure impact and the recovery effect; Generate a standardized test report based on a preset template, which includes basic test information, fault injection details, monitoring indicator analysis, recovery operation details, and system resilience score; The standardized test report, the fault injection operation log, the system monitoring data, the recovery operation log, and the anomaly alarm log are associated and stored in a designated storage system, generating a unique archive ID and associating it with the test management system.

[0012] This invention also provides a cloud-native-based automated fault injection and recovery testing system, comprising: The data acquisition module is used to acquire basic test data for the cloud-native system under test, including system resource metadata, service deployment data, and user fault test requirement data. The model building module is used to build a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules and fault resource association graphs based on the test basic data. The fault injection module is used to perform automated fault injection operations according to the fault test execution model and record fault injection operation logs. The monitoring and alarm module is used to collect and analyze system monitoring data based on the system operating status after the fault injection operation, generate an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously record the fault injection operation log. The fault recovery module is used to perform automated fault recovery operations based on the abnormal alarm logs; The report generation module is used to generate and archive a fault test report based on the fault injection operation log, the system monitoring data, and the recovery operation process log.

[0013] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a cloud-native automated fault injection and recovery testing method.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a cloud-native automated fault injection and recovery testing method.

[0015] The beneficial effects of this invention are as follows: This invention aims to solve the problems of low automation, narrow scenario coverage, weak monitoring and evaluation, and passive recovery in existing technologies for fault testing of cloud-native systems. It provides a comprehensive, automated, monitorable, and evaluable fault injection and recovery testing solution, specifically including automatic injection of multiple types of fault scenarios, support for user-defined fault parameters and priorities; real-time monitoring of multi-dimensional indicators of the system during fault injection, timely identification of anomalies and triggering alarms; automatic execution of fault recovery operations based on predefined strategies to ensure priority recovery of core services; and generation of standardized test reports containing fault impact and recovery effects, providing data support for system optimization. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a method flow according to an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the system structure according to an embodiment of this application.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] like Figure 1As shown, this application provides a cloud-native-based automated fault injection and recovery testing method, including: S1, obtain the basic test data of the cloud-native system to be tested, including system resource metadata, service deployment data and user fault test requirement data; S2, Based on the aforementioned test data, construct a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules, and fault resource association graph; S3. Based on the fault test execution model, perform automated fault injection operations and record the fault injection operation log; S4. Based on the system operating status after the fault injection operation, collect and analyze system monitoring data, generate an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously record the fault injection operation log. S5. Based on the abnormal alarm log, perform automated fault recovery operations; S6. Based on the fault injection operation log, the system monitoring data, and the recovery operation process log, generate a fault test report and archive it.

[0021] As described in steps S1-S6 above, this invention achieves the simulation injection, real-time monitoring, automatic recovery, and standardized evaluation of multiple types of faults in cloud-native systems through a fully automated design that includes acquiring basic test data, constructing a fault test execution model, performing fault injection, monitoring and alarming, automated recovery, and report archiving. Ultimately, this aims to improve the reliability, elasticity, and fault tolerance of cloud-native systems, reduce manual intervention, and increase the efficiency of fault testing.

[0022] With the widespread adoption of cloud computing and microservice architectures, cloud-native systems exhibit characteristics of distribution, loose coupling, and multiple nodes, significantly increasing system complexity. Production environments may face diverse failure scenarios such as network latency, service outages, and database deadlocks, placing extremely high demands on system reliability and resilience. Traditional testing methods rely on manual operation, which is not only inefficient and error-prone but also lacks comprehensive coverage of diverse failure scenarios, failing to fully simulate multi-dimensional and complex failures. Furthermore, the lack of real-time monitoring and standardized evaluation of the fault injection and recovery process makes the system prone to unforeseen problems during actual operation. Therefore, there is an urgent need for a comprehensive, automated, monitorable, and evaluable fault injection and recovery testing method, and this invention is a technical solution proposed to address this problem.

[0023] Traditional solutions for manual fault injection require operations personnel to manually operate cloud-native platform interfaces (such as manually calling the Kubernetes API to delete Pods to simulate service downtime), which is not only time-consuming but also prone to errors that could lead to biased test results. Existing monitoring tools can only collect basic metrics and cannot be linked with fault injection and recovery processes, making it difficult to form a closed-loop test. Furthermore, the lack of standardized test models and reports prevents test results from being effectively reused and analyzed. To address these shortcomings, this invention proposes a solution based on "basic test data as input, a fault test execution model as the core, and full-process automation as the means." Through structured data collection, model-based fault management, automated process execution, and standardized report archiving, it systematically solves the deficiencies of traditional methods.

[0024] Specifically, the first step, "obtaining the basic test data of the cloud-native system to be tested, including system resource metadata, service deployment data, and user fault test requirement data," is to provide complete and accurate basic data support for subsequent fault testing, ensuring that fault scenarios can be accurately mapped to actual cloud-native resources. From the perspective of data acquisition methods, system resource metadata is obtained through cloud-native platform interfaces (such as the `kubectlgetnodes` and `kubectlgetpods` interfaces of the Kubernetes API), including the hardware configuration, running status, and container instance identification information, resource quotas, and service information of the container. For example, by calling the interface, the name of a Pod instance, its Deployment, and CPU / memory quota can be extracted. Service deployment data is obtained through service registration components (such as Nacos and Eureka), covering the registration information, instance list, health status, and inter-service call relationships of microservices. For example, the Nacos interface can be used to query all instance IPs, ports, and dependent service B information of service A. User fault test requirement data is received through a visual configuration interface, including fault execution sequence, test priority, target fault type (such as network latency, service downtime), fault impact scope (such as all instances of specified service A), and fault duration. For example, users can set "Inject 200ms network latency into service A at 14:00 for 20 minutes, with high test priority" through the interface. The significance of this step lies in the fact that only by clearly defining the resource distribution (system resource metadata), service architecture (service deployment data), and user testing objectives (user fault testing requirement data) of the cloud-native system can we ensure that subsequent fault injection does not deviate from the actual scenario and avoid errors in fault injection targets or failure to achieve testing objectives due to missing data. For example, if the service information to which the container belongs is not obtained, there may be a problem of including containers that are not part of the target service in the fault injection scope. The structured data collection in this step can effectively avoid such risks and lay a data foundation for subsequent model building.

[0025] The second step, "Based on the aforementioned test foundation data, constructing a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules, and a fault resource association graph," transforms the scattered test foundation data into a machine-executable fault test logic framework, solving the problems of inconsistent management and chaotic execution order in traditional testing of fault scenarios. From a technical implementation perspective, the fault parameter mapping rules are established based on the association between system resource metadata and service deployment data. For example, combining the network configuration of containers in the system resource metadata and the average response time of services in the service deployment data, the ChaosMesh tool is matched to the "network latency" fault, and calling parameters such as `latency` (latency duration) and `selector` (target container selector) are defined to ensure that the fault injection tool can accurately target the target resource. The fault injection scheduling rules are formulated based on the fault execution sequence and test priority in the user's fault test requirement data. For example, "high-priority" test tasks are converted into a numerical weight of 100. The "medium priority" is converted to 50, and task sorting is achieved through Redis sorted sets. At the same time, the fault triggering time is set using the Quartz scheduling framework. If there are related tasks that "inject database fault first and then inject dependent service fault", a task dependency mechanism will be established to ensure that subsequent tasks are triggered only after the preceding fault is completed. The fault resource association graph is built based on system resource metadata and service deployment data. Container instances are used as nodes and labeled with attributes such as their respective services and node IPs. Service call relationships are used as edges and labeled with information such as communication protocols and ports. For example, the graph can clearly show that there is an HTTP call relationship between the Pod instance of service A and the Pod instance of service B. The significance of this step lies in building a unified fault test execution model, which solidifies the logic of "fault type-tool-parameter", "task priority-execution time", and "resource-service association", avoiding problems such as incorrect fault injection tool calls, chaotic task execution order, and uncontrolled fault impact scope caused by the lack of a unified framework in traditional testing. For example, when it is necessary to inject a "service downtime" fault, the model can directly match the Kubernetes API tools and `poddeletion` command parameters through fault parameter mapping rules, locate the target Pod through the fault resource association graph, and determine the execution time through fault injection scheduling rules, ensuring that the fault injection process is repeatable and controllable.

[0026] The third step, "According to the fault test execution model, perform automated fault injection operations and record fault injection operation logs," transforms the logic in the fault test execution model into actual fault injection actions, thereby achieving automatic simulation of fault scenarios. In terms of technical implementation, firstly, based on the fault parameter mapping rules in the fault test execution model and combined with the target fault type in the user's fault test requirements, the corresponding cloud-native fault injection tool and its calling parameters are matched. For example, when the target fault type is "service downtime", the Kubernetes API tool is matched, and the calling parameters include the target Pod name, namespace, and graceful exit time obtained from the fault resource association graph. Next, the target resource set is determined according to the fault resource association graph, and the current running status of the target resources is queried (e.g., querying whether the Pod is in the "Running" state via the Kubernetes API). After removing invalid resources, the final target resource list is formed. Then, according to the fault injection scheduling rules, the trigger condition is listened to through the scheduled task framework. When the preset time is reached and there are no high-priority tasks, the API interface of the fault injection tool is called to perform the fault injection operation, such as calling the `kubectldeletepod` command of the Kubernetes API to delete the target Pod. Finally, the fault injection operation log is recorded, including the fault identifier, target resource, start time, execution status, and tool calling parameters. The significance of this step lies in replacing manual operation with automated execution, solving the problems of low efficiency and error-proneness of traditional manual injection. At the same time, by recording operation logs, the fault injection process can be traced. For example, when fault injection fails, the logs can be used to find out the reasons such as incorrect tool call parameters or non-existent target resources, which facilitates quick troubleshooting. In addition, automated execution can also ensure the consistency of execution of the same fault scenario at different times and under different environments, improving the credibility of test results.

[0027] The fourth step, "Based on the system's operating status after the fault injection operation, collect and analyze system monitoring data, generate an anomaly alarm log containing anomaly indicator information, anomaly occurrence time, associated fault identifier, fault type, and associated resource identifier, and simultaneously record the fault injection operation log," is to capture the impact of faults on the system in real time, promptly identify anomalies and associate fault information, and solve the problem of disconnect between monitoring and fault injection in traditional testing. In terms of technical implementation, monitoring agents are first deployed at the infrastructure layer (nodes), container runtime layer (containers), and application service layer (microservices) of the cloud-native system under test. For example, NodeExporter is deployed at the node layer to collect CPU and memory usage, cAdvisor is deployed at the container layer to collect container resource usage, and the application layer uses OpenTelemetrySDK to collect metrics such as request latency and error rate. Next, historical system metrics data for the 7 days prior to fault injection are obtained, and baseline threshold ranges for each metric are calculated using statistical analysis methods (such as the 3σ principle) to generate baseline system metrics data. For example, the baseline threshold for request latency of service A is [50ms, 150ms]. After fault injection, the monitoring agents collect real-time monitoring metrics at a preset frequency (e.g., 1 second) and transmit them to the data processing center via a Kafka message queue for aggregation and verification to ensure data accuracy. Then, the real-time monitoring metrics are compared with the baseline system metrics data. If the real-time value exceeds the threshold (e.g., the request latency of service A exceeds 200ms for 10 seconds), it is identified as an abnormal metric. The fault resource association graph locates the scope of the anomaly's impact and generates anomaly alarm logs. These logs include anomaly indicator information (e.g., "request delay 220ms"), anomaly occurrence time (e.g., "2001-01-01 01:00:30"), associated fault identifiers (e.g., "fault-001"), fault type (e.g., "network latency"), and associated resource identifiers (e.g., "pod-service-A-xxx"). Simultaneously, the anomaly information is recorded in the fault injection operation log. The significance of this step lies in its ability to promptly identify the impact of faults on system performance through multi-dimensional real-time monitoring and anomaly analysis. This avoids the problem of not being able to monitor the system status in real time after fault injection, as is common in traditional testing. For example, if a network latency fault is injected, without real-time monitoring, it might be impossible to know that the service response time has exceeded the acceptable range for business operations. This step quickly identifies the anomaly and generates alarm logs with fault association information, providing accurate fault location data for subsequent automated recovery. Simultaneous log recording ensures the integrity of the fault "injection-anomaly" causal chain, facilitating subsequent analysis of the fault's impact.

[0028] The fifth step, "Execute automated fault recovery operations based on the aforementioned anomaly alarm logs," automatically triggers recovery actions after identifying fault anomalies, addressing the issues of manual and time-consuming fault recovery in traditional testing. Technically, firstly, based on the fault type and associated resource identifiers in the anomaly alarm logs, a predefined fault recovery strategy library is matched. This strategy library stores the correspondence between fault types and recovery operations, along with their execution priorities. For example, a "network latency" fault corresponds to the recovery operation "switch traffic to a backup link via Istio," with a priority of 80. Next, the cloud-native platform interface is called to execute the recovery operation. For instance, for a "network latency" fault, the Istio API is called to update the VirtualService configuration, routing traffic from service A to the backup instance. During the recovery process, the operation execution status is monitored in real time, and recovery operation logs are recorded, including the operation start time, execution steps, and intermediate states. After the recovery operation is completed, service health check interfaces (such as the ` / health` interface) and monitoring metric collection interfaces are called to verify the recovery effect. If the service health status is "UP" and the metrics return to within the baseline threshold, the recovery is considered successful. The significance of this step lies in the fact that automated recovery operations can significantly shorten the duration of a fault and reduce its impact on the system. For example, when service A experiences an anomaly due to network latency, traditional manual recovery requires maintenance personnel to log into the system to troubleshoot the fault, formulate a recovery plan, and execute the operation, which may take more than 10 minutes. However, this step can automatically complete the traffic switch within 1 minute, restoring service A's request latency to normal. At the same time, the recording of the recovery operation process log can provide a basis for subsequent evaluation of the recovery effect, ensuring that the recovery process is traceable and analyzable.

[0029] Step 6, "Generate and archive a fault test report based on the fault injection operation log, the system monitoring data, and the recovery operation log," aims to standardize the summary and storage of the entire fault testing process, solving the problems of no unified report and difficulty in data reuse in traditional testing. In its technical implementation, firstly, a unique fault identifier (e.g., "fault-001") is used to associate the fault injection operation log, system monitoring data (indicators before / during / after the fault), and recovery operation log to form a complete fault test data chain. Then, the associated data is analyzed to calculate the performance impact during the fault (e.g., "request latency increased by 120% compared to the baseline"), fault detection time (e.g., "anomaly alarm generation time - fault injection start time = 5 seconds"), recovery execution time (e.g., "recovery operation completion time - alarm generation time = 30 seconds"), and total recovery time. Simultaneously, a system fault test report is generated based on the fault impact scope (e.g., "affecting 2 Pod instances of service A") and recovery effect (e.g., "indicator recovery rate 95%)." The system first performs a resilience score. Then, a standardized test report is generated based on a pre-defined template. This report includes basic test information (test name, execution time, environment identifier), fault injection details (fault type, target resource, execution result), monitoring indicator analysis (comparison charts of indicators before and after the fault), recovery operation details (execution steps, time consumption), and a system resilience score. Finally, the test report and all-process log data (fault injection operation logs, system monitoring data, recovery operation process logs, and anomaly alarm logs) are associated and stored in a designated storage system (e.g., InfluxDB for time-series data, MySQL for structured logs), generating a unique archive ID and associating it with a test management system (e.g., JIRA) to achieve full lifecycle traceability of test records. The significance of this step is that the standardized test report clearly presents the entire process and results of the fault test, providing data support for system optimization. For example, the analysis conclusion in the report that "recovery decision-making time accounts for 40%" can guide operations personnel to optimize recovery strategy templates to shorten decision-making time. Data archiving ensures that test records are reusable, facilitating subsequent reproduction of fault scenarios or comparison of the resilience performance of different system versions, avoiding the problems of traditional testing where test results rely solely on manual recording, are easily lost, and difficult to trace.

[0030] In one embodiment of the present invention, the step of obtaining the test basic data of the cloud-native system to be tested includes: S11 obtains system resource metadata through cloud-native platform interfaces; S12, obtain service deployment data through the service registration component; S13, Receive user fault test requirement data through a visual configuration interface, the fault test requirement data includes fault execution sequence and test priority, target fault type; S14, perform consistency verification on the system resource metadata, the service deployment data and the user fault test requirement data, check the correspondence between service instances and container instances, the fault impact scope and the matching of system resources, and if there is any inconsistency, generate prompt information and guide the correction. S15, the verified system resource metadata, service deployment data and user fault test requirement data are stored in the corresponding directories of the system configuration center, and a unique identifier ID is generated for each type of data.

[0031] As described in steps S11-S15 above, by designing steps such as obtaining test basic data from different sources, multi-dimensional consistency verification, and structured storage and archiving, we can achieve accurate collection, validity verification, and standardized management of test basic data (system resource metadata, service deployment data, and user fault test requirement data). Ultimately, this provides complete, reliable, and consistent data input for the construction of the fault test execution model, ensuring that subsequent fault injection, monitoring, recovery, and other full-process operations can accurately match the actual resources of the cloud-native system and the user's test objectives.

[0032] Test foundation data forms the basis of the entire automated fault injection and recovery testing process. System resource metadata reflects the distribution of hardware and container resources in the cloud-native system and serves as the basis for identifying the "target" of fault injection. Service deployment data reflects the architecture and dependencies of microservices and is crucial for determining the scope of fault impact and recovery strategies. User fault testing requirement data clarifies the testing objectives and is the core guideline for fault scenario design. Due to the dynamic changes in resources and services in cloud-native systems (such as frequent creation and destruction of Pod instances and adjustments to service dependencies), and the potential for non-standard parameters in user testing requirements (such as the scope of fault impact exceeding actual resources), directly using unprocessed data to build models can lead to incorrect fault injection targets (such as injecting non-existent containers) and chaotic testing logic (such as unclear service dependencies leading to biased fault impact assessments). Therefore, it is necessary to implement precise acquisition, verification, and management of test foundation data through specialized steps.

[0033] Traditional solutions rely heavily on manual collection and organization of basic test data. For example, operations personnel manually query and record node and Pod information through the cloud-native platform console, export service lists from the service registry interface, and then manually organize the data into tables. This process is not only inefficient but also prone to data omissions or errors due to human error. Furthermore, the lack of a unified consistency verification mechanism means that if a user-inputted fault impact includes offline service instances, manual identification is difficult, leading to subsequent fault injection failures. Additionally, data storage lacks standardized specifications, often scattered across Excel spreadsheets or local documents, making it difficult for automated modules to access and hindering data traceability. To address these shortcomings, this invention proposes a solution combining source-based automated data acquisition, multi-dimensional verification, and structured storage. It achieves automatic data collection through integration with cloud-native platform interfaces and service registration components, eliminates invalid data through consistency verification, and implements standardized data management through a system configuration center, thus resolving the efficiency and reliability issues of traditional methods.

[0034] Specifically: The first step, "Obtaining System Resource Metadata via Cloud-Native Platform Interfaces," focuses on automatically collecting infrastructure and container resource information from the cloud-native system. This provides a basis for locating target resources during subsequent fault injection. Technically, cloud-native platform interfaces primarily refer to the Kubernetes API (if the system is deployed on Kubernetes). By calling the `kubectlgetnodes-ojson` interface, information such as hardware configuration (e.g., number of CPU cores, memory capacity), running status (e.g., "Ready" "NotReady"), and node IP address of each node in the cluster can be extracted, generating a node resource data table. Similarly, by calling the `kubectlgetpods-n{namespace}-ojson` interface, all container instances are traversed by namespace, yielding the name, container ID, Deployment (service identifier), resource quotas (CPU request / limit, memory request / limit), and running status (e.g., "Running" "Terminating") of each Pod, generating a container resource data table. Finally, the two types of data tables are integrated into complete system resource metadata through the association field of "node IP - Pod's node." For example, in a Kubernetes cluster, this step can retrieve information such as the IP address of node "node-01" ("10.0.0.1"), the number of CPU cores (8), and the name of the Pod running on that node ("service-A-xxx"), its Deployment ("service-A"), and its CPU quota ("1C / 2C"). The significance of this step lies in replacing manual queries with automated API calls. This not only significantly improves data collection efficiency (e.g., resource data for a 100-node cluster can be collected within one minute, whereas manual collection would take several hours), but also ensures the real-time nature and accuracy of the data. It avoids the problem of "resources being offline but data not updated" caused by manual recording. For instance, when a Pod instance is deleted, the next API call can synchronously update the system resource metadata, ensuring that subsequent fault injection will not locate invalid Pods.

[0035] The second step, "obtaining service deployment data through the service registration component," aims to automatically collect microservice registration information, instance distribution, and dependencies, providing service-level data support for assessing the scope of fault impact and formulating recovery strategies. In the technical implementation, service registration components include mainstream tools such as Nacos and Eureka. Taking Nacos as an example, by calling the ` / nacos / v1 / ns / catalog / services` interface, the names (e.g., "service-A", "service-B"), service groups (e.g., "dev", "prod"), and registration status (e.g., "healthy", "unhealthy") of all registered microservices in the system under test can be obtained. For each microservice, calling the ` / nacos / v1 / ns / instance / list` interface can extract the IP addresses, service ports, instance weights (e.g., "1.0"), and online times of all instances under it, generating a microservice instance data table. At the same time, through the `istioctlgetvirtualservices-oyaml` interface of the service mesh component (e.g., "traffic of service-A is routed to port 8080 of service-B"), the traffic routing configuration of each microservice can be obtained (e.g., "traffic of service-A is routed to port 8080 of service-B"), thereby sorting out the call relationships between services, constructing a service dependency graph in combination with the microservice instance data table, and finally integrating it into service deployment data. For example, this step clarifies that "service-A" contains three healthy instances (IPs "10.0.1.1", "10.0.1.2", and "10.0.1.3" respectively), and that "service-A" depends on the ` / api / query` interface of "service-B". The significance of this step lies in solving the problems of low efficiency and easy omissions in traditional manual service relationship analysis—cloud-native systems may have dozens or even hundreds of microservices, making it difficult for humans to fully grasp the instance distribution and dependency chains of all services. However, through the interface linkage between the service registration component and the service mesh component, complete service deployment data can be automatically constructed. For example, when "service-A" adds one instance, the service registration component interface will synchronize this information in real time, ensuring that the service deployment data is consistent with the actual service status, providing accurate service-level data for subsequent construction of a fault resource association graph (clarifying the correspondence between services and containers).

[0036] The third step, "receiving user fault test requirement data through a visual configuration interface, including fault execution sequence and test priority, and target fault type," transforms the user's testing intent into standardized structured data, providing clear objectives for subsequent fault scenario design. In terms of technical implementation, the visual configuration interface needs to provide interactive controls that conform to user operating habits: fault type selection options (drop-down list, including subcategories such as network latency, service downtime, database deadlock, and configuration errors, consistent with the fault injection type range mentioned above), fault impact range selector (supports selection by microservice name, container instance identifier, and node IP address; options are derived from acquired system resource metadata and service deployment data to avoid invalid user input), fault execution timing settings (supports "timed trigger" and "instant trigger"; timed trigger requires inputting a time in the format "YYYY-MM-DDHH:MM:SS," while instant trigger defaults to the current time), test priority selection options (radio buttons, divided into "high," "medium," and "low," used for subsequent fault injection scheduling), and parameter settings corresponding to the target fault type (e.g., "network latency" requires inputting latency duration, packet loss rate, etc.). After the user completes the configuration, the interface will convert the input parameters into structured data in JSON format, generating the user's fault test requirement data. For example, a user selects "Fault Type: Network Delay," "Affected Scope: All instances of service-A," "Execution Time: 2000-01-01 01:00 (Timed Trigger)," "Test Priority: High," and "Delay Duration: 200ms" through the interface. The interface then encapsulates these parameters into standardized user fault test requirement data. The significance of this step lies in avoiding unusable requirement data due to non-standard user input (such as misspelled fault types or incorrect time formats) through a visual interface and standardized parameter constraints. For instance, when a user inputs "Fault Type: Network Delay," the interface only provides preset parameter input items (delay duration, packet loss rate), ensuring that users do not need to remember complex parameter formats. Simultaneously, the options are derived from acquired system resource metadata and service deployment data (e.g., "Affected Scope" only displays the names of existing microservices), directly preventing users from selecting invalid scopes. For example, if "service-X" does not exist in the system, the dropdown menu will not display this option, reducing the correction cost of subsequent consistency checks from the source.

[0037] The fourth step, "Perform consistency verification on the system resource metadata, service deployment data, and user fault test requirement data, checking the correspondence between service instances and container instances, the scope of fault impact, and the matching of system resources. If inconsistencies exist, generate a prompt message and guide correction," eliminates data contradictions through multi-dimensional verification, ensuring the logical consistency of the three types of basic test data and providing reliable input for subsequent model construction. In the technical implementation, consistency verification includes two core dimensions: First, the correspondence verification between service instances and container instances, which checks whether the IP address and port of each microservice instance in the service deployment data can be found in the system resource metadata as a corresponding container instance (associated through "container service information"). For example, if the IP of a certain instance of "service-A" in the service deployment data is "10.0.1.1," it is necessary to verify whether there is a Pod instance in the system resource metadata with IP "10.0.1.1" and "container service information" as "service-A." If not, it is determined that "the service instance has no corresponding container instance," and a prompt message "service-A instance 10.0.1" is generated. 1. No valid Pod was matched. Please check the service registration data or container deployment status. 2. Verify the compatibility between the scope of the fault and system resources. This involves checking whether the scope of impact specified in the user's fault test requirement data (such as "all instances of service-A" or "node 10.0.0.1") is fully included within the valid resource scope of the system resource metadata. For example, if the user specifies "scope of impact: node 10.0.0.5", it is necessary to verify whether the node exists in the system resource metadata. If it does not exist, a prompt message will be generated: "The specified node 10.0.0.5 is not in the current cluster. The currently valid nodes are 10.0.0.1, 10.0.0.2, and 10.0.0.3", and the user will be guided to select again. The significance of this step lies in eliminating data logic contradictions in advance, avoiding problems such as "the fault resource association graph cannot associate services and containers" or "the target resource for fault injection does not exist" when building the fault test execution model later. For example, if the correspondence between service instances and container instances is not verified, it may be impossible to determine the container selector corresponding to "service-A" when building fault parameter mapping rules later, causing the fault injection tool to be unable to locate the target resource. By verifying through this step, data problems can be corrected before model building, greatly reducing the risk of failure in subsequent processes. At the same time, the generated prompt information can accurately guide users in the direction of correction, avoiding users repeatedly checking for data contradictions.

[0038] The fifth step, "Storing the verified system resource metadata, service deployment data, and user fault test requirement data to their respective directories in the system configuration center, and generating unique identifiers (IDs) for each type of data," is to achieve structured archiving and traceable management of basic test data, ensuring that subsequent modules can efficiently access data and associate it with its source. Technically, the system configuration center can employ distributed configuration management tools (such as Spring Cloud Config or Apollo), dividing storage directories by data type: system resource metadata is stored in the "resource / metadata" directory, service deployment data in the "service / deployment" directory, and user fault test requirement data in the "fault / test / requirement" directory. Simultaneously, a unique identifier (e.g., using UUID format) is generated for each type of data, which may include the data generation time (e.g., "200101010100-res-meta-xxx"), facilitating subsequent tracing of data generation time and type. In addition, the relationships between data must be recorded during storage. For example, the identifier ID of user fault test requirement data must be associated with the identifier ID of its dependent system resource metadata and the identifier ID of service deployment data to ensure that the resource and service background corresponding to the requirement data can be quickly located during subsequent queries. For example, the "service-A network latency test" requirement data that has passed verification has the identifier ID "200101010100-fault-req-001", the associated system resource metadata ID is "200101010050-res-meta-001", and the service deployment data ID is "200101010055-service-deploy-001", and is stored in the "fault / test / requirement" directory. The significance of this step lies in solving the problems of low retrieval efficiency and difficulty in tracing caused by traditional decentralized data storage. When building the fault test execution model, the model building module can directly call the specified data through the directory path and identifier ID of the system configuration center without re-collection. For example, when it is necessary to obtain the resource data corresponding to the requirement "200101010100-fault-req-001", it can be read directly from the "resource / metadata" directory through the associated resource metadata ID, which greatly improves the data retrieval efficiency. At the same time, the unique identifier ID and the associated relationship record can realize the traceability of the entire data lifecycle. For example, when the subsequent fault injection fails, the identifier ID of the requirement data can be used to trace its corresponding resource and service data to check whether the fault injection object is wrong due to untimely data updates, ensuring the traceability and reproducibility of the entire testing process.

[0039] In one embodiment of the present invention, the step of constructing a fault test execution model including fault parameter mapping rules, fault injection scheduling rules, and fault resource association graphs based on the test basic data includes: S21. Based on the association between system resource metadata and service deployment data, establish a mapping rule between fault types and fault parameters of cloud-native fault injection tools; S22, Based on the fault execution sequence and test priority of the user fault test requirement data, and in conjunction with the fault parameter mapping rules, formulate fault injection scheduling rules; S23, Based on the system resource metadata and the service deployment data, construct a fault resource association map between the fault impact range and physical resources; S24, verify the validity of the fault parameter mapping rule, the fault injection scheduling rule, and the fault resource association graph; S25, the verified fault parameter mapping rules, fault injection scheduling rules, and fault resource association graph are integrated into a fault test execution model.

[0040] As described in steps S21-S25 above, by establishing fault parameter mapping rules, formulating fault injection scheduling rules, constructing fault resource association graphs, validating the effectiveness, and integrating the models, the test base data (system resource metadata, service deployment data, and user fault test requirement data) verified in the aforementioned steps are transformed into a structured and executable fault test execution model. This ultimately provides clear tool matching logic, task scheduling order, and resource association basis for automated fault injection operations, ensuring that the fault injection process is accurate, orderly, and controllable. At the same time, it lays a unified model foundation for subsequent monitoring, alarming, and automated recovery.

[0041] Fault testing in cloud-native systems needs to address three core questions: which tools to use for fault injection, when to inject faults, and which resources to inject faults into. Fault parameter mapping rules address the "tool matching" problem, clarifying the correspondence and parameter configuration between different fault types and cloud-native fault injection tools. Fault injection scheduling rules address the "time and order" problem, determining the trigger time and priority order of fault injection based on user needs. Fault resource association graphs address the "resource location" problem, establishing a mapping between the scope of fault impact and actual physical resources. Due to the diverse fault types in cloud-native systems (e.g., network, service, database faults), dynamic resource changes (Pod creation / destruction), and potential priority differences in test tasks (core service fault tests should be executed first), without a unified model, fault injection can fall into a predicament of "chaotic tool invocation," "disordered task execution," and "incorrect resource location." For example, it may be impossible to determine whether to use ChaosMesh or another tool for a "network latency" fault, or multiple fault tasks may trigger simultaneously, leading to resource conflicts. Therefore, a dedicated step is needed to construct a fault test execution model that includes the above three rules / graphs, transforming scattered data into unified execution logic.

[0042] In traditional solutions, the selection of tools, task scheduling, and resource location for fault testing largely rely on manual experience-based decisions. For tool selection, operations personnel must manually search for suitable tools based on the fault type. For example, when injecting network faults, they need to manually determine whether to use ChaosMesh and consult the documentation for configuration parameters, which is inefficient and prone to failure due to incorrect parameters. Regarding task scheduling, there is no unified priority mechanism, and the execution order of multiple fault tasks needs to be manually coordinated. If core service and non-core service fault tests are triggered simultaneously, resource contention may cause the core test to fail. For resource location, the relationships between services and resources are manually analyzed. When system resources change dynamically (e.g., IP changes after a Pod restart), the relationship information cannot be updated in a timely manner, leading to fault injection into invalid resources. To address these shortcomings, this solution proposes a "data-driven model construction" approach. Based on basic test data, it automatically generates three major rules / graphs, ensures logical correctness through validity verification, and ultimately integrates them into a directly callable fault test execution model. This fundamentally replaces manual experience-based decisions, improving the automation and accuracy of fault testing.

[0043] Specifically: The first step, "Based on the association between system resource metadata and service deployment data, establish a mapping rule between fault types and fault parameters of cloud-native fault injection tools," is to solve the problem of "what tools to use and what parameters to use to inject faults," thereby achieving automated matching of fault types to tools and parameters. In terms of technical implementation, the process first extracts key resource configuration information from system resource metadata, such as container network configuration (CNI plugin type) and hardware resource quotas (CPU / memory limits). It then extracts service performance information from service deployment data, such as the average response time of microservices and the connection pool configuration of database services. Next, it compiles a pre-defined list of fault types (consistent with the fault types in user fault testing requirements, such as network latency, service downtime, database deadlock, and CPU overload). For each fault type, it matches a corresponding cloud-native fault injection tool, based on the tool's functional adaptability and resource compatibility—for example, "network latency" faults are adapted to ChaosMesh (because it supports container network fault injection), "service downtime" faults are adapted to the Kubernetes API (which allows direct Pod deletion), and "hardware CPU overload" faults are adapted to cloud vendor APIs (such as AWSFIS, which can simulate cloud server CPU load). Finally, based on the extracted resource configuration and service performance information, it defines the tool's calling parameters, forming a "fault type-tool-parameter" correspondence, i.e., fault parameter mapping rules. For example, for a "network latency" failure, if the container uses the CalicoCNI plugin in the system resource metadata and the average response time of service A in the service deployment data is 100ms, then the failure parameter mapping rule will define "failure type: network latency to tool: ChaosMesh to parameter: latency=200ms (higher than the average response time to ensure observable impact), selector=app:service-A (get the Pod tag of service A from the system resource metadata), duration=20m (refer to the user's possible failure duration requirements)"; for a "service downtime" failure, the rule will define "failure type: service downtime to tool: Kubernetes API to parameter: podName=service-A-xxx (get the Pod name of service A from the system resource metadata), gracePeriodSeconds=30 (get the graceful exit time configuration of service A from the service deployment data)".The significance of this step lies in replacing manual querying and configuration with data-driven tool and parameter matching, avoiding problems such as incorrect tool selection and improper parameter settings. For example, traditional manual methods might mistakenly inject a "CPU overload" fault using the Kubernetes API, while this step automatically matches the cloud vendor's API based on system resource metadata (cloud server instance information), ensuring tool compatibility. At the same time, parameter configuration is based on actual resource and service data. For example, the "latency" parameter is set with reference to the service's average response time, ensuring that the impact of the fault can be effectively observed, rather than being blindly set to a fixed value, thus improving the accuracy of fault testing.

[0044] The second step, "Based on the fault execution sequence and test priority in the user fault test requirement data, and combined with the fault parameter mapping rules, formulate fault injection scheduling rules," aims to solve the problems of "when to execute fault injection" and "in what order multiple fault tasks are executed," ensuring that fault tasks are triggered in an orderly manner according to user needs, avoiding resource conflicts and priority inversion. Technically, the fault execution sequence and test priority in the user fault test requirement data are first analyzed: the fault execution sequence includes "timed triggering" (e.g., "2001-01-01 01:00") and "instant triggering" (current time). Timed triggering requires setting the task trigger time point in conjunction with a timed task framework (e.g., Quartz), while instant triggering uses the current time as the trigger time. Test priorities are divided into three levels: "high," "medium," and "low," which need to be converted into numerical weights (e.g., high = 100, medium = 50, low = 30) for task sorting. Next, based on the resource requirements of tool calls in the fault parameter mapping rules (such as the need to occupy more node resources to inject CPU overload faults), the scheduling logic is formulated as follows: First, time-triggered logic, setting Quartz scheduled tasks based on the fault execution sequence to ensure that the tasks are triggered at the specified time; second, priority sorting logic, using Redis ordered sets to store fault tasks, sorting them according to priority weight, with tasks with higher weights being executed first; third, dependency execution logic, if multiple fault tasks have a relationship (such as "injecting database faults first, then injecting service faults that depend on the database", this relationship can be extracted from the "remarks" of user fault test requirement data or the dependency relationship of service deployment data), then a dependency is established through the task ID, and "the downstream task must be triggered after the upstream task has been executed and generated a success log". For example, if a user's fault testing requirement data contains two tasks: Task 1 is "Network latency test for core service A, high priority, triggered at 01:00", and Task 2 is "Service downtime test for non-core service B, low priority, triggered at 01:10", then the fault injection scheduling rule will set the weight of Task 1 to 100 and Task 2 to 30. The Redis sorted set will prioritize scheduling Task 1. After the fault injection tool for Task 1 is called (e.g., ChaosMesh successfully injects a delay) and the execution log is recorded, Task 2 will be scheduled. If Task 2 depends on Task 1 (the impact of the dependency on Service B needs to be observed after the fault injection of Task 1), then the rule will set Task 2 to execute 10 minutes after Task 1 is triggered.The significance of this step lies in solving the problems of disorder and inefficiency in traditional manual task coordination through unified scheduling rules. In cloud-native systems, multiple fault testing tasks may exist simultaneously, and it is difficult for humans to coordinate priorities and triggering order in real time. This step, however, ensures that core tasks are executed first and related tasks are connected in an orderly manner through time-triggered, priority sorting, and dependency execution logic. For example, the test of core service A occupies resources first, avoiding non-core tasks from preempting core tests and causing them to fail. At the same time, rules are formulated based on user demand data to ensure that the scheduling logic is consistent with the user's testing goals. For example, if the user wants to test the impact of database failures on services first, the scheduling rules can accurately realize this dependency relationship and improve the effectiveness of test results.

[0045] The third step, "Constructing a fault resource association map of the fault impact range and physical resources based on the system resource metadata and the service deployment data," is to solve the problem of "which resources to inject faults into" and establish a visual association mapping between the fault impact range (such as "service A" specified by the user) and actual physical resources (such as the Pod instance corresponding to service A), while also reflecting the dependency relationship between resources and services. In terms of technical implementation, firstly, physical resources in the system resource metadata are used as nodes, including cluster nodes and Pod instances. Node attributes need to be labeled with resource identifiers (such as node IP and Pod name), the service to which they belong (extracted from the "container service information" in the system resource metadata, such as Pod "service-A-xxx" belonging to the service "service-A"), resource status (such as node "Ready" and Pod "Running"), and resource configuration (such as Pod CPU quota). Next, service relationships in the service deployment data are used as edges, including the affiliation relationship between services and resources (such as "service-A" pointing to its corresponding Pod instance) and the dependency relationship between services (such as "service-A" calling "service-B"). Edge attributes need to be labeled with relationship type (such as "affiliation" or "HTTP call"), communication protocol (such as "TCP" or "HTTP"), and port information (such as "8080"). Then, a graph is used. Databases (such as Neo4j) store the relationships between nodes and edges. Relationships are created using Cypher statements. For example, `CREATE(s:Service{name:'service-A'})-[:BELONGS_TO]->(p:Pod{name:'service-A-xxx',ip:'10.0.1.1',status:'Running'})` creates the attribution relationship between service A and Pod, and `CREATE(sA:Service{name:'service-A'})-[:CALLS{protocol:'HTTP',port:8080}]->(sB:Service{name:'service-B'})` creates the calling relationship between service A and service B. The resulting fault resource association graph visually displays the attribution chain from "service to Pod to node" and the dependency chain from "service to service". For example, the graph can quickly show that "Service A" contains 3 Pod instances (IPs are 10.0.1.1, 10.0.1.2, and 10.0.1.3 respectively), all deployed on node 10.0.0.1, and Service A depends on port 8080 of Service B.The significance of this step lies in addressing the incompleteness and lag of traditional manual resource relationship analysis. In cloud-native systems, Pod instances may dynamically change due to scaling, failures, restarts, and other reasons, making it difficult for manual updates to service-resource relationships in real time. This step automatically constructs a graph based on system resource metadata and service deployment data. When a Pod instance is updated (e.g., added or deleted), the system resource metadata is updated synchronously, and the graph is refreshed accordingly, ensuring the real-time nature of resource relationships. Simultaneously, the graph can visually represent the scope of a fault's impact. For example, if a user specifies "the scope of the fault is service A," the graph can quickly locate all Pod instances corresponding to service A, preventing the fault from being injected into resources of other services or missing some Pods of service A, ensuring the accuracy of the fault injection scope. Furthermore, the display of service dependency chains can support subsequent monitoring and alerting. For instance, after a fault is injected into service A, the graph can predict that its downstream service B may be affected, allowing for early and focused monitoring of service B's metrics.

[0046] The fourth step, "validating the fault parameter mapping rules, the fault injection scheduling rules, and the fault resource association graph," involves multi-dimensional testing to ensure the logical correctness, parameter rationality, and association integrity of the three rules / graphs. This avoids subsequent fault injection failures due to model defects and provides a reliable prerequisite for model integration. Technically, the verification content and methods correspond to the three rules / graphs: For the fault parameter mapping rules, "tool call testing" is used for verification—for each fault type and tool combination in the rule, the tool's test interface is called with configuration parameters to check if the tool can execute fault simulation normally. For example, for the "network latency to ChaosMesh" rule, the ChaosMesh test API is called with parameters such as `latency=200ms` to check if the target Pod can successfully simulate a 200ms latency. If the tool returns "execution successful" and the latency is actually monitored, the rule is valid. For the fault injection scheduling rules, "task simulation scheduling" is used for verification—a simulated task set (containing tasks with different priorities, trigger times, and dependencies) is constructed, and scheduling is performed based on the rules to check if tasks are sorted by priority, triggered at the specified time, and so on. To verify whether tasks are executed in an orderly manner, for example, simulating "high-priority task 1 (triggered at 14:00)" and "low-priority task 2 (triggered at 14:10)", the scheduling rule is verified to ensure that task 1 is executed first and then task 2. If it meets expectations, the rule is effective. For the fault resource association graph, "resource matching test" and "dependency chain integrity test" are used for verification. The resource matching test checks whether the scope of the fault impact in the user's requirements (such as "service A") can be matched with all corresponding physical resources (such as all Pods of service A) in the graph. If the number of matches is consistent with the number of Pods of service A in the system resource metadata, the resource association is effective. The dependency chain integrity test checks whether the service dependency relationship in the graph is completely consistent with the service deployment data. For example, if service A depends on service B in the service deployment data, and there is a corresponding "CALLS" edge in the graph, the dependency chain is complete. During the verification process, if invalid items are found (such as incorrect tool call parameters, reversed task scheduling order, or missing resource matching), an error report will be generated, indicating the direction of correction. For example, "ChaosMesh's latency parameter format is incorrect and needs to be changed to '200ms' format," "Task 2 is not waiting for Task 1 to execute according to its dependency relationship and the scheduling logic needs to be adjusted," and "The Pod 'service-A-003' of Service A is not associated in the graph and the graph construction logic needs to be updated."The significance of this step lies in exposing defects in the model in advance, avoiding failure injection after the model is directly put into use. In traditional methods, the model lacks a verification stage. If parameters are incorrect (such as incorrect parameter format in ChaosMesh), they can only be discovered during failure injection, wasting testing time. This step, however, proactively verifies and fixes problems before model integration. For example, incorrect tool call parameters can be corrected during the verification stage without waiting for actual failure injection, significantly improving the efficiency and success rate of subsequent processes. At the same time, the verification results can also feed back into previous steps. If graph resource matching is missing, it can be traced back to the system resource metadata or service deployment data collection stage to check for data omissions, forming a closed loop from data collection to model building to verification to data correction, ensuring the consistency between the test base data and the model.

[0047] The fifth step, "integrating the verified fault parameter mapping rules, fault injection scheduling rules, and fault resource association graph into a fault test execution model," encapsulates the three verified rules / graphs into a unified, directly callable model entity. This provides a unified model entry point for subsequent fault injection, monitoring and alarming, and automated recovery, achieving "model built once, reused across multiple modules." Technically, a unified data structure for the model is first defined, stored in JSON format, containing three core fields: "faultParamMap" (fault parameter mapping rules), "faultInjectSchedule" (fault injection scheduling rules), and "faultResourceGraph" (fault resource association graph). Each field corresponds to verified rule / graph data. For example, the "faultParamMap" field stores key-value pairs such as "fault type: network latency to tool: ChaosMesh to parameter: {latency:200ms,selector:app:service-A}", and the "faultResourceGraph" field stores JSON-formatted data of the graph database's nodes and edges. Then... The model interface layer provides three core interfaces: rule query interfaces (e.g., `getParamMap(faultType)` for querying corresponding tools and parameters based on fault type), scheduling execution interfaces (e.g., `scheduleTask(taskId)` for triggering specified fault tasks based on scheduling rules), and resource association query interfaces (e.g., `getResourceByService(serviceName)` for querying associated physical resources based on service name). Finally, the model data and interface layer are encapsulated into independent service modules (e.g., "fault-test-model-service" in a microservice architecture), deployed to a cloud-native cluster, and made available for subsequent fault injection and monitoring / analysis modules to call via RESTful API or gRPC protocol. For example, when the fault injection module needs to handle the "network latency of service A" fault, it can call `getParamMap("network latency")` to obtain the ChaosMesh and parameters, call `getResourceByService("service-A")` to obtain the target Pod list, and then call `scheduleTask(taskId)` to trigger scheduling execution. The entire process does not require repeated parsing of rules or graphs, directly reusing the model interface.The significance of this step lies in achieving synergy and modular reuse of the three major rules / graphs through model integration. In traditional methods, tool matching, task scheduling, and resource location are fragmented logics, requiring each module to repeatedly process data. This step integrates these three into a unified model, with each module calling through standardized interfaces, reducing data redundancy and repetitive processing. For example, when the monitoring and analysis module needs to locate the scope of a fault's impact, it can directly call the resource association query interface without rebuilding the service-resource relationship, improving the collaboration efficiency of each module. At the same time, the independence of the model facilitates maintenance and updates. When cloud-native system resources or user needs change, only the rules or graph in the model need to be updated, and each calling module does not need to modify its code. For example, when adding a new fault type, only the corresponding entry needs to be added to the fault parameter mapping rule, and the fault injection module can automatically obtain the new rule through the interface, enhancing the system's scalability and adaptability.

[0048] In summary, by constructing a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules, and a fault resource association graph, scattered test data is transformed into a structured and executable logical system. This solves the problems of chaotic tool matching, disordered task scheduling, and incorrect resource location in traditional fault testing. Specifically, the fault parameter mapping rules ensure "using the right tools and setting the right parameters," the fault injection scheduling rules ensure "execution in sequence and triggering on time," and the fault resource association graph ensures "finding the right resources and clarifying dependencies." These three elements are validated and integrated to form a unified model, providing a precise, orderly, and controllable execution basis for subsequent automated fault injection, monitoring analysis, and recovery operations. Ultimately, this improves the automation level and reliability of test results in cloud-native system fault testing.

[0049] In one embodiment of the present invention, the step of performing automated fault injection operation according to the fault test execution model includes: S31. Based on the fault parameter mapping rules and the target fault type in the user's fault testing requirements, match the corresponding cloud-native fault injection tool and calling parameters to generate specific fault injection instructions. S32, determine candidate target resources based on the fault resource association graph, query their current running status and remove invalid resources to determine the target resource set for fault injection; S33, according to the fault injection scheduling rules, the task triggering conditions are monitored through the timed task framework, the API interface of the fault injection tool is called in order of priority, and the fault injection operation is performed on the target resource set according to the fault injection instruction; S34. Obtain the execution result of the fault injection operation. If the execution is successful, record the fault identifier, target resource, start time and execution status to the fault injection operation log. If the execution fails, record the reason for failure and trigger the retry mechanism. If the retry fails after a preset number of times, generate an alarm message. S35. After the fault injection operation is executed, the status changes of the target resource are monitored in real time to verify whether the fault has taken effect. If it has not taken effect, the cause is analyzed and the parameters are adjusted and the operation is re-executed until the fault takes effect or the operation limit is reached.

[0050] As described in steps S31-S35 above, the design of the steps from "generating fault injection instructions to determining the target resource set to executing fault injection according to scheduling rules to recording execution results and logs to verifying the fault effectiveness status" transforms user fault testing requirements into actual fault injection operations based on the constructed fault test execution model. This enables automated fault injection into cloud-native systems and ensures the traceability and effectiveness of operations through log recording and effectiveness verification, providing an execution basis for subsequent system monitoring analysis and automated recovery.

[0051] Fault injection is the core execution step in the entire testing process, and its quality directly determines the test results. Only accurate and effective fault injection can trigger real abnormal responses from the system, thereby verifying the system's resilience and recovery capabilities. Due to the dynamic nature (frequent resource creation and destruction) and complexity (multiple service dependencies) of cloud-native systems, fault injection faces three major challenges: First, accurate matching of tools and parameters. Different fault types require different tool and parameter configurations; incorrect matching will prevent the simulation of expected faults. Second, the real-time validity of target resources. Cloud-native resources (such as Pods) may dynamically go offline; injecting into invalid resources will lead to test failure. Third, the orderliness and verifiability of the execution process. When multiple tasks run concurrently, they must be executed according to priority, and it is necessary to confirm whether the fault has actually taken effect. Therefore, it is necessary to automate, accurately, and verifiably perform fault injection through specific steps, which is the core problem that this step aims to solve.

[0052] Traditional fault injection solutions often rely on manual operations: operations personnel select tools based on experience, manually input parameters to call APIs (e.g., using `kubectlexec` to enter a container and execute commands to simulate service downtime), or manually configure network latency parameters using the ChaosMesh web interface; target resource selection depends on manually querying the currently running Pod list, which is prone to selecting invalid resources due to dynamic resource changes; there is no automated scheduling for multi-task execution, requiring manual coordination of the order; and checking the effectiveness of faults requires manual login to the monitoring platform to view metrics, which is inefficient and prone to missed verifications. These methods are not only time-consuming and labor-intensive, but also suffer from problems such as incorrect parameter configuration, invalid resource selection, disordered execution, and untimely verification, seriously affecting testing efficiency and accuracy. To address these shortcomings, this solution proposes a "model-driven, automated execution, and full-process verification" approach. Based on a fault test execution model, it automatically matches tools and parameters, filters valid resources, schedules execution according to rules, and ensures operational quality through log recording and effectiveness verification, replacing manual operations and improving the automation level and reliability of fault injection.

[0053] Specifically: The first step, "Based on the fault parameter mapping rules and the target fault type in the user's fault testing requirements, match the corresponding cloud-native fault injection tool and calling parameters to generate specific fault injection instructions," is essentially to transform the user's fault testing requirements into specific instructions that can be executed by the tool, ensuring accurate matching between the fault type and the tool and parameters. In terms of technical implementation, firstly, the target fault type (such as "network latency" or "CPU overload") is extracted from the user's fault test requirement data. Then, the query interface for the fault parameter mapping rules in the fault test execution model (such as `getParamMap(faultType)`) is called to obtain the corresponding cloud-native fault injection tools (such as ChaosMesh, Kubernetes API, and AWSFIS) and basic parameters (such as `latency=200ms` for network latency and `cpuLoad=80%` for CPU overload). Next, combined with personalized parameters from the user's requirements (such as the fault duration `duration=30m`, which is input by the user in the visual configuration interface and is part of the user's fault test requirement data), the basic parameters are supplemented and adjusted to form a complete parameter set. Finally, according to the tool's API specification, the tool identifier and parameter set are converted into an executable instruction format. For example, the ChaosMesh network latency fault instruction is an experimental definition in JSON format: `{"apiVersion":"chaos-mesh.org / v1alpha1", The Kubernetes API Pod deletion command is `kubectldelete pod service-a-xxx--grace-period=30`. The corresponding HTTP request is `DELETE / api / v1 / namespaces / default / pods / service-a-xxx`, with the request header containing `GracefulDeleteSeconds:30`.For example, if a user's requirement is to "inject a 200ms network latency into service A for 30 minutes," the tool "ChaosMesh" and the basic parameter "selector=app:service-A" are obtained by querying the fault parameter mapping rules. Combined with the user-specified "latency=200ms" and "duration=30m," the above-mentioned JSON-formatted fault injection command is generated. The significance of this step is that model-driven command generation avoids errors caused by manual tool selection and configuration parameters. Traditionally, manual intervention might lead to errors such as writing "200s" as the network latency unit (far exceeding expectations) due to unfamiliarity with tool parameters. This step, based on parameter constraints in the fault parameter mapping rules (such as the latency unit must be "ms"), automatically verifies and corrects the parameter format, ensuring the validity of the command. Furthermore, the command generation process is fully automated, eliminating the need for manual writing of complex tool configurations, such as eliminating the need for operations personnel to manually write the ChaosMesh YAML configuration, significantly improving operational efficiency.

[0054] The second step, "determining candidate target resources based on the fault resource association graph, querying their current operating status and eliminating invalid resources, and determining the target resource set for fault injection," aims to ensure that the resource objects injected for fault injection are real and valid, avoid injecting faults into offline or abnormal resources, and ensure the accuracy of the test. In terms of technical implementation, firstly, based on the scope of the fault impact in the user's fault testing requirements (e.g., "all instances of service A," which is specified by the user in the visual configuration interface and belongs to the user's fault testing requirement data), the query interface of the fault resource association graph (e.g., `getResourceByService(serviceName)`) is called to obtain the list of associated candidate target resources (e.g., all Pod instance names and IPs corresponding to service A); then, the current running status of each candidate resource is queried through the cloud-native platform interface (e.g., `kubectlgetpods-ojson` of Kubernetes API). The status judgment criteria include: the Pod instance must be in the "Running" state (not "Terminating", "Error", or "Pending"), and the node must be in the "Ready" state (not "NotReady"); finally, invalid resources that do not meet the status requirements are removed, and the remaining valid resources are combined into a target resource set, and the unique identifier of each resource (e.g., the UID of the Pod) is recorded. For example, if the user specifies the scope of the fault impact as "service A", the candidate Pods "service-A-001", "service-A-002", and "service-A-003" are found through the fault resource association graph. If the current status is found that "service-A-003" is in the "Terminating" state (being destroyed), then the Pod is removed, and the target resource set is finally "service-A-001" and "service-A-002". The significance of this step lies in resolving the issue of invalid targets caused by dynamic changes in cloud-native resources. In cloud-native systems, Pods may be restarted due to failed health checks or destroyed due to scaling up or down. If a static resource list is used directly, it is easy to inject faults into invalid resources (such as injecting network latency into destroyed Pods), leading to test failure. This step, however, ensures that all resources in the target resource set are in a normal operating state by querying and filtering resource status in real time. For example, it only injects faults into Pods in the "Running" state, ensuring that the fault can be effectively triggered. At the same time, resource filtering is based on the fault resource association graph to ensure the consistency between resources and the scope of fault impact. For example, it only selects Pods corresponding to service A to avoid mistakenly injecting resources for service B.

[0055] The third step, "According to the fault injection scheduling rules, the timed task framework listens for task triggering conditions, calls the API interface of the fault injection tool in priority order, and executes fault injection operations on the target resource set according to the fault injection instructions," is primarily aimed at ensuring the orderly execution of fault injection tasks. This ensures tasks are triggered according to the user-specified timing and priority, avoiding resource conflicts and execution chaos. Technically, the fault injection task (including the target fault type, target resource set, and fault injection instructions) is first registered with a timed task framework (such as Quartz). The framework sets the trigger time based on the time triggering logic in the fault injection scheduling rules (extracted from the fault execution timing of the user's fault test requirement data, such as "2001-01-01 01:00"). If it's an immediate trigger, it immediately enters the task queue. Next, the timed task framework sorts the tasks in the task queue based on the priority sorting logic in the scheduling rules (high-priority tasks have higher weight than low-priority tasks), prioritizing tasks with higher weights. When the task triggering conditions are met (reaching the specified time or immediate triggering), the framework... The system sequentially calls the API interfaces of the fault injection tool, sending fault injection instructions to the target resource set. For example, it calls the ` / apis / chaos-mesh.org / v1alpha1 / networkchaos` interface of ChaosMesh to create a network latency experiment, or calls the Pod deletion interface of the Kubernetes API to perform service failure fault injection. During execution, a distributed lock (such as RedisLock) is used to prevent multiple tasks from injecting faults into the same resource simultaneously. For instance, when both Task 1 and Task 2 need to inject faults into "service-A-001", the locking mechanism ensures that Task 2 is executed only after Task 1 has finished. For example, if there are two tasks in the task queue: Task 1 (high priority, triggered at 01:00, injecting network latency into service A) and Task 2 (low priority, triggered at 01:10, injecting CPU overload into service B), the scheduled task framework will prioritize calling the ChaosMesh API to execute Task 1 at 01:00. After the API call of Task 1 returns successfully, it releases the resource lock of service A and then calls the cloud vendor's API to execute Task 2. The significance of this step lies in solving the problem of disorder in the concurrent execution of multiple tasks through automated scheduling. When manually executing multiple faulty tasks, it is difficult to accurately control the triggering time and order, which may lead to core tasks (such as payment service fault testing) being preempted by non-core tasks, affecting the test results. This step, through a scheduled task framework and priority sorting, ensures that core tasks are executed first and are triggered strictly at the specified time. For example, the high-priority payment service fault test is executed precisely at 01:00, without being disturbed by other tasks. At the same time, the distributed lock mechanism avoids resource conflicts and ensures that the same Pod is not injected with multiple faults at the same time (such as simultaneously injecting network latency and CPU overload), ensuring the singularity and analyzability of the fault impact.

[0056] The fourth step, "obtaining the execution result of the fault injection operation, records the fault identifier, target resource, start time, and execution status to the fault injection operation log if successful; if it fails, records the reason for failure and triggers a retry mechanism. If the retry fails after a preset number of attempts, an alarm message is generated," aims to achieve traceability of the fault injection operation and improve the success rate through the retry mechanism, while also providing timely feedback on failures. Technically, it first listens to the return result of the fault injection tool's API interface. A 2xx status code (e.g., 200 OK, 201 Created) indicates successful execution, while non-2xx status codes (e.g., 404 Not Found, 500 Internal Server Error) indicate failure. For successful operations, the unique fault identifier (a unique ID of the user's fault test requirement data obtained from the system configuration center), target resource set (e.g., "service-A-001", "service-A-002"), operation start time (API call time), and execution status ("successful") are written to the fault injection operation log (using JS). The logs are in ON format and stored in a distributed log system such as ELK. For operations that fail, the system parses the error messages returned by the API (such as "Podservice-A-001 not found" or "ChaosMesh experiment creation failed"), records the reason for failure in the log, and triggers a retry mechanism. The retry interval is set according to an exponential backoff strategy (e.g., 10s for the first time, 20s for the second, and 40s for the third), with a default retry count of 3. If the 3 retries still fail, an alarm message is generated through a message notification service (such as DingTalk robot or Email), which includes the fault identifier, the reason for failure, the target resource, etc., to notify the operations and maintenance personnel to intervene. For example, when injecting network latency into "service-A-001", the ChaosMesh API returns a 404 error (because the Pod has been destroyed). The system will record the reason for failure "Podservice-A-001 not found" and retry after 10s. If it fails 3 times in a row, an alarm message "Fault task ID: req-001 execution failed, reason: Pod does not exist, please check resource status" is sent. The significance of this step lies in enabling full-process traceability of the fault injection operation through logging, facilitating the query of operation details when generating subsequent test reports. For example, if the test report needs to statistically analyze metrics such as "number of successfully injected faults" and "failure rate," these can be directly extracted from the logs. At the same time, the retry mechanism can resolve execution failures caused by temporary network fluctuations and occasional failures of tool interfaces, improving the success rate of operations. For example, if an API call times out due to network latency, it may succeed after a retry. Furthermore, alarm information can promptly report problems that cannot be resolved through retries (such as the target resource being permanently offline), preventing fault injection tasks from being blocked for extended periods.

[0057] The fifth step, "After the fault injection operation is executed, monitor the status changes of the target resource in real time to verify whether the fault has been successfully implemented. If it has not been implemented, analyze the reasons and adjust the parameters to re-execute until the fault is implemented or the operation limit is reached," ensures that the fault is truly injected through actual status verification, avoiding invalid tests due to the tool executing successfully but not actually producing any effect. Technically, firstly, corresponding effectiveness verification indicators are defined based on the fault type. For example, the verification indicator for network latency faults is "network outbound latency of the target Pod," and the verification indicator for CPU overload faults is "CPU utilization of the target Pod." Next, the verification indicator data of the target resource is collected in real time through a monitoring system (such as Prometheus), with a collection frequency set to 1 time / second, continuously collected for 30 seconds (to ensure data stability). Then, the collected indicator data is compared with the expected fault effect. For example, the expected effect of network latency faults is "latency ≥ 200ms," and the expected effect of CPU overload faults is "to make..." "Usage rate ≥ 80%" means that if the metrics at 5 consecutive data collection points meet the expectations, the fault is considered effective. If the expected results are not met (e.g., network latency is only 50ms), the reasons for failure to take effect are analyzed. These may include incorrect tool parameter configuration (e.g., the latency unit is written as "us"), incompatibility between the target resource and the tool (e.g., ChaosMesh does not support a certain type of CNI plugin). Based on the cause, parameters are adjusted (e.g., the unit is corrected to "ms") or the tool is changed (e.g., another network fault injection tool is used), and the fault injection operation is re-executed. The preset operation limit is usually 3 times. If it still does not take effect after 3 attempts, "fault not effective" is recorded in the log, and the operation is terminated. For example, after injecting 200ms network latency into a Pod of service A, monitoring shows that the actual latency is only 50ms. Log analysis reveals that the latency unit in the fault injection command was mistakenly written as "us". After adjusting the parameter to "200ms" and re-executing, the latency reaches 200ms again, and the fault is considered effective. The significance of this step lies in resolving the issue that "successful tool execution ≠ actual fault effectiveness." Traditional methods assume fault injection as long as the tool's API returns a success, ignoring "false successes" caused by tool configuration errors, resource incompatibility, etc. For example, the tool might successfully execute a command, but due to incorrect parameter units, it might not produce the expected delay, preventing subsequent monitoring and analysis from capturing the anomaly. This step, however, verifies the fault's effectiveness through actual metrics. For instance, fault injection is only confirmed as successful when the monitored network latency of the Pod reaches 200ms, providing a realistic fault scenario for subsequent system anomaly response analysis. Simultaneously, the parameter adjustment mechanism automatically corrects some configuration errors, reducing manual intervention. For example, incorrect units can be resolved by readjusting parameters, improving the effectiveness of fault injection.

[0058] In summary, by employing model-driven fault injection command generation, dynamic and effective target resource selection, rule-based orderly task scheduling and execution, end-to-end logging and retry mechanisms, and fault effectiveness verification based on monitoring metrics, the automation, precision, and verifiability of fault injection in cloud-native systems have been achieved. This process resolves issues such as incorrect tool parameter matching, invalid target resources, disordered task execution, and faults not actually taking effect in traditional manual operations. It ensures that fault injection operations can accurately simulate fault scenarios required by users, providing a realistic and effective fault environment for subsequent system monitoring, analysis, and automated recovery, ultimately improving the reliability and efficiency of the entire fault testing process.

[0059] In one embodiment of the present invention, the steps of collecting and analyzing system monitoring data according to the fault test execution model, generating an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously recording fault injection operation logs include: S41. Based on the hierarchical structure of the cloud-native system under test, deploy corresponding monitoring agents in the infrastructure layer, container runtime layer, and application service layer, and configure the metric collection range, frequency, and transmission method for each layer. S42, obtain historical system indicator data before fault injection, perform data cleaning and aggregation, use statistical analysis methods to calculate the baseline threshold range of each indicator, and generate system indicator baseline data; S43: Collect real-time monitoring metrics of the cloud-native system to be tested after the fault injection operation begins, and transmit them to the data processing center for aggregation and verification via message queue; S44, compare and analyze the real-time monitoring indicators with the baseline data of the system indicators, identify abnormal indicators that exceed the baseline threshold range, locate the scope of the abnormal impact by combining the fault resource association map, and generate an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier. S45 performs in-depth analysis of abnormal alarm logs and associated real-time monitoring metrics, calculates the deviation of abnormal metrics, marks the scope of impact on downstream services, and records the abnormal metrics, deviation, and scope of impact in the fault injection operation log.

[0060] As described in steps S41-S45 above, the system is designed with the steps of "deploying monitoring agents, generating system indicator baseline data, collecting and transmitting real-time monitoring indicators, comparing and analyzing to generate abnormal alarm logs, and performing in-depth analysis and supplementing logs". Based on automated fault injection operations, it realizes real-time collection of multi-dimensional indicators, anomaly identification and alarm generation after cloud-native system failures. At the same time, it marks the scope of the fault impact through in-depth analysis and supplements it to the fault injection operation log, providing accurate anomaly location basis for automated fault recovery and accumulating complete monitoring data support for the subsequent generation of test reports.

[0061] Post-fault injection system monitoring and analysis is the key link between "fault injection" and "automated recovery"—only by accurately capturing the impact of faults on system metrics, identifying anomalies, and locating the scope of impact can targeted recovery operations be triggered. Simultaneously, monitoring data is also the core basis for assessing the degree of fault impact and verifying recovery effectiveness. Due to the multi-layered (infrastructure, containers, applications) and distributed nature of cloud-native systems, the impact of faults may propagate from underlying resources to upper-layer applications (e.g., CPU overload causing service response delays). Furthermore, the monitoring requirements for different layers differ significantly (the infrastructure layer needs to focus on CPU / memory, while the application layer needs to focus on request latency / error rate). Incomplete monitoring coverage, lack of baseline comparison, or untimely anomaly identification can lead to inaccurate assessment of fault impact, delayed recovery, and even the omission of key anomaly indicators. Therefore, specialized steps are needed to achieve multi-layered, end-to-end monitoring data collection and analysis.

[0062] Traditional solutions for system monitoring suffer from three major limitations: First, incomplete monitoring coverage, often focusing on the infrastructure layer or a single application layer, lacking unified monitoring across multiple levels of cloud-native systems. For example, monitoring only node CPU utilization without monitoring application request error rates can fail to fully capture the impact of faults. Second, the lack of dynamic baselines or rigid baseline settings often employ fixed thresholds (such as alarms for CPU utilization exceeding 80%) without adjusting based on historical system operating conditions, making them prone to false or missed alarms due to business fluctuations (such as sudden traffic surges during peak periods). Third, the disconnect between anomaly identification and fault correlation means that monitoring tools can only identify isolated metric anomalies, unable to correlate them with specific fault-injected tasks (such as being unable to determine which fault task caused the "increased response latency") or locate the downstream services affected by the anomalies, resulting in a lack of precise targeting for subsequent recovery operations. To address these shortcomings, this invention proposes a solution of "multi-level agent deployment + dynamic baseline generation + fault correlation analysis". By collecting data through monitoring agents covering all levels, a dynamic baseline is generated based on historical data. Combined with fault resource correlation graphs, the scope of impact is located and fault identifiers are associated. Finally, an abnormal alarm log with complete context is generated, which completely solves the fragmentation and isolation problems of traditional monitoring.

[0063] Specifically: The first step, “Based on the hierarchical structure of the cloud-native system to be tested, deploy corresponding monitoring agents at the infrastructure layer, container runtime layer, and application service layer, and configure the metric collection scope, frequency, and transmission method for each layer,” is to build a monitoring data collection system covering all layers of the cloud-native system to ensure that the impact of faults on different layers is captured without any blind spots.In terms of technical implementation, the monitoring objectives and agent selection for each layer are first clearly defined: The infrastructure layer focuses on the status of node hardware resources, deploying NodeExporter as the monitoring agent. This is done through Ansible, batch-installed on all cluster nodes. The data collection scope includes CPU utilization (`node_cpu_seconds_total`), memory utilization (`node_memory_usage_bytes`), disk I / O rate (`node_disk_read_bytes_total`), and network bandwidth (`node_network_transmit_bytes_total`). The collection frequency is set to 1 second (to ensure real-time performance), and the transmission method uses PrometheusPull mode (the agent exposes port 9100 for Prometheus to pull data). The container runtime layer focuses on container resource usage and health status, utilizing Kubernetes' InitContainer mechanism to inject the cAdvisor agent into each Pod. The data collection scope includes container CPU utilization (`container_cpu_usage_seconds_total`), memory utilization (`container_memory_usage_bytes`), and container health status (`container_cpu_usage_seconds_total`). The `container_health_status` metric is generated based on the results of `livenessProbe` and `readinessProbe`. The collection frequency is set to 1 second, and the transmission method is to output to the node-level PrometheusAgent via DockerSocket. The application service layer focuses on business metrics and call chains. The microservice code is instrumented through OpenTelemetrySDK (no intrusive modification is required, the SDK automatically instrumentes the data). The collection scope is configured to include request latency (`traces_http_server_duration_ms`), request error rate (`traces_http_server_errors_count`), database connection pool utilization (`application_db_connection_pool_usage`), and service call volume (`traces_http_server_requests_count`). The collection frequency is set to 0.5 seconds (business metrics change faster, so a higher frequency is required). The transmission method is to send the data to OpenTelemetryCollector via OTLP protocol, and then the Collector forwards it to Prometheus and Jaeger (distributed tracing system).For example, in a certain e-commerce cloud-native system, the infrastructure layer collects the CPU utilization of node "node-01" through NodeExporter, the container runtime layer collects the memory utilization of service APod through cAdvisor, and the application service layer collects the order interface request latency of service A through OpenTelemetrySDK. The significance of this step lies in solving the problem of "incomplete layer coverage" in traditional monitoring. Cloud-native faults have a cascading effect; for example, container CPU overload (container layer) can lead to increased application request latency (application layer). If only the application layer is monitored, it is impossible to pinpoint the root cause of the fault as insufficient container resources. This step, through full-layer proxy deployment, can completely capture the impact chain of the fault from the underlying resources to the upper-layer applications. For example, after a CPU overload fault is injected into service A, it can simultaneously monitor "container CPU utilization exceeding 90%" (container layer) and "request latency increasing from 100ms to 500ms" (application layer), providing complete data for subsequent root cause analysis. At the same time, differentiated collection frequencies and transmission methods ensure the real-time and reliability of the collection of indicators at each layer. For example, high-frequency collection at the application layer meets the monitoring needs of rapidly changing business indicators, while the pull mode at the infrastructure layer reduces proxy resource consumption.

[0064] The second step, "Acquiring historical system indicator data before fault injection, performing data cleaning and aggregation, and using statistical analysis methods to calculate the baseline threshold range for each indicator to generate system indicator baseline data," aims to provide dynamic and reasonable judgment criteria for anomaly identification, avoiding false alarms or missed alarms caused by fixed thresholds. Technically, it first obtains historical system indicator data for the 7 days prior to fault injection through Prometheus's ` / api / v1 / query_range` interface, with a time window set to 10 minutes (balancing data granularity and computational load). For example, it obtains request latency data for Service A's order interface from 01:00 to 01:10 each day for the past 7 days. Next, it performs data cleaning, removing outliers (judgment criteria: a data point deviates from the average of 10 adjacent data points by more than 100%, such as a single 1000ms delay caused by network fluctuations), retaining normal business fluctuation data. Then, it uses the 3σ statistical analysis method to calculate the baseline threshold range: first calculating the cleaned data... The mean (μ) and standard deviation (σ) of the data points are calculated. The mean μ is the arithmetic mean of all data points, and the standard deviation σ is the square root of the sum of the squares of the differences between each data point and the mean. A baseline threshold range is then determined from "μ-3σ" to "μ+3σ". If the calculated threshold range exceeds the acceptable upper limit for the business (e.g., the business requires a maximum request latency of no more than 300ms, but "μ+3σ" is 400ms), the upper limit of the threshold is adjusted to 300ms based on the business requirements. Finally, the baseline thresholds of each indicator (e.g., the baseline request latency for service A is [80ms, 220ms]) are stored as system indicator baseline data and associated with the corresponding service and resource identifiers (e.g., the Pod tag for service A). For example, the average request latency of the order interface for service A over the past 7 days is μ=150ms, the standard deviation σ=23ms, and 3σ=69ms, with a baseline threshold range of [81ms, 219ms]. The business requires a maximum latency of 300ms, so no adjustment is needed; this range is the system indicator baseline data for the request latency of service A. The significance of this step lies in solving the problem of "poor adaptability" of traditional fixed thresholds. Cloud-native system business traffic fluctuates periodically (e.g., e-commerce system traffic is 3 times higher during promotional periods). Fixed thresholds (e.g., alarms for request latency exceeding 200ms) will generate a large number of false alarms during promotional periods due to normal traffic fluctuations. This step, based on a 3σ dynamic baseline of historical data, can automatically adapt to business fluctuations. For example, if the average request latency of service A rises to 200ms during promotional periods, the baseline threshold range will be adjusted accordingly to [143ms, 257ms], avoiding false alarms. At the same time, business requirement corrections ensure that the baseline conforms to the actual application scenario. For example, even if the statistical baseline allows a latency of 400ms, but the business only accepts 300ms, the correction can accurately capture anomalies exceeding the business threshold, improving alarm accuracy.

[0065] The third step, "collecting real-time monitoring metrics of the cloud-native system to be tested after the fault injection operation begins, and transmitting them to the data processing center for aggregation and verification via a message queue," ensures the real-time nature, completeness, and accuracy of the monitoring data after a fault, providing high-quality data input for subsequent anomaly analysis. Technically, after the fault injection operation begins (starting from the execution time of the fault injection command in the aforementioned steps), monitoring agents at each level collect real-time monitoring metrics at preset frequencies. For example, NodeExporter collects node CPU utilization once per second, and OpenTelemetrySDK collects request latency every 0.5 seconds. The collected metric data is first sent to a Kafka message queue (Kafka was chosen due to its high throughput, supporting the transmission of hundreds of thousands of metrics per second to avoid data loss), and then divided into different topics (e.g., "metrics.infrastructure" and "metrics.application") according to metric type (e.g., infrastructure metrics and application metrics). The data processing... The data processing center subscribes to the corresponding Kafka topic and aggregates metric data in 1-second time windows (calculating the average and maximum values ​​within each time window to avoid data redundancy). For example, it aggregates 10 request latency data points (180ms, 190ms…200ms) for service A within 1 second into an average value of 192ms. Simultaneously, it performs data validation, checking whether the metric values ​​are within reasonable ranges (e.g., CPU utilization not exceeding 100%, request latency not negative). Invalid data (e.g., CPU utilization 120%) is marked and discarded to ensure the aggregated data accurately reflects the system status. Validated aggregated data is stored in the Prometheus time-series database and associated with a unique fault identifier (the fault ID obtained from the fault injection operation log). For example, after injecting 200ms network latency into service A, OpenTelemetrySDK collects 10 request latency data points (280ms, 290ms…310ms), transmits them to the data processing center via Kafka, aggregates them into an average value of 295ms, and after verification without anomalies, stores them in Prometheus and associates them with the fault ID "fault-001". The significance of this step lies in solving the problem of "unreliable data transmission" in traditional monitoring. After fault injection, a large amount of sudden indicator data may be generated (such as monitoring the indicators of 100 Pods at the same time). Direct transmission to the data processing center is prone to data loss due to network congestion. However, the buffering effect of Kafka message queues can smooth the data transmission peaks and ensure that all real-time indicators are received completely. At the same time, time window aggregation reduces the amount of data (such as reducing the amount of data to 1 / 10 of the original after 1 second aggregation), reducing the pressure of subsequent analysis and calculation. Data verification avoids invalid data from interfering with anomaly identification (such as an incorrect CPU utilization of 120% will lead to false anomalies), ensuring the quality of the data input for anomaly analysis.

[0066] The fourth step, "comparing and analyzing the real-time monitoring indicators with the baseline data of the system indicators, identifying abnormal indicators that exceed the baseline threshold range, locating the scope of the abnormal impact by combining the fault resource association map, and generating an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type, and associated resource identifier," is essentially about achieving accurate identification of abnormal indicators and association with fault context, providing complete information for subsequent recovery operations on "what the abnormality is, which fault caused it, and which resources are affected." In terms of technical implementation, firstly, PromQL queries are used to extract real-time monitoring metrics (e.g., service A request latency of 295ms) aggregated after a fault from Prometheus, along with the corresponding system metric baseline data (e.g., service A request latency baseline [81ms, 219ms]). Each metric is compared to determine if it exceeds the threshold range. If the real-time value exceeds the range for 10 consecutive seconds (to avoid misjudgment due to instantaneous fluctuations), it is considered an abnormal metric, and the abnormal metric information is recorded (e.g., "metric name: service A request latency, real-time value: 295ms, baseline range: [81ms, 219ms]"). Next, the associated fault identifier (e.g., "fault-001") and fault type (e.g., "network latency") of the current fault are obtained from the fault injection operation log. Then, the fault is invoked... The query interface of the resource association graph (such as `getDownstreamServices(resourceId)`) locates the scope of the anomaly based on the resource identifier corresponding to the anomaly indicator (such as the PodID of service A). For example, downstream services B and C associated with service A are marked as "potentially affected objects". Finally, an anomaly alarm log is generated according to a preset format. The log fields include anomaly indicator information, anomaly occurrence time (the start time of the real-time indicator continuously exceeding the threshold, such as "2001-01-01 01:02:30"), associated fault identifier ("fault-001"), fault type ("network latency"), associated resource identifier (a list of PodIDs of service A), and the log is written to the ELK distributed logging system. For example, if a service A request is delayed by 295ms for 10 seconds, exceeding the baseline [81ms, 219ms], it is determined to be an abnormal indicator. By querying the fault resource association graph, it is found that the downstream of service A is service B, and an abnormal alarm log is generated: "Abnormal indicator: Service A request delay of 295ms (baseline [81ms, 219ms]), abnormal time: 2001-01-01 01:02:30, fault identifier: fault-001, fault type: network delay, associated resources: pod-service-A-001, pod-service-A-002, affected downstream: service-B".The significance of this step lies in solving the problem of "disconnect between anomalies and faults" in traditional monitoring. Traditional monitoring can only inform that "service A has high latency," but it cannot associate it with specific fault tasks or locate the affected downstream services. This forces operations and maintenance personnel to manually investigate "which operation caused the high latency" and "which services will be affected," which is time-consuming and prone to omissions. However, the anomaly alarm logs generated in this step are directly bound to the corresponding fault injection task by associating fault identifiers (such as the "fault-001" network latency fault). The fault resource association graph clearly identifies the affected downstream services (such as service B), providing precise guidance for subsequent automated recovery. For example, recovery operations can directly target service A corresponding to the "fault-001" fault, while focusing on monitoring service B, improving recovery efficiency and targeting.

[0067] The fifth step, "conducting in-depth analysis of the abnormal alarm logs and associated real-time monitoring metrics, calculating the deviation of the abnormal metrics, and marking the scope of impact on downstream services, recording the abnormal metrics, deviation, and scope of impact in the fault injection operation log," supplements and deepens the abnormal alarm information, providing quantitative data on the degree of fault impact for subsequent test report generation, while also improving the completeness of the fault injection operation log. Technically, the deviation of the abnormal metrics is first calculated using the formula "(real-time metric value - baseline mean) / baseline mean × 100%". For example, if the real-time request latency for service A is 295ms and the baseline mean is 150ms, the deviation is approximately 96.7% (295-150) / 150 × 100%). This value quantitatively reflects the degree of impact of the fault on the metrics. Next, based on the fault resource association graph and real-time monitoring metrics, the scope of impact on downstream services is further marked—by querying the real-time monitoring metrics (such as request latency and error rate) of downstream services (e.g., service B). If the request latency of service B increases from 80ms to 180ms (within its baseline [60ms, 200ms]), it is marked as "potentially affected." If the error rate increases from 0% to 5% (exceeding its baseline [0%, 1%]), it is marked as "already affected." Finally, the abnormal indicator name, deviation range (e.g., "96.7%)", and scope of impact (e.g., "service A is abnormal, service B is potentially affected, service C is unaffected") are added to the fault injection operation log (associated with the corresponding fault identifier) ​​to ensure that the log contains complete information of "operation-abnormality-impact." For example, in the fault injection operation log for the network latency fault of service A, a new record is added: "Abnormal indicator: service A request latency, deviation range: 96.7%, scope of impact: service A (abnormal), service B (potentially affected, latency 180ms), service C (unaffected)." The significance of this step lies in addressing the problem of "lack of impact assessment" in traditional monitoring. Traditional monitoring can only identify anomalies but cannot quantify the degree of anomalies (such as how much latency has increased) or determine whether the fault has propagated downstream, making it impossible to assess the severity of the fault. In contrast, the deviation calculation in this step can quantify the impact of the fault (e.g., a deviation of 96.7% indicates that the fault has a significant impact on service A), and the downstream impact range marking can predict the risk of fault propagation (e.g., service B, although not exceeding the baseline, has already experienced increased latency and needs to be monitored in advance). At the same time, the data added to the fault injection operation log provides direct data support for the "Fault Impact Analysis" section in the subsequent test report (e.g., the report can show "Service A's request latency deviated by 96.7%, affecting 2 downstream services"), improving the objectivity and reference value of the test report.

[0068] In summary, by constructing a full-level monitoring and data acquisition system, generating dynamic system indicator baselines, reliably transmitting and verifying real-time data, identifying and locating anomalies and impacts related to faults, and conducting in-depth analysis to supplement logs, this approach achieves more comprehensive data collection, more accurate identification, strong correlation, and in-depth analysis of monitoring data after cloud-native system failures. This process solves the problems of incomplete monitoring layer coverage, rigid baselines, disconnect between anomalies and faults, and lack of impact assessment in traditional systems. It not only provides more accurate anomaly location and impact scope information for automated fault recovery, ensuring that recovery operations can address specific problems and mitigate downstream risks, but also accumulates complete quantitative data for the generation of subsequent test reports, ultimately improving the monitoring effectiveness and data value of the entire fault testing process.

[0069] In one embodiment of the present invention, the step of performing automated fault recovery operations based on the abnormal alarm log includes: S51, Construct a predefined fault recovery strategy library, which stores the correspondence between fault types and recovery strategies, as well as the execution priority rules and triggering conditions of the recovery strategies; S52, extract the fault type, fault duration and associated resource identifier according to the abnormal alarm log, match the recovery strategy that meets the triggering conditions in the fault recovery strategy library, and select the optimal recovery strategy according to the execution priority rule; S53, according to the optimal recovery strategy, call the cloud-native platform interface to perform recovery operations, including service instance restart, traffic routing switch, configuration version rollback and resource quota adjustment; S54 monitors the execution status of recovery operations in real time and records the recovery operation process log; S55, verify the recovery effect of the recovery operation. If the recovery fails, trigger the secondary recovery mechanism. After multiple recovery failures, generate a manual intervention work order.

[0070] As described in steps S51-S55 above, through the design of the steps of "building a fault recovery strategy library, matching and filtering the optimal recovery strategy, executing recovery operations, recording recovery status logs, and verifying recovery effects and failure handling", the system realizes automated recovery operations for cloud-native system faults based on abnormal alarm logs. At the same time, the system ensures that the recovery process is traceable and the effect is evaluable through log recording and effect verification, and finally restores the faulty system to a normal operating state. This provides complete data support for the recovery process for subsequent test report generation and completes the closed-loop test process from "fault injection to monitoring and alarm to automated recovery".

[0071] Fault recovery is the core closed-loop link in the fault testing process. The ultimate goal of fault injection is to verify the system's resilience and recovery capabilities. Only through effective recovery operations can we determine whether the system has the ability to recover from faults. At the same time, recovery operations can minimize the impact of faults on the test environment and prevent the fault from spreading and causing other test tasks to fail. Due to the diverse types of faults in cloud-native systems (such as network, service, and configuration faults) and the complex relationships between resources and services (such as service A depending on service B, and service B depending on the database), recovery operations need to address three major issues: "what strategy to use for recovery," "which resources to prioritize for recovery," and "how to verify the recovery effect." Different fault types require different recovery strategies (such as switching traffic for network faults and rolling back configurations for configuration errors); faults in core services (such as payment services) should be prioritized for recovery to avoid impacting critical business operations; after recovery, it is necessary to verify whether the system has truly returned to normal to avoid "false recovery." Without a unified recovery mechanism, recovery operations will fall into the predicament of "chaotic strategies," "disordered priorities," and "unclear effects." For example, a service restart strategy may be mistakenly used for network faults, or non-core service faults may preempt core service recovery resources, resulting in low recovery efficiency or even failure. Therefore, it is necessary to implement strategic, automated, and verifiable fault recovery through specific steps.

[0072] Traditional solutions often rely heavily on manual intervention for fault recovery: In selecting recovery strategies, operations personnel must manually query suitable strategies based on the fault type. For example, after a service crash, they must manually decide whether to restart the Pod or switch to a standby instance, which is inefficient and prone to extending downtime due to incorrect strategy selection. Regarding priority ranking, there is no unified mechanism; the recovery order of core and non-core services needs manual coordination. If core service recovery is delayed, critical business operations may be unavailable for extended periods. Verifying recovery effectiveness requires manual login to monitoring platforms to view metrics or calling health check interfaces, which is time-consuming and prone to missing key verification items (e.g., only verifying service availability without checking the status of downstream dependent services). Furthermore, traditional methods lack complete recovery log records, making it impossible to trace operational details during the recovery process, hindering subsequent analysis of recovery bottlenecks. To address these shortcomings, this invention proposes a solution of "predefined strategy library + automated matching and filtering + full-process monitoring and verification." By constructing a fault recovery strategy library containing strategies, priorities, and trigger conditions, it automatically matches the optimal strategy based on anomaly alarm logs, executes recovery operations according to priority and records logs, and finally verifies the recovery effect, replacing manual recovery and improving recovery efficiency and reliability.

[0073] Specifically: The first step, "Building a predefined fault recovery strategy library, which stores the correspondence between fault types and recovery strategies, as well as the execution priority rules and triggering conditions of the recovery strategies," is fundamentally about providing clear recovery solutions and execution rules for different fault types, addressing the problem of "no unified standard for recovery strategies," and providing a strategic foundation for automated recovery. Technically, this involves first identifying common fault types in cloud-native systems (consistent with the fault injection types mentioned above, such as network latency, service downtime, configuration errors, database deadlocks, and CPU overload), and then matching corresponding recovery strategies for each fault type. The matching criteria include the root cause of the fault and the effectiveness of the strategy—for example, the root cause of a "network latency" fault is an abnormal network link, corresponding to a "traffic routing switch" (switching traffic to a backup link via Istio) or "enabling CDN acceleration" strategy; the root cause of a "service downtime" fault is an abnormal Pod instance, corresponding to a "service instance restart" (calling the Kubernetes API to restart the Deployment) or "ReplicaSet scaling" (increasing the number of replicas) strategy; and the root cause of a "configuration error" fault is abnormal configuration parameters, corresponding to a "configuration version rollback" (pulling historical configurations from Spring Cloud Config) strategy. Next, execution priority rules (numerical weights, ranging from 1 to 100, with higher values ​​indicating higher priority) are defined for each recovery strategy. Strategies related to core business have higher priority; for example, the priority of the "payment service downtime to service instance restart" strategy is set to 100, and the priority of the "log service delay to traffic switching" strategy is set to 30. Simultaneously, triggering conditions (fault duration, degree of metric anomaly, etc.) are defined. For example, the triggering condition for the "traffic switching" strategy for a "network latency" fault is "service response time exceeds the baseline by 50% for 1 minute," to avoid triggering recovery operations due to instantaneous fluctuations.Finally, this information is stored in a structured format (such as JSON) as a fault recovery strategy library. Each strategy includes the fields `faultType` (fault type), `recoveryStrategy` (recovery strategy name), `operation` (specific operation steps, such as the path and parameters of calling the Istio API), `priority` (priority weight), and `triggerCondition` (trigger condition). For example, a complete strategy is: `{"faultType":"network latency","recoveryStrategy":"traffic routing switch","operation":"call IstioAPI / apis / networking.istio.io / v1alpha3 / virtualservices to update VirtualService and route traffic to the standby instance IP list [10.0.2.1,10.0.2.2]","priority":80,"triggerCondition":"service response time > baseline × 1.5 and duration ≥ 60s"}`. The significance of this step lies in solidifying recovery experience into executable rules through a predefined policy library, avoiding the subjectivity and errors of manual policy selection. Traditional manual methods may, due to insufficient experience, mistakenly adopt the "service restart" policy for "configuration error" failures (which cannot solve the configuration problem). However, the policy library clearly defines the "configuration rollback" policy corresponding to "configuration error," ensuring the accuracy of policy selection. At the same time, priority rules and triggering conditions ensure that recovery operations are executed "on demand" and "in an orderly manner." For example, traffic switching is only triggered when network latency lasts for 1 minute and exceeds the baseline by 50%, avoiding frequent recovery operations. Prioritizing the execution of core service policies ensures the rapid recovery of critical businesses.

[0074] The second step, "extracting the fault type, fault duration, and associated resource identifier from the abnormal alarm logs, matching recovery strategies that meet the triggering conditions in the fault recovery strategy library, and selecting the optimal recovery strategy according to execution priority rules," aims to accurately match and select the most suitable recovery strategy based on actual fault information, solving the problem of "disordered strategy matching" and ensuring that recovery operations are highly targeted and prioritized reasonably. Technically, key information is first extracted from the abnormal alarm logs: fault type (e.g., "network latency," obtained directly from the "fault type" field of the log), fault duration (calculated from the "fault occurrence time" of the log and the current time, e.g., if the fault occurrence time is 14:00 and the current time is 14:01, the duration is 60 seconds), and associated resource identifier (e.g., the PodID of service A, "pod-service-A-001," obtained from the "associated resource identifier" field of the log). Then, the fault recovery strategy library is traversed to match strategies that meet the "fault type matches + fault duration meets the triggering conditions," for example, if the fault type is "network latency" and the duration is... For 60 seconds, the system matches two policies in the policy library with the trigger condition of "duration ≥ 60s": "Traffic Routing Switch" and "Enable CDN Acceleration". Then, it sorts the policies by their `priority` field value from highest to lowest (e.g., "Traffic Routing Switch" has a priority of 80, and "Enable CDN Acceleration" has a priority of 60). The policy with the highest priority, "Traffic Routing Switch", is selected as the optimal recovery policy. If multiple policies have the same priority (e.g., both have a priority of 80), the system further filters based on the business importance of the associated resource. For example, if the associated resource is core service A, the policy with less impact on the business (instead of "CDN Acceleration," which has more complex configuration) is selected. For instance, if the anomaly alarm log shows "Fault Type: Network Delay, Anomaly Occurrence Time: 14:00, Associated Resource: pod-service-A-001", current time 14:01, duration 60 seconds, two policies are matched from the policy library, and "Traffic Routing Switch" is selected as the optimal recovery policy based on priority. The significance of this step lies in solving the problems of "low efficiency and poor targeting" in traditional manual matching strategies. Manual matching requires querying the strategy library one by one, which is time-consuming and prone to overlooking triggering conditions (such as triggering recovery without waiting for 1 minute). This step, on the other hand, automatically extracts key information from logs, quickly matches strategies that meet the conditions, and filters by priority to ensure that core needs are prioritized (such as prioritizing more efficient traffic switching strategies). At the same time, the combination of associated resource identifiers makes strategy selection more in line with the actual resource situation. For example, the backup instance information corresponding to the Pod of Service A has been preset in the strategy operation steps, without the need for manual querying, thus improving matching efficiency and accuracy.

[0075] The third step, "to execute recovery operations by calling the cloud-native platform interface according to the optimal recovery strategy, including service instance restart, traffic routing switching, configuration version rollback, and resource quota adjustment," transforms the selected recovery strategy into actual operations. By calling the cloud-native platform interface, automated recovery is achieved, solving the problem of "recovery operations relying on manual execution." In terms of technical implementation, firstly, the `operation` field of the optimal recovery strategy is parsed to clarify the operation type and required parameters: If it is a "service instance restart" operation, the Deployment name (e.g., "deployment-service-A") and namespace (e.g., "default") of the target service need to be obtained from the system resource metadata. Then, the ` / apis / apps / v1 / namespaces / {namespace} / deployments / {name} / rollout / restart` interface of the Kubernetes API is called, and `--timeout=30s` (graceful exit time, obtained from the service deployment data for the graceful exit configuration of service A) is set in the request parameters. If it is a "traffic routing switch" operation, the list of backup instance IPs (e.g., [10.0.2.1, 10.0.2.2]) needs to be obtained from the fault resource association graph. Then, the ` / apis / networking.istio` interface of the Istio API is called. The `.io / v1alpha3 / virtualservices` interface updates `spec.http.route.destination.host` in the request body to the Service name corresponding to the backup instance IP. For "configuration version rollback" operations, the configuration version number (e.g., "200101011300") one hour before the failure needs to be retrieved from the Spring Cloud Config configuration history, and the configuration update interface ` / actuator / refresh` is called to roll back the target Pod's ConfigMap to that version. For "resource quota adjustment" operations (e.g., CPU overload failure), the current CPU limit value of the target Pod (e.g., "1C") needs to be retrieved from the system resource metadata, and the Kubernetes API's ` / api / v1 / namespaces / {namespace} / pods / {name} / patch` interface is called to adjust the CPU limit value to 1.5 times (e.g., "1.5C"). During execution, the cloud-native platform's authentication mechanism (e.g., ServiceAccount token) ensures interface call permissions and prevents unauthorized operations. For example, when performing the "service instance restart" operation, the Kubernetes API is called to restart "deployment-service-A". The API returns 200 OK, indicating that the operation was successfully initiated and the Pod restart process begins.The significance of this step lies in its ability to significantly improve recovery efficiency by replacing manual operations with automated API calls. Traditional manual service restarts require logging into the Kubernetes console or executing commands, taking approximately 5 minutes. In contrast, this step can initiate the operation within 10 seconds via API calls, without any manual intervention. Furthermore, the operation parameters are directly obtained from fundamental data such as system resource metadata and fault resource association graphs, avoiding human input errors (such as misspellings of Deployment names) and ensuring the accuracy of the recovery operation. In addition, the diverse operation types cover the main fault scenarios of cloud-native systems, such as network, service, configuration, and resource overload faults, all with corresponding recovery operations, achieving fully automated recovery across all scenarios.

[0076] The fourth step, "Real-time monitoring of the execution status of the recovery operation and recording of the recovery operation process log," aims to track the entire process status of the recovery operation, ensure that the operation is traceable, and provide a basis for subsequent effect verification and problem investigation. In terms of technical implementation, the methods and metrics for status monitoring are first determined for different recovery operation types: For the "service instance restart" operation, the `status.readyReplicas` (number of ready replicas) and `status.updatedReplicas` (number of updated replicas) are monitored through the Kubernetes API's ` / apis / apps / v1 / namespaces / {namespace} / deployments / {name} / status` interface to determine if the Pod has restarted successfully (e.g., `readyReplicas` equals the expected number of replicas); for the "traffic route switching" operation, the ` / apis / networking.istio.io / v1alpha3 / virtualservices / {name} / status` interface is monitored to determine if the `Ready` status in `status.conditions` is "True" to confirm if the routing configuration has taken effect; for the "configuration version rollback" operation, the Spring Cloud Config configuration query interface is used to monitor whether the target Pod's configuration version has been updated to a historical version. Next, these status indicators are collected at 1-second intervals, recording the operation start time, intermediate state changes (e.g., "Service restarting: readyReplicas=0" "Service restart completed: readyReplicas=3"), and API call return information (e.g., API status code, error message) to the recovery operation process log. The log format is consistent with the fault injection operation log (JSON format, associated with a unique fault identifier) ​​and stored in the ELK log system. For example, the process log of the service A restart operation records: "Fault ID: fault-001, Operation type: Service instance restart, Start time: 14:01:00, Intermediate state: 14:01:05 readyReplicas=0; 14:01:15 readyReplicas=3, API return: 200 OK, Operation completion time: 14:01:15".The significance of this step lies in solving the problem of "invisible status" in traditional recovery operations. After manually performing a recovery operation, it is necessary to frequently check the status to confirm the progress. However, this step monitors and records the status in real time, which can clearly trace the entire operation process. For example, if the log shows "service restart is stuck at readyReplicas=2", it can quickly locate that one of the Pods failed to start. At the same time, the process log provides data support for the "Recovery Operation Details" section of the subsequent test report (such as "service restart took 15 seconds" shown in the report), and also provides a basis for root cause analysis after recovery failure (such as recording that the API returned a 500 error in the log, which can help investigate interface permission issues).

[0077] The fifth step, "verifying the recovery effect of the recovery operation, triggering a secondary recovery mechanism if recovery fails, and generating a manual intervention work order after multiple recovery failures," ensures that the recovery operation is truly effective, avoiding "false recovery" (such as successful operation but the system still being abnormal), and providing a manual fallback solution for faults that cannot be automatically recovered. Technically, recovery effect verification is divided into two parts: service availability verification and metric recovery verification. Service availability verification involves calling the target service's health check interface (e.g., the ` / health` interface, obtaining the interface path from the service deployment data), sending an HTTP GET request, and if a status code of 200 is returned and the `status` field in the response body is "UP", then the service is considered available. Metric recovery verification uses the Prometheus query interface to obtain key metrics (such as service response time, error rate, and CPU utilization) within 5 minutes after recovery, calculates the average value, and compares it with the system metric baseline data. If all metrics are within the baseline threshold range (e.g., service A's response time recovers to [81ms, 219ms]), then the metrics are considered to have recovered normally. If both criteria are met, the recovery is considered successful. If either criterion is not met (e.g., the service is available but the response time still exceeds 300ms), the recovery is considered unsuccessful. The cause of failure is analyzed (e.g., the database connection was not rebuilt after the service restart, resulting in response delay), and a secondary recovery mechanism is triggered. The secondary recovery adopts an alternative strategy (e.g., after the "service instance restart" fails, the "ReplicaSet scaling" strategy is used to increase the number of replicas). The retry interval is set to 3 minutes, and the default number of retries is 2. If it still fails after 2 retries, the API of the ticket management system (e.g., JIRA) is called to generate a manual intervention ticket. The ticket content includes the fault identifier (fault-001), fault type (network latency), recovery operations performed (traffic switching, service restart), failure reason (response time not recovered), and associated resource identifier (pod-service-A-001), and is pushed to the operation and maintenance alarm platform (e.g., PagerDuty) via Webhook. For example, after service A restarts, the health check returns "UP", but the response time is still 280ms (exceeding the baseline [81ms, 219ms]), indicating that the recovery has failed. Analysis reveals that the database connection pool has not been released, triggering a second recovery (calling the database API to kill the old connection). If the response time drops to 190ms after the second recovery, the recovery is considered successful; if it still fails, a manual work order is generated.The significance of this step lies in solving the problem of "unverified effectiveness" in traditional recovery methods. Manual recovery may simply confirm that the service has started without checking whether the indicators have returned to normal, resulting in a false recovery where "the service has started but the business is still unavailable." This step, however, ensures that the system truly returns to normal through dual verification of availability and indicators. The secondary recovery mechanism can automatically correct some recovery failures (such as database connection problems), reducing manual intervention. Manual work orders provide a backup for complex faults (such as underlying network hardware failures), preventing faults from remaining unresolved for a long time, forming a three-level protection system of "automatic recovery - secondary retry - manual backup."

[0078] In summary, by constructing a predefined fault recovery strategy library, automatically matching and filtering the optimal strategy based on abnormal alarm logs, calling cloud-native interfaces to execute automated recovery operations, recording recovery status logs in real time, dual-verifying recovery effects, and establishing a three-level failure handling mechanism, the "strategic, automated, traceable, and verifiable" fault recovery of cloud-native systems has been achieved. This process solves the problems of incorrect strategy selection, low execution efficiency, invisible status, and lack of effect verification in traditional manual recovery. It can not only quickly restore the faulty system to a normal state and verify the system's resilience and recovery capabilities, but also provide complete recovery operation data for subsequent test reports. Ultimately, it completes the closed loop of the entire automated fault injection and recovery test, improving the completeness and reliability of the test process.

[0079] In one embodiment of the present invention, the step of generating and archiving a fault test report based on the fault injection operation log, system monitoring data, and recovery operation process log includes: S61, the fault injection operation log, the system monitoring data and the recovery operation process log are associated with each other by a unique fault identifier to form a fault test data chain; S62 calculates the performance impact during the fault, fault detection time, recovery execution time, and total recovery time, and generates a system resilience score based on the fault impact range and recovery effect. S63 generates a standardized test report based on a preset template, which includes basic test information, fault injection details, monitoring indicator analysis, recovery operation details, and system resilience score. S64, associate and store the standardized test report, the fault injection operation log, the system monitoring data, the recovery operation process log, and the abnormal alarm log in the designated storage system, generate a unique archive ID, and associate it with the test management system.

[0080] As described in steps S61-S64 above, the system is designed with the steps of "establishing data associations to form data chains, quantifying key indicators and system resilience scores, generating standardized test reports, and storing, archiving, and tracing related data." Based on the fault injection operation logs, system monitoring data, and recovery operation logs from the aforementioned steps, the system achieves aggregated analysis, quantitative evaluation, standardized presentation, and standardized archiving of data throughout the entire fault testing process. This results in complete test outcomes that support subsequent tracing and system optimization, completing the closed loop from "fault test execution" to "results accumulation."

[0081] Test report generation and archiving are the "output and value solidification" stages of the fault testing process. The ultimate goal of fault injection, monitoring, and recovery is to verify system resilience and form actionable optimization criteria. The test report is a quantitative summary of this process, intuitively presenting the impact of faults, recovery effects, and system shortcomings. Archiving ensures that test data and reports are traceable in the long term, facilitating subsequent reproduction of test scenarios, comparison of resilience changes across different system versions, or compliance audit requirements. Because cloud-native fault testing involves data from multiple stages (injection logs, monitoring metrics, recovery logs), and the data formats and sources are scattered (logs are stored in ELK, metrics in Prometheus), a lack of a unified data association and analysis mechanism can lead to fragmented report content (e.g., only presenting fault injection results, lacking recovery effects) and a lack of quantitative basis for evaluation (e.g., only describing "system recovery to normal" without specific time consumption and metric data). Furthermore, traditional archiving methods are chaotic (e.g., reports are stored locally, logs are stored on servers), easily leading to data loss or inability to associate data. Therefore, dedicated steps are needed to achieve data association, quantitative analysis, standardized report generation, and associated archiving.

[0082] This invention proposes a solution of "data-driven correlation + quantitative analysis + templated reports + associated archiving." It automatically correlates data from multiple stages using a unique fault identifier, calculates quantitative indicators and resilience scores, generates standardized reports based on preset templates, and finally stores the reports and end-to-end data in a designated system. This completely solves the inefficiency and non-standardization problems of traditional methods. Specifically: The first step, "establishing a data association between the fault injection operation log, the system monitoring data, and the recovery operation log through a unique fault identifier to form a fault test data chain," aims to break down the data silos across multiple stages and construct a complete causal data chain of "injection-monitoring-recovery," providing a unified data foundation for subsequent analysis and report generation. Technically, the source of the unique fault identifier is first clarified—this identifier is a unique ID generated when storing user fault test requirement data in the aforementioned steps (e.g., "fault-20010101-001"). The subsequent fault injection operation log and recovery operation log are all associated with this identifier, and the system monitoring data is bound to this identifier through the "associated fault identifier" field (added when generating abnormal alarm logs in the aforementioned steps). Next, using a data association engine (such as the Flink stream processing engine or a Python data processing script), with the unique fault identifier as the association key, corresponding data is extracted from each data source: the fault injection operation log (including injection time, target resource, and execution status) and the recovery operation process are extracted from ELK. Logs (including recovery steps, time taken, and status) are used to extract system monitoring data (such as service response time and CPU utilization time-series data) before, during, and after the fault from Prometheus. Information on the time of anomaly occurrence and deviations from metrics are extracted from anomaly alarm logs. Finally, the correlated data is organized chronologically into a fault test data chain. This data chain includes four main modules: "Basic Test Information" (fault type, impact range, priority), "Fault Injection Data" (injection time, tool, execution result), "Monitoring Metric Data" (baseline before fault, peak value during fault, average value after recovery), and "Recovery Operation Data" (recovery strategy, execution time, effect verification result). This ensures that data at each stage can be traced back to the same fault identifier. For example, the data chain associated with the unique fault identifier "fault-20010101-001" contains complete data such as "Fault Type: Network Latency," "Injection Time: 14:00," "Peak Response Time During Fault: 295ms," "Recovery Strategy: Traffic Switching," "Recovery Time: 15 seconds," and "Post-Recovery Response Time: 180ms." The significance of this step lies in solving the problem of traditional data being "scattered and unrelated"—when manually organizing data, a lot of time is spent matching "which monitoring data corresponds to a certain fault injection log," while this step automatically associates data through the unique fault identifier, integrating data from multiple stages within minutes to form a complete data chain; at the same time, the data chain ensures that no key information is missed when the report is generated. For example, if the report needs to present "fault detection time," the difference between "anomaly occurrence time and fault injection start time" can be directly extracted from the data chain without querying the log again, thus improving the efficiency of report generation.

[0083] The second step, "calculating the performance impact magnitude, fault detection time, recovery execution time, and total recovery time during the fault, and generating a system resilience score based on the fault impact range and recovery effect," aims to evaluate the fault impact and recovery efficiency through quantitative indicators, comprehensively measure the system's fault resistance capability through the resilience score, provide objective data support for the report, and avoid the ambiguity of qualitative descriptions. Technically, the calculation logic and data sources for each quantitative indicator are clear: the performance impact magnitude is calculated as "(mean value of indicators during the fault - mean value of baseline before the fault) / mean value of baseline before the fault × 100%", with data sourced from system monitoring data (mean value of indicators during the fault) and system indicator baseline data (mean value of baseline before the fault). For example, if the baseline mean delay for service A before the fault is 150ms, and the mean delay during the fault is 295ms, the performance impact magnitude = (295-150) / 150 × 100% ≈ 96.7%; the fault detection time is calculated as "time of anomaly occurrence - time of fault injection start", with data... The data is derived from the fault injection operation log (injection start time) and the anomaly alarm log (anomaly occurrence time). For example, the injection start time is 14:00, the anomaly occurrence time is 14:00:30, and the detection time is 30 seconds. The recovery execution time is calculated as "recovery operation completion time - recovery operation start time". The data comes from the recovery operation process log. For example, the recovery start time is 14:01, the completion time is 14:01:15, and the execution time is 15 seconds. The total recovery time is calculated as "recovery operation completion time - fault injection start time", that is, 14:01:15 - 14:00 = 1 minute and 15 seconds. The system resilience score is calculated using a multi-dimensional weighted approach, with weights set based on business importance. It comprises four dimensions: fault tolerance (weight 30%, with higher scores for smaller performance impacts; e.g., ≤30% impact earns 30 points, 30%-50% earns 20 points, and >50% earns 10 points), fault detection efficiency (weight 25%, with higher scores for shorter detection times; e.g., ≤1 minute earns 25 points, 1-3 minutes earns 15 points, and >3 minutes earns 5 points), recovery effectiveness (weight 30%, with higher scores for smaller deviations from baseline after recovery; e.g., ≤10% deviation earns 30 points, 10%-20% earns 20 points, and >20% earns 10 points), and recovery efficiency (weight 15%, with higher scores for shorter total recovery time; e.g., ≤1 minute earns 15 points, 1-3 minutes earns 10 points, and >3 minutes earns 5 points). The total score is 100 points. For example, if the performance impact of a network latency failure in Service A is 96.7% (fault resistance 10 points), the detection time is 30 seconds (detection efficiency 25 points), the post-recovery indicator deviation is 20% (recovery effectiveness 20 points), and the total recovery time is 1 minute and 15 seconds (recovery efficiency 10 points), the system resilience score is 10+25+20+10=65 points.The significance of this step lies in addressing the problem of "lack of quantification" in traditional assessments—qualitative descriptions cannot accurately reflect system shortcomings, while quantitative indicators can intuitively present key information such as "failure caused a 96.7% increase in response time" and "recovery time of 75 seconds," making it easier to pinpoint weak links in the system (e.g., short detection time but long recovery time indicates that the recovery strategy needs optimization); the system resilience score provides a cross-scenario and cross-version comparison standard. For example, the V1 version system resilience score is 65 points, while the V2 version score is 80 points after optimization, which can directly quantify the resilience improvement brought about by version iteration and provide a clear direction for system optimization.

[0084] The third step, "Generate a standardized test report based on a preset template, including basic test information, fault injection details, monitoring indicator analysis, recovery operation details, and system resilience score," focuses on ensuring a consistent report structure and complete content through template design, while automatically filling in quantitative data and charts to enhance the report's professionalism and readability. In terms of technical implementation, a pre-designed report template is first designed, built using the Freemarker or Velocity template engine. The template structure is fixed and includes five main modules: Basic test information (test name, execution time, environment identifier, unique fault identifier, tester), with data sourced from user fault testing requirements and the system configuration center; Fault injection details (fault type, target resource, injection tool, execution result, effectiveness verification result), with data sourced from fault injection operation logs; Monitoring indicator analysis (pre-fault baseline, indicator changes during fault, post-recovery indicators, performance impact, including line chart comparison), with data sourced from system monitoring data and system indicator baseline data, and indicator comparison charts automatically generated using ECharts or Matplotlib; Recovery operation details (recovery strategy, execution steps, time consumption of each stage, effect verification result), with data sourced from recovery operation process logs; System resilience score (scores for each dimension, total score, score explanation, optimization suggestions), with data sourced from the quantitative indicators and resilience score calculated in the second step, and optimization suggestions generated based on the score shortcomings (e.g., if the recovery efficiency score is low, it is recommended to "optimize the recovery strategy execution logic and reduce interface call time"). The templates support dynamic data population. For example, the indicator analysis module automatically inserts a "line chart of response time before, during, and after the fault," and the recovery details module automatically generates a "recovery operation timeline table." When generating reports, the data association engine maps and populates the data in the fault test data chain according to the template fields, generating an HTML format report, which is then converted into multiple format files using a PDF / Excel export tool (meeting different scenario needs, such as PDF for archiving and Excel for secondary data analysis). For example, in the standardized report for a network latency fault in Service A, the test basic information module displays "Unique fault identifier: fault-20010101-001," the monitoring indicator analysis module includes a response time line chart and a label stating "Performance impact 96.7%," and the system resilience score module displays "Total score 65 points, recovery efficiency needs optimization" and specific suggestions.The significance of this step lies in solving the problems of "disorganized format and incomplete content" in traditional reports. The templated design ensures that all reports have a unified structure, including five core modules regardless of the tester, facilitating cross-team sharing (e.g., the operations and maintenance team and the development team can quickly find the "recovery operation details" or "metric analysis" they are interested in); automatic data and chart filling avoids human error (e.g., data deviation when manually drawing charts), while improving the professionalism of the reports. For example, line charts can intuitively show the trend of indicator changes and are easier to understand than pure text descriptions; optimization suggestions are generated based on quantitative data, avoiding subjective assumptions (e.g., "recovery time 75 seconds" corresponds to the suggestion of "optimizing recovery strategy"), providing actionable directions for system optimization.

[0085] The fourth step, "Associating and storing the standardized test report, fault injection operation log, system monitoring data, recovery operation log, and anomaly alarm log in a designated storage system, generating a unique archive ID and associating it with the test management system," aims to achieve long-term, standardized association and storage of test results and end-to-end data, ensuring traceability and reusability. Technically, the selection of the designated storage system is first determined: standardized test reports and structured logs (fault injection and recovery operation logs) are stored in a relational database (such as MySQL) for easy querying and association; unstructured system monitoring time-series data (such as Prometheus metrics) are stored in a time-series database (such as InfluxDB) to optimize time-series data read / write performance; and anomaly alarm logs are stored in the ELK system to maintain consistency with the original logs. During storage, all data is associated with a unique fault identifier and a newly generated unique archive ID (using UUID format, such as "archive-20010101-001"). The archive ID serves as the query entry point, allowing quick retrieval of all data (reports + logs + metrics) for a specific test. Next, the API of the test management system (such as JIRA or TestRail) is called to associate the archive ID with the test record. For example, a test record is created in JIRA, and the test name, unique fault identifier, archive ID, and report download link are filled in to achieve linked queries of "test task - archived data". Finally, data lifecycle management rules are established (such as retaining archived data for 3 years, and automatically archiving it to low-cost storage after expiration), balancing storage costs and compliance requirements. For example, standardized reports, fault injection logs, monitoring metrics, recovery logs, and anomaly alarm logs for service A network latency faults are all associated with the archive ID "archive-20010101-001" and stored in MySQL, InfluxDB, and ELK. At the same time, this archive ID is associated in the JIRA test task. When a query is needed later, entering the archive ID will retrieve all relevant data. The significance of this step lies in solving the problems of "disorganized and untraceable" traditional archiving. When archiving manually, reports and logs are stored in a scattered manner, and querying test data requires opening folders and systems one by one. However, this step associates all data through the archive ID, and the query can be completed in seconds (e.g., entering the archive ID allows you to download the report and view the corresponding log). At the same time, associating with the test management system facilitates project management. For example, when viewing all test tasks of an iteration in JIRA, you can directly jump to the corresponding report and data through the archive ID without switching systems. Data lifecycle management avoids the waste of storage resources and ensures long-term compliance.

[0086] In summary, by using a unique fault identifier to link data from multiple stages to form a data chain, calculating quantitative indicators and system resilience scores, generating standardized test reports based on preset templates, and finally storing the reports and end-to-end data in a designated system and linking them to a test management platform, this approach achieves automated association, quantitative evaluation, standardized reporting, and standardized archiving of cloud-native fault test data. It not only efficiently generates objective and complete test reports, providing accurate evidence for system optimization, but also ensures long-term traceability of test data, supporting subsequent reproduction, comparison, and compliance audits. Ultimately, it completes the value loop of the entire automated fault injection and recovery test, improving the usability and management efficiency of test results.

[0087] like Figure 2 As shown, the present invention also provides a cloud-native-based automated fault injection and recovery testing system, comprising: The data acquisition module is used to acquire basic test data for the cloud-native system under test, including system resource metadata, service deployment data, and user fault test requirement data. The model building module is used to build a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules and fault resource association graphs based on the test basic data. The fault injection module is used to perform automated fault injection operations according to the fault test execution model and record fault injection operation logs. The monitoring and alarm module is used to collect and analyze system monitoring data based on the system operating status after the fault injection operation, generate an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously record the fault injection operation log. The fault recovery module is used to perform automated fault recovery operations based on the abnormal alarm logs; The report generation module is used to generate and archive a fault test report based on the fault injection operation log, the system monitoring data, and the recovery operation process log.

[0088] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a cloud-native automated fault injection and recovery testing method.

[0089] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a cloud-native automated fault injection and recovery testing method.

[0090] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0091] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A cloud-native-based automated fault injection and recovery testing method, characterized in that, include: Obtain basic test data for the cloud-native system to be tested, including system resource metadata, service deployment data, and user fault testing requirements data; Based on the aforementioned test data, a fault test execution model is constructed, which includes fault parameter mapping rules, fault injection scheduling rules, and fault resource association graphs. Based on the fault test execution model, perform automated fault injection operations and record fault injection operation logs; Based on the system operating status after the fault injection operation, collect and analyze system monitoring data, generate an anomaly alarm log containing abnormal indicator information, anomaly occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously record the fault injection operation log. Based on the aforementioned abnormal alarm log, perform automated fault recovery operations; A fault test report is generated and archived based on the fault injection operation log, the system monitoring data, and the recovery operation process log.

2. The cloud-native-based automated fault injection and recovery testing method according to claim 1, characterized in that, The steps to obtain the basic test data for the cloud-native system to be tested include: Obtain system resource metadata through cloud-native platform interfaces; Obtain service deployment data through the service registration component; The system receives user fault test requirement data through a visual configuration interface. The fault test requirement data includes fault execution sequence and test priority, and target fault type. The system resource metadata, service deployment data, and user fault test requirement data are subjected to consistency verification. The correspondence between service instances and container instances, the scope of fault impact, and the matching of system resources are checked. If inconsistencies are found, a prompt message is generated and guidance is provided for correction. The verified system resource metadata, service deployment data, and user fault test requirement data are stored in the corresponding directories of the system configuration center, and a unique identifier ID is generated for each type of data.

3. The cloud-native-based automated fault injection and recovery testing method according to claim 2, characterized in that, Based on the aforementioned test data, the steps for constructing a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules, and a fault resource association graph include: Based on the relationship between system resource metadata and service deployment data, establish a mapping rule between fault types and fault parameters of cloud-native fault injection tools; Based on the fault execution sequence and test priority of the user fault test requirement data, and in conjunction with the fault parameter mapping rules, a fault injection scheduling rule is formulated. Based on the system resource metadata and the service deployment data, construct a fault resource association map between the fault impact range and physical resources; The validity of the fault parameter mapping rule, the fault injection scheduling rule, and the fault resource association graph is verified. The verified fault parameter mapping rules, fault injection scheduling rules, and fault resource association graph are integrated into a fault test execution model.

4. The cloud-native-based automated fault injection and recovery testing method according to claim 3, characterized in that, According to the fault test execution model, the steps for performing automated fault injection operations include: Based on the fault parameter mapping rules and the target fault type in the user's fault testing requirements, the corresponding cloud-native fault injection tool and calling parameters are matched to generate specific fault injection instructions. Based on the fault resource association graph, candidate target resources are determined, their current running status is queried and invalid resources are eliminated to determine the target resource set for fault injection. According to the fault injection scheduling rules, the task triggering conditions are monitored through the timed task framework, the API interface of the fault injection tool is called in order of priority, and the fault injection operation is performed on the target resource set according to the fault injection instructions. Obtain the execution result of the fault injection operation. If the execution is successful, record the fault identifier, target resource, start time and execution status to the fault injection operation log. If the execution fails, record the reason for failure and trigger the retry mechanism. If the retry fails after a preset number of times, generate an alarm message. After the fault injection operation is executed, the status changes of the target resource are monitored in real time to verify whether the fault has taken effect successfully. If it has not taken effect, the cause is analyzed and the parameters are adjusted and the operation is re-executed until the fault takes effect or the operation limit is reached.

5. The cloud-native-based automated fault injection and recovery testing method according to claim 4, characterized in that, Based on the fault test execution model, the steps of collecting and analyzing system monitoring data, generating anomaly alarm logs containing abnormal indicator information, anomaly occurrence time, associated fault identifiers, fault types, and associated resource identifiers, and synchronously recording fault injection operation logs include: Based on the hierarchical structure of the cloud-native system to be tested, deploy corresponding monitoring agents in the infrastructure layer, container runtime layer, and application service layer, and configure the metric collection range, frequency, and transmission method for each layer. Obtain historical system indicator data before fault injection, perform data cleaning and aggregation, use statistical analysis methods to calculate the baseline threshold range of each indicator, and generate system indicator baseline data; After the fault injection operation begins, real-time monitoring metrics of the cloud-native system to be tested are collected and transmitted to the data processing center for aggregation and verification via a message queue. The real-time monitoring indicators are compared and analyzed with the baseline data of the system indicators to identify abnormal indicators that exceed the baseline threshold range. The scope of the abnormal impact is located by combining the fault resource association map and an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier is generated. Perform in-depth analysis on abnormal alarm logs and associated real-time monitoring metrics, calculate the deviation of abnormal metrics, and mark the scope of impact on downstream services. Record the abnormal metrics, deviation, and scope of impact in the fault injection operation log.

6. The cloud-native-based automated fault injection and recovery testing method according to claim 5, characterized in that, Based on the aforementioned abnormal alarm log, the steps for performing automated fault recovery operations include: Construct a predefined fault recovery strategy library, which stores the correspondence between fault types and recovery strategies, as well as the execution priority rules and triggering conditions of the recovery strategies; Extract the fault type, fault duration, and associated resource identifier from the abnormal alarm log; match the recovery strategy that meets the triggering conditions in the fault recovery strategy library; and select the optimal recovery strategy according to the execution priority rule. The optimal recovery strategy is used to call the cloud-native platform interface to perform recovery operations, including service instance restart, traffic routing switching, configuration version rollback, and resource quota adjustment. Monitor the execution status of recovery operations in real time and record the recovery operation process log; Verify the recovery effect of the recovery operation. If the recovery fails, a secondary recovery mechanism is triggered. After multiple recovery failures, a manual intervention work order is generated.

7. The cloud-native-based automated fault injection and recovery testing method according to claim 6, characterized in that, The steps for generating and archiving a fault test report based on the fault injection operation log, system monitoring data, and recovery operation log include: The fault injection operation log, the system monitoring data, and the recovery operation process log are linked together using a unique fault identifier to form a fault test data chain. Calculate the performance impact during the failure, failure detection time, recovery execution time, and total recovery time, and generate a system resilience score based on the scope of the failure impact and the recovery effect; Generate a standardized test report based on a preset template, which includes basic test information, fault injection details, monitoring indicator analysis, recovery operation details, and system resilience score; The standardized test report, the fault injection operation log, the system monitoring data, the recovery operation log, and the anomaly alarm log are associated and stored in a designated storage system, generating a unique archive ID and associating it with the test management system.

8. A cloud-native-based automated fault injection and recovery testing system, used to implement the method described in any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to acquire basic test data for the cloud-native system under test, including system resource metadata, service deployment data, and user fault test requirement data. The model building module is used to build a fault test execution model that includes fault parameter mapping rules, fault injection scheduling rules and fault resource association graphs based on the test basic data. The fault injection module is used to perform automated fault injection operations according to the fault test execution model and record fault injection operation logs. The monitoring and alarm module is used to collect and analyze system monitoring data based on the system operating status after the fault injection operation, generate an abnormal alarm log containing abnormal indicator information, abnormal occurrence time, associated fault identifier, fault type and associated resource identifier, and synchronously record the fault injection operation log. The fault recovery module is used to perform automated fault recovery operations based on the abnormal alarm logs; The report generation module is used to generate and archive a fault test report based on the fault injection operation log, the system monitoring data, and the recovery operation process log.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images