Automatic toughness verification system and method for cloud native system

Abstract

The invention discloses an automatic toughness verification system and method for a cloud native system. The system is composed of an intelligent scene management module, a compilation period fault injection actuator, a semantic perception integrated interface, a four-dimensional verification and evaluation engine and a panoramic visualization and continuous optimization module. According to the method, a semantic test plan is generated through a structured service fault scene library, fault logic and atomic task identification are bound in a deterministic mode through condition compiling in the construction stage, and a special test program is generated; a preset fault is triggered during business execution, function correctness, recovery compliance, state consistency and performance influence four-dimensional verification is carried out in parallel, an aggregation result is generated to generate a score quantitative evaluation toughness index, and a high-value test scene is intelligently recommended based on a coverage rate to form a closed loop. According to the method, fault injection and business semantics are accurately associated, binary zero test code residues are produced, the verification depth and efficiency are improved by one order of magnitude, and the method is suitable for high-compliance scenes such as finance and AI training.

Description

Technical Field

[0001] This application relates to the field of cloud-native software engineering technology, and in particular to an automated resilience verification system, verification method, computer-readable storage medium, and electronic device for cloud-native systems. Background Technology

[0002] With the increasing adoption of cloud-native architectures, the complexity of automated business processes based on task orchestration continues to rise. Existing fault injection and resilience verification technologies are fragmented, mainly exhibiting the following four technical paths, all of which struggle to meet the needs of systematic and in-depth resilience verification. Furthermore, they also have significant shortcomings in achieving accurate business semantic association, production security isolation, and multi-dimensional in-depth verification:

[0003] 1. Chaos engineering tools (such as Chaos Mesh and LitmusChaos): These focus on the infrastructure resource layer (nodes, networks, Pods, containers), injecting random, non-specific faults. Their core drawback is their inability to perceive business semantics, making it difficult to simulate precise business errors (such as the "USER_ALREADY_EXISTS" error code) in specific business steps like "creating a MySQL database user." The test scenarios are disconnected from real-world fault modes, essentially constituting "indiscriminate blind injection," and cannot verify fault-tolerance logic for specific business scenarios. For example, testing the startup of a Hive service can only randomly terminate Pods, failing to simulate business errors such as "database user creation conflicts."

[0004] 2. Traditional test execution frameworks (integrated into CI / CD): These frameworks perform functional assertions by executing predefined test cases through scripts. Their drawbacks include a single verification dimension and superficial results, outputting only a binary "pass / fail" judgment. They cannot automatically quantify and evaluate key resilience metrics such as Recovery Time Objective (RTO), distributed resource state consistency, and recovery path compliance. Furthermore, they rely on manually writing numerous verification scripts, resulting in low efficiency. For example, when verifying database failure recovery, they can only check the service restart status and cannot automatically verify deeper metrics such as data consistency and recovery time.

[0005] 3. Passive monitoring and observability systems (e.g., Prometheus + Grafana + distributed tracing): These systems passively monitor, collect, and analyze the operational status of the production environment after an incident, operating on a "post-incident" basis. They cannot proactively and systematically simulate failures and verify recovery behavior before deployment. While they can provide rich runtime data, they lack proactive failure injection capabilities and structured verification methods. For example, they can only analyze logs after a production failure and cannot verify recovery capabilities under specific failure modes in advance.

[0006] 4. Code-embedded fault injection: Conditional branches or test switches are hard-coded in the business code, and fault triggering is controlled by configuration. The drawbacks include production code pollution (test logic remains in the production binary, violating compliance requirements in heavily regulated sectors such as finance and government), performance overhead (runtime condition judgments introduce additional CPU consumption), maintenance complexity (inconsistent configurations across multiple environments), and technology stack limitations (often customized for specific languages, lacking a unified cross-language solution). For example, inserting an `if config.InjectError` branch into a Go code block results in test logic being present in the production binary.

[0007] In summary, the shortcomings of existing technologies mainly include fragmented technology systems, semantic disconnect, shallow verification, and high compliance risks. Specifically, these shortcomings are: (1) lack of semantic awareness, with no semantic connection between fault injection and business operations, and a disconnect between test scenarios and real fault modes; (2) shallow verification with a single dimension, lacking a multi-dimensional parallel verification framework, making it difficult to quantify key resilience indicators; (3) insufficient precise control, with difficulty in precisely controlling the timing and location of fault injection, and an inability to simulate complex multi-step fault scenarios; (4) high production safety and compliance risks, with test code remaining in the production environment; (5) poor technology stack adaptability, lacking a unified cross-language solution, especially insufficient support for static languages ​​such as Go and Rust; (6) performance overhead cannot be ignored, with runtime interception or conditional judgment introducing additional losses; (7) lack of closed-loop optimization mechanisms, with a disconnect between testing and system improvement, and no intelligent feedback closed loop of "test-analysis-optimization" has been formed. It is evident that existing technologies have systemic deficiencies in terms of compliance, accuracy, verification depth, and technology adaptability. Therefore, it is urgent to develop a fault injection and resilience verification method that can ensure absolute production safety and support accurate and in-depth verifiable verification. Summary of the Invention

[0008] To address the aforementioned issues, this application proposes a novel automated resilience verification system and method for cloud-native systems. In summary, this invention proposes an automated resilience verification system and method characterized by "scenario-driven, semantic awareness, precise injection, multi-dimensional verification, and closed-loop optimization." It constructs a closed-loop workflow platform driven by a structured business fault scenario library, centered on four-dimensional parallel automated verification, using a recoverable task orchestration system as the verification object, and employing a compile-time conditionally isolated fault injection executor to achieve precise fault simulation.

[0009] The present invention mainly adopts the following technical strategies:

[0010] I. Core Verification Methods

[0011] Semantic test plan generation: Based on a structured business fault scenario library and user test intent, automatically generate test plans that include clear fault injection points, fault types, and multi-dimensional verification rules;

[0012] Semantic-aware fault triggering: Through deep integration with the task execution engine of the recoverable task orchestration system, it can perceive the business context in real time and trigger preset faults at precise business task execution times;

[0013] Compile-time bound fault simulation: A compile-time fault injection executor is used to strongly bind the fault simulation logic with the specific atomic task identifier of the system under test during the software construction phase, generating a dedicated test program containing deterministic fault behavior;

[0014] Multi-dimensional parallel verification: When the system executes the recovery process, automated verification and data collection are performed in parallel across four dimensions: functional correctness, recovery compliance, state consistency, and performance impact assessment.

[0015] Closed-loop optimization and recommendation: Based on historical test data, the coverage rate is calculated, and the next round of high-value test scenarios is intelligently recommended to drive the continuous evolution of resilience capabilities.

[0016] II. Accurate Compile-Time Fault Simulation Techniques Oriented to Task Orchestration Semantics

[0017] Semantic binding mechanism: The fault simulation code is deterministically associated with the unique identifier of the atomic task defined in the recoverable task orchestration system at compile time;

[0018] Behavioral substitution: Fault logic is designed as a substitution or wrapper for the normal execution function of the target task to ensure the accuracy of the injection timing;

[0019] Complex Fault Mode Simulation: Capable of simulating real-world fault behaviors such as standard error codes for cloud services, intermediate resource states, long-term operations with progress feedback, and controllable delays.

[0020] Environmental isolation guarantee: By building tags, physical isolation between fault codes and production code is achieved, ensuring that testing behavior is not leaked into the production environment;

[0021] Deterministic execution guarantee: The fault behavior is completely fixed after compilation, and the performance is consistent in each test, ensuring that the results are repeatable and verifiable.

[0022] III. Semantic-Aware Automated Resilience Verification System

[0023] The verification system includes the following core modules that work together:

[0024] Intelligent Scene Management Module: Responsible for managing the structured scene library, parsing test intent, and automatically generating test plans;

[0025] Semantic-aware integration interface: Provides standardized interface with recoverable task orchestration systems to achieve accurate awareness of business context;

[0026] Compile-time fault injection executor: Based on the test plan, it drives the compilation and construction of the system under test to achieve precise implantation and triggering of fault logic;

[0027] Four-dimensional verification and evaluation engine: It has the ability to perform parallel execution and real-time comparison of functional correctness, recovery compliance, state consistency and performance impact assessment;

[0028] Panoramic Visualization and Continuous Optimization Module: Provides panoramic visualization of tests, comprehensive report generation, coverage analysis, and intelligent scenario recommendation functions.

[0029] IV. Methods for Constructing a Structured Business Fault Scenario Library

[0030] Hierarchical classification system: Based on cloud-native system domain analysis, a multi-level fault classification system is established, including resource supply, network connectivity, data services, application runtime, and configuration security.

[0031] Semantic mapping relationship: Establish a precise semantic mapping between each type of fault scenario and the underlying technical implementation (cloud service API, standard error code, atomic task identifier);

[0032] Scene composition and orchestration rules: Supports flexible combination of basic fault scenarios, dependency settings and trigger condition configuration to form composite business scenarios;

[0033] Scenario Value Quantification Model: Dynamically evaluates the testing value of a scenario by combining factors such as business risk level, historical failure frequency, and test coverage.

[0034] V. A Four-Dimensional Parallel System Resilience In-Depth Verification and Analysis Method

[0035] Parallel verification architecture:

[0036] Functional correctness verification: Verify the achievement of the final business objectives and key intermediate states;

[0037] Recovery compliance verification: Verify the consistency between the actual recovery steps and the preset design specifications and state machine;

[0038] State consistency verification: Verify data consistency and resource leak-free status by comparing distributed resource snapshots;

[0039] Performance impact assessment: Quantifying key indicators such as recovery time objective (RTO) and recovery point objective (RPO);

[0040] Comprehensive evaluation algorithm: A configurable weighted algorithm is used to aggregate the four-dimensional results and generate a comprehensive resilience score;

[0041] Root cause localization mechanism: Based on correlation analysis of multi-dimensional validation data, automatically locate bottlenecks and design flaws in the recovery process;

[0042] Real-time comparison capability: Supports parallel acquisition, real-time comparison and immediate alarm of four-dimensional verification data.

[0043] VI. Intelligent Optimization and Recommendation Methods for Resilience Testing Based on Coverage

[0044] Dynamic coverage calculation model: Based on a structured scene library, it calculates the proportion of covered scenes in real time and identifies coverage blind spots;

[0045] Multi-factor value assessment algorithm: comprehensively evaluates the testing value of scenarios based on factors such as business criticality, density of historical problems, implementation complexity, and risk exposure.

[0046] Adaptive recommendation strategy: Based on the current coverage gap, team testing resources, and system evolution stage, intelligently recommend the optimal combination of test scenarios for the next round;

[0047] Optimal resource allocation model: A resource allocation strategy that maximizes the overall resilience verification value under the constraint of limited test resources;

[0048] Evolution trend analysis: Track the changes in resilience scores across multiple rounds of testing, predict the system's resilience level, and guide the direction of architecture optimization.

[0049] VII. Resilience Verification Methods for Special Application Scenarios (for Complex Systems such as AI Training)

[0050] Enhanced upon the general four dimensions:

[0051] Deep verification of distributed state consistency: Focuses on verifying the synchronization of multi-node model weights, consistency of training progress, and integrity of checkpoints;

[0052] Hardware heterogeneous compatibility verification: Fault simulation and recovery verification for dedicated hardware such as GPUs and RDMA;

[0053] Training convergence guarantee verification: Verify whether model training can continue correctly after fault recovery and whether the convergence trend is normal;

[0054] High-performance network degradation verification: Simulating the impact of performance degradation of high-speed networks such as RDMA and InfiniBand on training efficiency and the system's response capability.

[0055] Specifically, this application provides the following technical solutions:

[0056] The first aspect of this application provides an automated resilience verification system for cloud-native systems, such as... Figure 8 As shown, the system includes:

[0057] The intelligent scenario management module is used to parse user test intentions based on a structured business fault scenario library and generate semantic test plans that include target atomic task identifiers, fault types and multi-dimensional verification rules.

[0058] The compile-time fault injection executor is used to deterministically bind fault simulation logic to the corresponding atomic task identifiers in the recoverable task orchestration system according to the test plan during the software build phase, based on the conditional compilation mechanism, to generate a test program containing solidified fault behavior, and physically exclude all fault code during production build.

[0059] A semantically aware integration interface is used to deeply interface with the task execution engine of the recoverable task orchestration system, perceive the business context in real time, and accurately trigger the preset fault in the test program when the target atomic task is executed; wherein, the recoverable task orchestration system refers to a cloud-native task orchestration framework based on a directed acyclic graph (DAG) that supports atomic task checkpoints and incremental recovery (specifically embodied in the embodiments of this application as Hive full-stack Operator, PyTorch training cluster Operator, etc.).

[0060] The four-dimensional verification and evaluation engine is used to perform functional correctness verification, recovery compliance verification, state consistency verification and performance impact assessment in parallel during the system recovery process, and aggregate them to generate a comprehensive resilience score.

[0061] The panoramic visualization and continuous optimization module is used to calculate the failure scenario coverage based on historical test data, identify verification blind spots, and intelligently recommend high-value test scenarios to drive a closed-loop iteration of test-analysis-optimization.

[0062] Furthermore, in the system of this application, the compile-time fault injection actuator achieves production safety isolation through the following mechanism:

[0063] A tagging system is built natively using a programming language, creating independent code branches for the fault simulation logic at compile time;

[0064] During test builds, the build flag is activated to include the fault simulation logic; during production builds, the build flag is disabled to physically eliminate all fault codes.

[0065] The test program and the production program are built on the same source code and are distinguished only by build markers, ensuring that the production binary file has zero fault code residue.

[0066] Furthermore, in this application system, the four-dimensional verification and evaluation engine specifically includes:

[0067] A functional correctness validator is used to verify the degree of achievement of the final business goal and the correctness of intermediate states after fault recovery, based on business semantics.

[0068] The recovery compliance verifier is used to analyze whether the sequence and logic of the actual recovery steps are compliant.

[0069] State consistency verifier is used to perform automated snapshot collection and comparison of distributed resources to detect data inconsistencies and resource leaks;

[0070] Performance impact evaluator is used to quantify the impact of recovery time objective (RTO), recovery point objective (RPO), and fault injection on system performance.

[0071] The comprehensive analyzer is used to aggregate four-dimensional verification results using a configurable weighted algorithm, calculate a comprehensive resilience score, and automatically locate bottlenecks in the recovery process based on multi-dimensional data correlation analysis.

[0072] Furthermore, in this application system, the structured business fault scenario library is constructed using a hierarchical classification system:

[0073] The primary categories include System and Resources, Network and Connectivity, Cloud Component Operations, Storage and Disk, Database Operations, Metadata and Configuration, and Components and Status.

[0074] For each fault scenario, a three-layer semantic mapping is established, consisting of a business semantic layer, a technical implementation layer, and a verification rule layer. The technical implementation layer is associated with cloud service APIs, standard error codes, and atomic task identifiers.

[0075] It supports the combination and orchestration of basic fault scenarios, the setting of dependencies, and the configuration of trigger conditions to form complex business scenarios.

[0076] Furthermore, in the system of this application, the intelligent scene management module includes:

[0077] The scenario library management unit is used to maintain the version, lifecycle, and semantic mapping relationship of the structured business fault scenario library;

[0078] A semantic parsing engine is used to convert test intentions described in natural language into specific fault scenarios and technical implementation parameters;

[0079] The test plan generator is used to automatically generate structured test plans, explicitly specifying target atomic task identifiers, fault injection conditions, recovery strategies, and four-dimensional verification rules.

[0080] Furthermore, in the system of this application, the panoramic visualization and continuous optimization module is used to achieve:

[0081] Dynamic coverage calculation: Based on the structured business fault scenario library, the proportion of covered scenarios is calculated in real time and coverage gaps are identified.

[0082] Multi-factor value assessment, comprehensively evaluating the value of scenario testing based on business criticality, density of historical issues, implementation complexity, and risk exposure;

[0083] Adaptive scenario recommendation: Based on the current coverage status and test resource constraints, intelligently recommend scenario combinations to maximize the overall resilience verification value;

[0084] Evolution trend analysis tracks changes in resilience scores across multiple rounds of testing to predict system resilience levels and guide architecture optimization.

[0085] Furthermore, in this application system, for AI training cluster scenarios, the enhancements to the four-dimensional verification and evaluation engine include:

[0086] Distributed state consistency deep verification is used to verify the weight synchronization of multi-node models, the consistency of training progress, and the integrity of checkpoints.

[0087] Hardware heterogeneous compatibility verification is used for fault simulation and recovery verification of GPU and RDMA dedicated hardware.

[0088] Training convergence guarantee verification is used to verify whether model training can continue correctly and with a normal convergence trend after fault recovery.

[0089] High-speed network degradation verification is used to simulate the impact of RDMA and InfiniBand network performance degradation on training efficiency.

[0090] A second aspect of this application provides an automated resilience verification method for cloud-native systems, wherein the method is applied to the aforementioned system, such as... Figure 7 As shown, the method includes:

[0091] S1: Based on the structured business fault scenario library, the user's test intent is parsed to generate a semantic test plan. The test plan includes verification rules in four dimensions: target atomic task identifier, fault type and functional correctness, recovery compliance, state consistency, and performance impact.

[0092] S2: During the software build phase, the conditional compilation mechanism is used to deterministically bind the fault simulation logic with the corresponding atomic task identifiers in the recoverable task orchestration system, generating a test program containing solidified fault behavior, and physically eliminating all fault codes during production build.

[0093] S3: Start the test program and interact with the task execution engine in real time through the semantic awareness interface. When the target atomic task is executed, a preset fault is precisely triggered to drive the system under test to execute the recovery process.

[0094] S4: During the recovery process, perform four-dimensional verification in parallel, collect verification data in four dimensions: functionality, compliance, consistency, and performance, and perform real-time comparison and quantitative evaluation based on preset rules;

[0095] S5: Aggregate the four-dimensional verification results, calculate the comprehensive resilience score, generate an in-depth test report, and calculate the test coverage based on the structured business fault scenario library to identify verification blind spots;

[0096] S6: Based on coverage gaps, scenario testing value, and system evolution status, intelligently recommend the next round of high-value testing scenarios, forming a closed-loop iteration of testing-analysis-optimization.

[0097] Furthermore, in the method of this application, the four-dimensional verification in step S4 specifically includes:

[0098] Functional correctness verification, based on business semantics, verifies the degree to which the final business goal is achieved and the correctness of intermediate states after fault recovery;

[0099] Restore compliance verification by analyzing whether the sequence and logic of the actual restoration steps are compliant;

[0100] State consistency verification involves automatically collecting and comparing snapshots of distributed resources to detect data inconsistencies and resource leaks.

[0101] Performance impact assessment, quantitative analysis of the impact of recovery time objective (RTO), recovery point objective (RPO), and fault injection on system performance.

[0102] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0103] Memory: Used to store computer programs;

[0104] Processor: Used to execute the computer program to implement the steps of the aforementioned automated resilience verification method for cloud-native systems.

[0105] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned automated resilience verification method for cloud-native systems.

[0106] In summary, compared with the prior art, the present invention has the following technical advantages:

[0107] 1. Precise mapping of business semantics: Deep integration with a recoverable task orchestration system enables unambiguous binding between fault injection points and specific business operations (such as "creating a MySQL instance"), solving the problem of scenario and impact decoupling caused by blind injection at the infrastructure layer in traditional chaos engineering, and significantly improving the attributability and effectiveness of test results.

[0108] 2. Multi-dimensional and comprehensive evaluation: It is the first to create a four-dimensional parallel verification architecture that includes functional correctness, recovery compliance, state consistency and performance impact. It breaks through the limitations of the traditional binary judgment of "pass / fail" test, and can quantitatively evaluate core resilience indicators such as RTO and distributed state consistency, and effectively identify hidden defects such as resource leakage and state inconsistency.

[0109] 3. Significantly improved verification efficiency: Based on intelligent scenario management, automated test plan generation, and verification result aggregation, the entire process is automated, reducing the test cycle from "person / day" to "person / hour," improving efficiency by an order of magnitude and supporting high-frequency, full-coverage routine resilience verification.

[0110] 4. Compile-time injection security isolation: Employing compile-time fault injection technology, fault logic is deterministically implanted during the build phase to generate dedicated test programs. This ensures repeatable fault behavior and eliminates the risk of residual test logic through build markers, meeting the stringent compliance requirements of fields such as finance and government.

[0111] 5. Intelligent closed-loop continuous driving: Through panoramic visualization and continuous optimization modules, test coverage is automatically calculated, verification blind spots are identified, and high-value scenarios are intelligently recommended, forming a data-driven closed loop of "test-analysis-optimization" to promote the iterative evolution of the system's resilient architecture.

[0112] 6. Lifecycle cost optimization: Reduce economic losses and emergency response costs from production failures through proactive defect prevention before deployment; reduce resource consumption through precise targeted injection; shorten the development and debugging cycle of fault-tolerant code by automating the verification and recovery process, thereby comprehensively reducing the quality assurance cost of the system throughout its lifecycle.

[0113] 7. Industry Standardization Promotion Value: Provides a complete and practical standardized methodology and toolchain for the resilience verification of cloud-native systems, systematically improves the reliability verification capabilities of core components such as task orchestration, promotes the upgrade of software quality assurance from functionality to non-functional dimensions such as resilience and recoverability, and facilitates the large-scale and in-depth application of cloud-native technologies. Attached Figure Description

[0114] Figure 1 is a schematic diagram of the overall architecture and complete working loop of the system in this application. Figure 1a Method diagram for generating intelligent scenario management and test plans. Figure 1b A diagram of compile-time semantic binding and conditional compilation methods. Figure 1c A diagram illustrating the construction method for a two-track program based on conditional compilation. Figure 1d A diagram illustrating the precise runtime triggering method for preset faults.

[0115] Figure 2This is a schematic diagram of the tree-like classification system of the structured business fault scenario library in this application.

[0116] Figure 3 is a flowchart illustrating the specific implementation of the compile-time conditional isolation fault injection method in this application. Figure 3a Overview of the core features and value of compile-time conditional fault injection methods. Figure 3b A flowchart for panoramic visualization and intelligent analysis. Figure 3c A complete workflow diagram for resilience verification. Figure 3d Diagram of production safety verification and assurance mechanism.

[0117] Figure 4 This is a schematic diagram of the four-dimensional verification and evaluation engine architecture in this application.

[0118] Figure 5 is a timing diagram for fault testing in the scheme of this application, wherein... Figure 5a This is a timing diagram for a scenario A - a recoverable database connection failure test. Figure 5b This is the timing diagram for the unrecoverable image pull failure test in scenario B.

[0119] Figure 6 This is the closed-loop diagram of the coverage-driven intelligent optimization in this application.

[0120] Figure 7 This is a flowchart illustrating the overall implementation process of the resilience verification method in this application.

[0121] Figure 8 This is a structural diagram of the toughness verification system of this application. Detailed Implementation

[0122] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0123] Example 1: Automated Resilience Verification System and Method for Cloud-Native Systems

[0124] I. Core Design of the Structured Business Fault Scenario Library

[0125] This system incorporates a structured, hierarchical, and categorized business fault scenario library. This library is built upon in-depth domain analysis of cloud-native systems, forming a complete fault classification system and semantic mapping relationships. For example... Figure 2As shown, this scenario library adopts a tree-like classification structure, systematically classifying business failures into multiple primary categories, with each category further subdivided into specific failure scenarios.

[0126] 1. Primary categories include, but are not limited to:

[0127] System and resource categories: covering underlying infrastructure failures such as system anomalies, insufficient resources, and waiting timeouts.

[0128] Network and connectivity issues: These include network layer failures such as network timeouts, network errors, and failures to connect critical components.

[0129] Cloud component operation category: This involves cloud platform service operation failures, such as EIP binding failure, security group configuration failure, etc.

[0130] Storage and Disk Category: Includes storage system failures such as disk mount / unmount failure and disk expansion failure.

[0131] Database operation class: covers operation failures such as database application, recycling, user creation, whitelist configuration, etc.

[0132] Metadata and configuration: This category includes management-related failures such as metadata read / write failures and configuration parsing errors.

[0133] Components and State Classes: These include runtime failures such as component dependency exceptions, state exceptions, and mode switching.

[0134] 2. Semantic mapping mechanism:

[0135] A three-layer semantic mapping relationship was established for each fault scenario:

[0136] (1) Business semantic layer: Clarify the meaning and scope of impact of the fault in specific business scenarios.

[0137] (2) Technical implementation layer: associated with specific cloud service APIs, standard error codes, and task orchestration node identifiers.

[0138] (3) Verification rule layer: Preset four-dimensional verification rules and expected indicators for the fault.

[0139] Through this structured scenario library, the system can transform natural language-based test intentions into precise, executable technical instructions and verification rules.

[0140] II. System Overall Architecture

[0141] This system achieves a complete closed loop from "test intent" to "resilience optimization" through the coordinated work of five core modules:

[0142] 1. Intelligent Scene Management Module

[0143] Function: Based on a structured business fault scenario library, it transforms user testing intentions into precise and executable technical instructions and verification rules.

[0144] Implementation method:

[0145] Scene library management: maintenance such as Figure 2 The structured fault classification system shown supports adding, deleting, modifying, and querying scenarios, as well as version management.

[0146] Semantic parsing engine: maps natural language descriptions to specific fault scenarios and technical implementations.

[0147] Test plan generator: Automatically generates structured test plans based on scenario selection, clearly specifying target task nodes, fault types, injection conditions, and verification rules.

[0148] Output: A structured test plan serves as a blueprint to guide the entire verification process.

[0149] 2. Recoverable task orchestration system (verification object)

[0150] Technical Foundation: A system based on recoverable task orchestration and directed acyclic graph (DAG) is used as the verification object. Such systems decompose complex business processes (such as service activation) into multiple independently executable atomic tasks with checkpointable state, rollback or retry capability, and clearly define the dependencies and execution order between tasks through DAG.

[0151] Core Value: As the core verification object of this system, its atomic tasks and their orchestration logic (DAG) provide natural, high-level, and precise business semantics for resilience verification. The system interfaces with the task orchestration engine through a semantic-aware integration interface to obtain the specific business steps currently being executed (such as executing CreateMysqlInstanceTask) and their context in real time. This provides precise binding targets for compile-time fault injection executors and provides a business semantic-based verification benchmark for the four-dimensional verification and evaluation engine.

[0152] 3. Compile-time conditionally isolated fault injection executor

[0153] Function: As a core technical means to achieve "semantic awareness and precise injection", it deterministically implants the pre-set fault logic into the system under test during the software construction phase.

[0154] Working principle:

[0155] Semantic binding: Based on the target atomic task identifier and fault type specified in the test plan, locate the corresponding task implementation function.

[0156] Conditional compilation: Utilizing the build tags system of programming languages ​​(such as Go's Build Tags), a code branch containing preset fault logic is created and enabled for the target function at compile time, achieving "equivalent substitution".

[0157] Build Generation: Generates an executable program specifically for testing, containing defined fault behaviors.

[0158] Core advantages:

[0159] Absolute guarantee of production safety: Physical elimination of fault codes is achieved through the construction of tags, all test logic is excluded during production builds, and a clean binary file with zero residue is generated.

[0160] Compile-time conditional isolation mechanism: faulty code only takes effect when compiled with specific test tags, completely eliminating the risk of test code polluting the production environment.

[0161] Fault behavior is precise and controllable: It supports precise configuration of multiple dimensions such as the number of faults, probability, and recovery strategy (as shown in Example 2: the first 3 failures, the 4th recovery).

[0162] Semantic deep binding: Faults are bound to specific business task identifiers at compile time, achieving precise association at the business semantic level.

[0163] Highly deterministic behavior: Fault logic is solidified after construction, avoiding the uncertainty of random injection at runtime.

[0164] Wide cross-language compatibility: Based on the compiler's native mechanism, it naturally supports statically compiled languages ​​such as Go and Rust.

[0165] 4. Four-dimensional verification and evaluation engine

[0166] Function: To perform real-time, parallel, and multi-dimensional automated verification and quantitative evaluation throughout the entire process of fault triggering and system recovery.

[0167] Implementation: A parallel analysis architecture deeply integrated with the task execution engine is adopted to perform automated verification from four core dimensions:

[0168] (1) Functional correctness verifier: Based on business semantics, verify whether the final business goal has been achieved after fault recovery.

[0169] (2) Recovery compliance verifier: Analyze whether the order and logic of the recovery steps actually taken by the system are compliant.

[0170] (3) State consistency verifier: performs automated snapshot collection and deep comparison on distributed resources to check for "hidden" problems such as data inconsistency and resource leakage.

[0171] (4) Performance impact evaluator: Quantitatively analyzes the time taken for the entire recovery process and calculates key resilience indicators such as RTO and RPO.

[0172] Comprehensive Analyzer: Aggregates validation results from four dimensions, calculates a comprehensive resilience score, and automatically identifies bottlenecks in the recovery process.

[0173] 5. Panoramic Visualization and Continuous Optimization Module

[0174] Functionality: Provides a user insight interface and transforms testing activities into an intelligent closed loop for continuous improvement of system resilience.

[0175] Implementation method:

[0176] Real-time visualization: Dynamically displays test status, resilience radar chart, and resource topology changes.

[0177] Intelligent Reporting: Automatically generates multi-dimensional in-depth analysis reports.

[0178] Coverage-driven optimization: Coverage is calculated based on a scenario library, and high-value test scenarios are intelligently recommended.

[0179] Closed-loop feedback: driving a continuous improvement cycle of "testing → analysis → optimization → retesting".

[0180] III. Method and Flow

[0181] The method corresponding to the system of the present invention includes the following steps:

[0182] Scenario building and plan generation: Based on the business failure scenario library and user testing intent, generate a test plan that includes precise failure injection points and multi-dimensional verification rules.

[0183] System initialization and baseline establishment: In the isolated test environment, start the test program built based on the compile-time fault injection executor to establish the environment baseline state.

[0184] Preset fault triggering and execution: Drives the execution of the system under test. When the target business task is scheduled, the fault logic implanted during compilation is automatically activated and executed.

[0185] Multi-dimensional parallel verification and evaluation: During the system's fault response and recovery process, four-dimensional automated verification and full-link data collection are performed in parallel.

[0186] Comprehensive scoring and report generation: Comprehensive analysis of the verification results across various dimensions generates a comprehensive resilience score and an in-depth test report.

[0187] Continuous optimization and scenario recommendation: Based on coverage analysis and resilience scoring, intelligently recommend the next round of high-value test scenarios to drive the continuous evolution of system resilience capabilities.

[0188] IV. Comparison of Technological Advantages

[0189]

[0190] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0191] Example 2: Multi-scenario Fault Resilience Testing During the Deployment of Cloud-Native Hive Services

[0192] 1. Architecture of the system under test

[0193] 1.1 Hive Full-Stack Operator Based on Recoverable Task Orchestration

[0194] The system under test is a Hive service operator based on a recoverable task orchestration system. Its design includes theoretical capabilities for task checkpointing and incremental recovery, but the verification system of this invention is needed to actually verify and quantify these capabilities.

[0195]

[0196] Compile-time isolation implementation:

[0197] Go's build tags enable physical isolation of faulty code. Production builds (without tags) exclude all fault injection logic, while test builds (with tags chaos) include the complete fault injection implementation. Business code calls functions indirectly via variables; at compile time, the build tags determine whether the variables point to the actual implementation or the fault injection implementation.

[0198] 1.2 Rollback Strategy

[0199] Dependency-aware rollback:

[0200] Partial fault recovery: Recoverable faults do not affect the execution of subsequent tasks.

[0201] Global process termination: An unrecoverable failure triggers a complete DAG rollback.

[0202] Intelligent status cleanup: Dedicated cleanup logic is preset for each task type.

[0203] 2. Test Scenario Configuration

[0204] 2.1 Configuration of Intelligent Scene Management Module

[0205] Fault injection core strategy:

[0206] Technical implementation: Go language build tag conditional compilation + configuration file driven.

[0207] Build tags: The chaos tag controls the compilation of fault codes.

[0208] Configuration-driven: Fault behavior is dynamically controlled by configuration files, not hard-coded. The triggering conditions for faults (such as the number of retries and fault type) are specified by configuration files; however, the fault logic itself is embedded at compile time, forming an architecture of "compile-time determined logic + runtime configurable parameters".

[0209] Dual timeout fault point configuration:

[0210]

[0211] Fault configuration details:

[0212] Fault Point A: Simulating a scenario of scarce database connection pool resources, configured as a recoverable fault, the first 3 calls to inject a connection failed with errors, and the 4th call restored normal execution.

[0213] Fault Point B: Simulating a large image pull timeout scenario, configured as an unrecoverable fault, the first two calls to inject pull timeout errors, the third call triggers process termination and starts rollback.

[0214] 2.2 Definition of Verification Rules

[0215] Four-dimensional verification system:

[0216]

[0217] Compile-time isolation verification rules:

[0218] Production build verification: Verify that there are no fault codes remaining in the production binary.

[0219] Test build verification: Verify that the test binary contains full fault injection capabilities.

[0220] Runtime behavior verification: Verify that there are no faults during production runtime, and inject faults according to the configuration during test runtime.

[0221] 3. Scenario Execution and Platform Verification

[0222] Scenario A: Verification of recoverable database connection failure

[0223] Execution process and fault injection logic:

[0224]

[0225] System verification results summary:

[0226]

[0227] Scenario B: Verification of Unrecoverable Image Pull Failure

[0228] Execution process and fault injection logic:

[0229]

[0230] System verification results summary:

[0231]

[0232] 4. Comprehensive evaluation across four dimensions

[0233] Verification execution method:

[0234] Functional correctness verification: Real-time monitoring of task completion events to trigger verification;

[0235] Restore compliance verification: Asynchronously analyze execution logs and verify the sequence of steps;

[0236] State consistency verification: The background periodically collects and compares resource snapshots;

[0237] Performance impact assessment: Continuously monitor performance metrics and calculate RTO / RPO;

[0238] The four validators run independently and communicate through message queues to achieve true parallel execution.

[0239] 4.1 Dual-Scenario Resilience Assessment Results

[0240]

[0241] Formula for calculating overall resilience score:

[0242] Scenario A: 100×0.30 + 98×0.25 + 99×0.25 + 96×0.20 = 98.4

[0243] Scenario B: 100×0.30 + 97×0.25 + 98×0.25 + 95×0.20 = 97.6

[0244] 4.2 Verification of Compile-Time Isolation Technology

[0245]

[0246] Test build verification results:

[0247]

[0248] 4.3 Key Improvement Findings

[0249]

[0250] 4.4 Test Coverage Analysis

[0251]

[0252] Current coverage: 2 / 4 = 50% (calculated based on failure mode).

[0253] Next goal: Add probabilistic and condition-dependent fault testing, increasing coverage to 100%.

[0254] Example 3: Verification of the Fault Resilience of AI Training Clusters in Multiple Scenarios

[0255] 1. Test System Architecture

[0256] The system under test is a PyTorch distributed training cluster Operator based on recoverable task orchestration.

[0257] Task orchestration DAG:

[0258]

[0259] Task characteristics description:

[0260] Tasks 1-3: Infrastructure preparation phase, involving GPU resource supply, driver installation, and high-speed network configuration.

[0261] Tasks 4-6: Software deployment phase, involving cluster deployment, storage mounting, and model initialization.

[0262] Tasks 7-8: Training startup phase, involving distributed training startup and monitoring activation.

[0263] Key dependency: The successful execution of the preceding task is a prerequisite for the subsequent task.

[0264] 2. Test Scenario Configuration

[0265] 2.1 Test Configuration Instructions

[0266] This embodiment uses the same compile-time fault injection method, adapted to the hardware dependencies and performance sensitivity of AI training scenarios.

[0267] Key points of fault injection:

[0268] Accurate simulation of two types of faults specific to AI training scenarios:

[0269] (1) Hardware dependency failures: such as GPU driver compatibility issues.

[0270] (2) Performance degradation faults: such as a decrease in high-speed network bandwidth.

[0271] 2.2 Multi-task fault configuration

[0272] Test two related faults in the same workflow:

[0273]

[0274] Explanation of implementation principle:

[0275] Fault point A: Retry logic is injected during compilation. The first few calls return installation failure, and execution resumes after the configured number of calls is reached.

[0276] Fault Point B: Network latency code is injected during compilation to simulate bandwidth degradation by controlling the data transmission rate.

[0277] 3. Scenario Execution and Verification

[0278] Scenario Execution Flow

[0279]

[0280] 4. Verification Results and Technical Value

[0281] 4.1 Summary of Verification Results

[0282]

[0283] 4.2 Compile-time isolation verification

[0284] Key validation results:

[0285] Production code purity: Production build binaries have been verified to contain no faulty code.

[0286] Test functionality integrity: The test build includes full fault injection capabilities.

[0287] Behavioral difference verification: The runtime behavior of production / test builds matches the expected differences.

[0288] Figure 1 shows the overall architecture and complete working loop of the system of the present invention, which is divided into the following four parts:

[0289] Figure 1a This diagram illustrates the core process of intelligent scenario management and test plan generation in this invention. Users input their test intentions via natural language. The system understands the requirements through a semantic parsing engine, matches or expands scenario templates from a structured fault scenario library, extracts key elements such as target atomic tasks and fault types, and ultimately generates a structured, machine-readable automated test plan.

[0290] Figure 1bThis diagram illustrates the compile-time semantic binding and conditional compilation method. It showcases the core compile-time semantic binding process of this invention. Based on the structured test plan, the system precisely locates the source code of the target atomic task, selects the corresponding pre-built fault logic, and implements conditional compilation by adding build flags. Depending on whether build flags are included, the compiler generates a test program containing fault code or a production program that excludes fault code.

[0291] Figure 1c This diagram illustrates the dual-track program construction method based on conditional compilation. It showcases the key mechanism for achieving production safety in this invention—dual-track program construction. The system uses the same set of source code, generating two completely different executable programs depending on whether the build command includes test flags: 1) a test-specific program containing deterministic fault logic implanted at compile time; 2) a production program that completely excludes all test code, ensuring binary-level purity and security.

[0292] Figure 1d This diagram illustrates the precise runtime triggering method for preset faults. When the task execution engine schedules atomic tasks, it triggers the preset fault logic when it executes a target task marked at compile time; for non-target tasks, the business logic executes normally. After a fault is triggered, the system's built-in fault tolerance mechanism is activated, executing the preset recovery process to verify the system's resilience.

[0293] Figure 2 This diagram illustrates the tree-like classification system of the structured business fault scenario library in this solution. The diagram shows the complete classification system of the system's built-in structured business fault scenario library. As shown, fault scenarios are organized in a hierarchical tree structure, including seven primary categories: System and Resources, Network and Connections, Cloud Component Operations, Storage and Disk, Database Operations, Metadata and Configuration, and Components and Status. Each primary category is further subdivided into specific fault scenarios. This classification system is built based on in-depth domain analysis of cloud-native systems, providing a standardized and systematic fault model foundation for scenario-driven resilience verification.

[0294] Figure 3 illustrates the specific implementation flow of the compile-time conditional isolation fault injection method in this invention, which consists of the following four parts:

[0295] Figure 3aThis diagram provides an overview of the core features and value of the compile-time conditional fault injection method. It summarizes the core technical features and value of the method: 1) Compile-time conditional isolation: achieving physical isolation of faulty code through the construction of markers; 2) Semantic precise association: achieving source code-level binding between faults and business tasks; 3) Behavioral determinism guarantee: ensuring the consistency and repeatability of fault triggering; 4) Production safety verification: ensuring zero binary test code residue in the production environment; 5) Intelligent continuous optimization: driving closed-loop iteration based on coverage. These features collectively achieve a fundamental unification of deep testing and production safety, driving a spiral increase in system resilience.

[0296] Figure 3b This is a flowchart illustrating the panoramic visualization and intelligent analysis process after validation. The system collects multidimensional data generated from the four-dimensional validation, providing an intuitive view of resilience and health through a panoramic dashboard. The intelligent report generator automatically analyzes the root causes and scope of impact of problems, the coverage analysis tool assesses the completeness of the test, and finally generates targeted optimization recommendations.

[0297] Figure 3c This diagram illustrates the complete workflow for resilience verification of the present invention. Starting with intelligent scenario management, it progresses through compile-time fault injection, runtime fault triggering, four-dimensional verification and evaluation, panoramic visualization analysis, and coverage-driven optimization, forming a complete closed-loop workflow. Each stage is closely integrated to jointly achieve continuous verification and improvement of system resilience.

[0298] Figure 3d This diagram illustrates the production safety verification and assurance mechanism of this invention. In the production program build process, the compiler parses all conditional compilation instructions, actively excludes all test-marked code blocks, and verifies through intermediate representations to ensure no test code remains. A clean production binary is generated after passing the safety verification check; if the check fails, the build fails and an alarm is triggered. Residual test code is located, repaired, and the build is restarted, fundamentally ensuring the safety of the production environment.

[0299] Figure 4 This diagram illustrates the architecture of the four-dimensional verification and evaluation engine proposed in this invention. It includes four core verification components: 1) Functional correctness verifier: verifies the achievement of business objectives and key intermediate states; 2) Recovery compliance verifier: verifies the compliance of recovery steps and state machine conformity; 3) State consistency verifier: verifies resource state snapshot comparison and data consistency; 4) Performance impact evaluator: evaluates the Recovery Time Objective (RTO) and Recovery Point Objective (RPO). The results from each dimension are aggregated by a comprehensive analyzer to generate a comprehensive resilience score, root cause analysis, and optimization recommendations.

[0300] Figure 5a This is a sequence diagram for scenario A - recoverable database connection failure test in this solution. The diagram shows the complete sequence flow of recoverable failure verification in Example 1: After receiving the user's instruction, the verification system drives the task orchestration system to execute the Hive service activation process. When Task 3 (CreateMysqlInstanceTask) is executed, the recoverable failure logic injected at compile time is triggered. The first three database API calls all return a "DB_CONNECTION_ERROR" connection failure error, triggering the business retry mechanism; after the fourth call reaches the configured MaxFailures limit, execution resumes and the MySQL cluster is successfully created; subsequent tasks 4-10 execute normally. After the workflow is completed, it enters the four-dimensional automated verification to verify the system's recovery capability.

[0301] Figure 5b This is the timing diagram for the unrecoverable image pull failure test in Scenario B of this solution. The diagram shows the complete timing flow of the unrecoverable failure verification in Implementation Example 1: Based on the successful execution of tasks 1-7, the unrecoverable failure logic injected during compilation is triggered when task 8 (CreateHiveServer2Task) is executed. The first two Pod deployment attempts both return an "IMAGE_PULL_TIMEOUT" image pull timeout error; after the third attempt reaches the MaxFailures limit, the system returns a process termination error, triggering a complete dependency-aware reverse rollback (task 8→7→6→5→4→3→2→1), cleaning up all created resources; after the rollback is completed, a four-dimensional in-depth verification is performed to verify the system's safe termination and resource cleanup capabilities.

[0302] Figure 6 This diagram illustrates the coverage-driven intelligent optimization closed-loop mechanism of this invention. Based on a structured business fault scenario library, the system calculates the current test coverage, identifies coverage blind spots, and evaluates scenario value through panoramic visualization and continuous optimization modules, intelligently recommending the next round of test scenarios. After executing the recommended tests and performing four-dimensional verification, the system updates the test asset library or generates optimization insights based on the results, driving system architecture and process optimization. By judging whether the coverage metric meets the target, the system achieves continuous iteration and closed-loop attainment of resilience verification goals.

[0303] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions in the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions in the embodiments of this application.

Claims

1. An automated resilience verification system for cloud-native systems, characterized in that, The system includes: The intelligent scenario management module is used to parse user test intent based on a structured business fault scenario library and generate a semantic test plan that includes target atomic task identifiers, fault types and multi-dimensional verification rules. The compile-time fault injection executor is used to deterministically bind fault simulation logic to the corresponding atomic task identifiers in the recoverable task orchestration system according to the test plan during the software build phase, based on the conditional compilation mechanism, to generate a test program containing solidified fault behavior, and physically exclude all fault code during production build. A semantically aware integration interface is used to deeply interface with the task execution engine of the recoverable task orchestration system, perceive the business context in real time, and accurately trigger the preset faults in the test program when the target atomic task is executed. The four-dimensional verification and evaluation engine is used to perform functional correctness verification, recovery compliance verification, state consistency verification and performance impact assessment in parallel during the system recovery process, and aggregate them to generate a comprehensive resilience score. The panoramic visualization and continuous optimization module is used to calculate the failure scenario coverage based on historical test data, identify verification blind spots, and intelligently recommend high-value test scenarios to drive a closed-loop iteration of test-analysis-optimization.

2. The system according to claim 1, characterized in that, The compile-time fault injection actuator achieves production safety isolation through the following mechanism: A tagging system is built natively using a programming language, creating independent code branches for the fault simulation logic at compile time; During test builds, the build flag is activated to include the fault simulation logic; during production builds, the build flag is disabled to physically eliminate all fault codes. The test program and the production program are built on the same source code and are distinguished only by build markers, ensuring that the production binary file has zero fault code residue.

3. The system according to claim 1, characterized in that, The four-dimensional verification and evaluation engine specifically includes: A functional correctness validator is used to verify the degree of achievement of the final business goal and the correctness of intermediate states after fault recovery, based on business semantics. The recovery compliance verifier is used to analyze whether the sequence and logic of the actual recovery steps are compliant. State consistency verifier is used to perform automated snapshot collection and comparison of distributed resources to detect data inconsistencies and resource leaks; Performance impact evaluator is used to quantify the impact of recovery time objective (RTO), recovery point objective (RPO), and fault injection on system performance. The comprehensive analyzer is used to aggregate four-dimensional verification results using a configurable weighted algorithm, calculate a comprehensive resilience score, and automatically locate bottlenecks in the recovery process based on multi-dimensional data correlation analysis.

4. The system according to claim 1, characterized in that, The structured business fault scenario library is constructed using a hierarchical classification system: The primary categories include System and Resources, Network and Connectivity, Cloud Component Operations, Storage and Disk, Database Operations, Metadata and Configuration, and Components and Status. For each fault scenario, a three-layer semantic mapping is established, consisting of a business semantic layer, a technical implementation layer, and a verification rule layer. The technical implementation layer is associated with cloud service APIs, standard error codes, and atomic task identifiers.

5. The system according to claim 1, characterized in that, The intelligent scene management module includes: The scenario library management unit is used to maintain the version, lifecycle, and semantic mapping relationship of the structured business fault scenario library; A semantic parsing engine is used to convert test intentions described in natural language into specific fault scenarios and technical implementation parameters; The test plan generator is used to automatically generate structured test plans, explicitly specifying target atomic task identifiers, fault injection conditions, recovery strategies, and four-dimensional verification rules.

6. The system according to claim 1, characterized in that, The panoramic visualization and continuous optimization module is used to achieve: Dynamic coverage calculation: Based on the structured business fault scenario library, the proportion of covered scenarios is calculated in real time and coverage gaps are identified. Multi-factor value assessment, comprehensively evaluating the value of scenario testing based on business criticality, density of historical issues, implementation complexity, and risk exposure; Adaptive scenario recommendation: Based on the current coverage status and test resource constraints, intelligently recommend scenario combinations to maximize the overall resilience verification value; Evolution trend analysis tracks changes in resilience scores across multiple rounds of testing to predict system resilience levels and guide architecture optimization.

7. The system according to claim 1, characterized in that, For AI training cluster scenarios, the enhancements to the four-dimensional verification and evaluation engine include: Distributed state consistency deep verification is used to verify the weight synchronization of multi-node models, the consistency of training progress, and the integrity of checkpoints. Hardware heterogeneous compatibility verification is used for fault simulation and recovery verification of GPU and RDMA dedicated hardware. Training convergence guarantee verification is used to verify whether model training can continue correctly and with a normal convergence trend after fault recovery. High-speed network degradation verification is used to simulate the impact of RDMA and InfiniBand network performance degradation on training efficiency.

8. An automated resilience verification method for cloud-native systems, characterized in that, The method is applied to the system as described in any one of claims 1-7, and the method comprises: S1: Based on the structured business fault scenario library, the user's test intent is parsed to generate a semantic test plan. The test plan includes verification rules in four dimensions: target atomic task identifier, fault type and functional correctness, recovery compliance, state consistency, and performance impact. S2: During the software build phase, the conditional compilation mechanism is used to deterministically bind the fault simulation logic with the corresponding atomic task identifiers in the recoverable task orchestration system, generating a test program containing solidified fault behavior, and physically eliminating all fault codes during production build. S3: Start the test program and interact with the task execution engine in real time through the semantic awareness interface. When the target atomic task is executed, a preset fault is precisely triggered to drive the system under test to execute the recovery process. S4: During the recovery process, perform four-dimensional verification in parallel, collect verification data in four dimensions: functionality, compliance, consistency, and performance, and perform real-time comparison and quantitative evaluation based on preset rules; S5: Aggregate the four-dimensional verification results, calculate the comprehensive resilience score, generate an in-depth test report, and calculate the test coverage based on the structured business fault scenario library to identify verification blind spots; S6: Based on coverage gaps, scenario testing value, and system evolution status, intelligently recommend the next round of high-value testing scenarios, forming a closed-loop iteration of testing-analysis-optimization.

9. The method according to claim 8, characterized in that, The four-dimensional verification mentioned in step S4 specifically includes: Functional correctness verification, based on business semantics, verifies the degree to which the final business goal is achieved and the correctness of intermediate states after fault recovery; Restore compliance verification by analyzing whether the sequence and logic of the actual restoration steps are compliant; State consistency verification involves automatically collecting and comparing snapshots of distributed resources to detect data inconsistencies and resource leaks. Performance impact assessment, quantitative analysis of the impact of recovery time objective (RTO), recovery point objective (RPO), and fault injection on system performance.

10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the automated resilience verification method for cloud-native systems as described in claim 8 or 9.

Images