Cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system

Abstract

The application relates to the field of cloud computing operation and maintenance management, and discloses an operation and maintenance fault simulation and training and fault scene automatic arrangement system based on a cloud environment, which comprises a fault modeling module, a fault prediction module, a fault arrangement module, a script execution module and a review optimization module; the fault modeling module models the component state and the fault propagation path of the cloud platform, and generates a fault scene model; the fault prediction module calculates the probability of the fault propagation path, and dynamically determines the optimal fault triggering path; the fault arrangement module automatically generates a containerized fault arrangement script; the script execution module executes the fault arrangement script, and triggers the fault; and the review optimization module generates a review report based on the data of the fault simulation process. Through automatic fault modeling, dynamic prediction, script arrangement and review optimization, the application improves the accuracy and efficiency of fault simulation, and is suitable for complex and multi-component cloud environment operation and maintenance fault simulation and training requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing operation and maintenance, and particularly to an operation and maintenance fault simulation and automatic fault scenario arrangement system based on a cloud environment. BACKGROUND

[0002] With the rapid development of cloud computing and microservice architecture, cloud platforms have become the infrastructure support of modern information technology. However, the large-scale deployment and complex component interaction of cloud platforms make the system structure more complex, the coupling and dependency between components continue to increase, and the types and forms of faults gradually diversify. The faults in the cloud platform not only involve the state abnormalities of a single component, but also can be transmitted to other components through various channels such as network, resources or service calls, forming complex fault links and cascading effects. This complexity brings great challenges to operation and maintenance personnel in the process of fault troubleshooting and repair.

[0003] Existing operation and maintenance fault management systems mostly focus on fault detection and recovery, for example, identifying fault alarms and performing repair operations through monitoring tools, but these technologies only stay at the response level after the fault occurs, lacking preventive functions such as fault simulation and drill. In addition, traditional fault drills mainly rely on static scenario design and manual fault simulation, which have problems such as insufficient flexibility, limited scenario coverage, low execution efficiency, and cannot effectively meet the actual operation and maintenance needs of complex and multi-fault combinations in cloud platforms.

[0004] Further, the existing technology is difficult to realize automatic modeling, dynamic prediction, arrangement execution and review optimization of faults in an integrated platform. Especially in the modeling process of fault scenarios, existing methods usually rely on static rules and cannot flexibly respond to the dynamic changes of multi-component states and fault propagation paths. At the same time, the analysis of fault propagation paths lacks intelligent means, and the fault occurrence probability and influence range are difficult to accurately evaluate, resulting in deviations between fault simulation results and actual situations. In addition, the lack of automatic script generation and execution mechanism makes operation and maintenance personnel need to invest a lot of manpower cost in the process of fault simulation and drill, and the precision and reusability of fault execution are poor.

[0005] Therefore, how to realize dynamic simulation and continuous optimization of complex fault scenarios through accurate fault modeling technology, efficient fault propagation analysis method, automatic fault arrangement and execution mechanism, and systematic review optimization function in the cloud platform environment has become a technical problem that needs to be solved in the current operation and maintenance field. SUMMARY

[0006] In view of the deficiencies of the prior art, the application provides a cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement system, and solves the problems of inaccurate cloud platform fault scenario modeling, unintelligent fault propagation path analysis, low fault simulation and execution efficiency and lack of review optimization mechanism in the prior art.

[0007] To achieve the above object, the application is implemented by the following technical scheme: a cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement system, comprising:

[0008] A fault modeling module for modeling the component state and fault propagation path of the cloud platform, and generating a fault scenario model;

[0009] A fault prediction module for dynamically reasoning the fault propagation path based on a Bayesian network, calculating the probability path of fault propagation, and optimizing the fault trigger sequence;

[0010] A fault arrangement module for automatically generating an executable fault arrangement script according to the fault scenario model and the optimized fault trigger sequence;

[0011] A script execution module for executing the fault arrangement script, simulating the fault scenario in the cloud platform, and performing fault triggering and state monitoring;

[0012] A review optimization module for generating a fault review report based on the data and results of the fault simulation process, and feeding back the optimized fault scenario model and fault propagation probability path.

[0013] Preferably, the fault modeling module models the fault propagation path through a Petri net, and the Petri net comprises:

[0014] A place set for representing the state of the cloud platform components;

[0015] A transition set for representing the triggering and recovery operations of the fault;

[0016] A weight function for describing the state transition weight in the fault propagation path;

[0017] An initial mark set for describing the initial state of the fault propagation.

[0018] Preferably, the fault prediction module dynamically reasons the fault propagation probability through a Bayesian network, and the Bayesian network comprises:

[0019] A node set for representing the state of each component of the cloud platform;

[0020] A directed edge set for describing the dependency relationship of fault propagation between each component;

[0021] Conditional probability distribution, used to represent the probability of a node state given its parent node state.

[0022] Preferably, the fault orchestration module converts the fault scenario model and the fault propagation probability path into a containerized script, which supports execution in a Kubernetes cluster or a Docker Compose environment.

[0023] Preferably, the containerized script includes fault trigger instructions for simulating fault scenarios such as high load, network interruption, and service crash.

[0024] Preferably, the script execution module monitors resource usage, network status, and service logs of cloud platform components during the fault rehearsal process, and records the time points of fault propagation and recovery.

[0025] Preferably, the post-mortem optimization module generates a fault post-mortem report based on the monitoring data during fault propagation, which includes:

[0026] Fault trigger time and fault recovery time;

[0027] Fault propagation path and propagation probability;

[0028] Fault repair operation and execution result.

[0029] Preferably, the post-mortem optimization module optimizes the fault propagation path weight in the Petri net and dynamically adjusts the conditional probability distribution parameters in the Bayesian network based on the fault rehearsal results.

[0030] Preferably, the dynamic adjustment of the conditional probability distribution parameters in the Bayesian network includes:

[0031] Receiving monitoring data provided by the script execution module during the fault rehearsal process;

[0032] Based on the actual data of the fault propagation path, updating the weight of the fault propagation path in the Petri net, and dynamically adjusting the weight value according to the fault trigger frequency and the probability of path occurrence;

[0033] According to the monitoring data and the actual results of the fault propagation path, recalculating the conditional probability distribution of each node in the Bayesian network;

[0034] Feedback the dynamically adjusted conditional probability distribution to the fault prediction module for optimizing the calculation accuracy of the fault propagation probability;

[0035] Synchronously updating the optimized fault propagation path and conditional probability result to the fault modeling module to further improve the fault scenario model, ensuring that the subsequent fault rehearsal is closer to the actual fault state of the cloud platform.

[0036] The application also provides a cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement method, comprising the following steps:

[0037] The component state and fault propagation path of the cloud platform are modeled based on Petri nets to generate a fault scenario model;

[0038] The probability of the fault propagation path is calculated based on a Bayesian network to determine an optimal fault triggering path;

[0039] The fault scenario model and the optimal fault triggering path are converted into a containerized fault arrangement script;

[0040] The fault arrangement script is executed to trigger a fault and monitor the state of each component of the cloud platform;

[0041] A review report is generated based on the data of the fault simulation process, and the fault scenario model and the fault propagation probability path are fed back for optimization.

[0042] The application provides a cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement system, which has the following beneficial effects:

[0043] 1. The application can accurately describe the state changes and mutual dependency of each component by introducing Petri nets to model the component state and fault propagation path of the cloud platform. Compared with the traditional manual fault modeling method, Petri nets have strong expression ability and can support various fault scenario combinations, including concurrent, sequential and nested complex fault scenarios, thereby providing an accurate fault scenario model for subsequent fault propagation analysis and execution.

[0044] 2. The application uses a Bayesian network to perform probability reasoning on the fault propagation path, dynamically evaluates the propagation probability of each component fault by calculating the conditional probability distribution between nodes, and selects the optimal fault triggering path. Unlike the static path analysis method, the Bayesian network can be updated and optimized in real time according to actual simulation data, thereby significantly improving the accuracy and applicability of fault propagation path prediction.

[0045] 3. The application converts the fault scenario model and the optimal triggering path into a standardized containerized script through a fault arrangement module, and automatically executes fault simulation in a cloud platform environment. This process does not require manual intervention and can automatically generate reusable fault scripts according to fault types, triggering sequences and parameter configurations. In addition, it supports various execution environments, ensuring the flexibility and convenience of fault scenario execution.

[0046] 4. The script execution module monitors the status of cloud platform components in real time during the fault triggering process, collecting fault simulation data including resource utilization, service logs, and status changes. This data provides a reliable basis for fault propagation analysis, resource impact assessment, and post-mortem optimization, ensuring that the fault simulation process is observable and traceable, which helps in the rapid location and analysis of problems.

[0047] 5. This invention analyzes the data from the fault simulation process through a debriefing and optimization module, generating a detailed debriefing report, including the fault propagation path, fault causes, resource impact, and repair process. Simultaneously, the system optimizes the fault propagation path weights and Bayesian network conditional probability distribution based on the feedback from the simulation data, thereby continuously iteratively optimizing the fault scenario model, forming a closed-loop optimization mechanism, and improving the system's fault simulation accuracy and prediction performance.

[0048] 6. This invention automates the entire fault drill process through functions such as automatic modeling, dynamic prediction, automatic orchestration, real-time execution, and closed-loop optimization. This reduces manual operation and script maintenance work, improving the efficiency of maintenance fault drills. Simultaneously, the system supports the rapid construction of complex, multi-fault combination scenarios, helping maintenance personnel to efficiently conduct fault diagnosis and emergency drills.

[0049] 7. This invention supports multiple fault scenario types and training environments, and can adapt to different cloud platform architectures, including domestic operating systems and hardware platforms in the domestic IT innovation environment. At the same time, the fault scripts and models are highly reusable, supporting flexible expansion of new fault types and scenarios according to actual needs, meeting diverse operation and maintenance requirements.

[0050] 8. The debriefing reports and drill data generated by the system can be used to build and maintain a fault knowledge base, helping operations and maintenance personnel accumulate fault handling experience. Simultaneously, through debriefing analysis and model optimization, the system can continuously improve fault scenario design and execution strategies, providing strong technical support and decision-making basis for subsequent fault investigation and repair. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the system architecture of the present invention;

[0052] Figure 2 This is a schematic diagram illustrating an example of a fault orchestration script for the present invention;

[0053] Figure 3 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0055] Please refer to the drawings Figure 1 -attached Figure 2 The present application provides a cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement system, which is mainly used for simulating fault scenarios, optimizing fault propagation paths, realizing fault triggering simulation, and analyzing and optimizing the simulation results. The various modules of the system of the present application will be described in detail below with reference to the drawings.

[0056] As Figure 1 shown, the cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement system can include a fault modeling module, a fault prediction module, a fault arrangement module, a script execution module, and a review optimization module.

[0057] In this embodiment, the fault modeling module is used to model the component state and fault propagation path of the cloud platform based on the Petri net, forming a fault scenario model that can be used for fault scenario simulation. This module describes the state, fault triggering and propagation path of each component in the cloud platform through a formal method, providing a basic model for subsequent fault prediction, arrangement and execution.

[0058] In one possible implementation, the Petri net is defined by the following five elements:

[0059] The set P of places is used to represent the different states of each component in the cloud platform;

[0060] The transition set T is used to describe the transition operation between states, including fault triggering and fault recovery;

[0061] The directed arc set F is used to connect the place and the transition, representing the flow relationship between states;

[0062] The weight function W is used to describe the weight on the directed arc, representing the weight of the fault propagation path;

[0063] The initial marking M0 is used to describe the initial state of fault propagation.

[0064] Specifically, in the cloud platform, the state of each component can be abstracted as different places, including but not limited to the following states:

[0065] Normal state p normal represents that the component is in a normal working state;

[0066] Faulty state p fault denotes that the component is faulty;

[0067] Recovery state p recovery denotes that the component recovers from the fault.

[0068] In some embodiments, the fault triggering and recovery process can be described by a set of transitions. For example, the process of a component switching from a normal state to a faulty state can be represented as:

[0069]

[0070] where W denotes the weight of triggering the fault, used to describe the probability or impact degree of fault triggering.

[0071] Alternatively, the process of a component recovering from a faulty state to a normal working state can be represented as:

[0072]

[0073] where W ′ is the weight of the recovery operation, used to measure the cost or time of fault recovery.

[0074] In one possible implementation, the modeling of the fault propagation path is realized by the directed arcs of the Petri net. The directed arcs connect the places and transitions, forming the path of fault propagation. For example, the fault of component A can propagate through the network to affect component B, and the path can be represented as:

[0075]

[0076] where W A→B denotes the weight of fault propagation, which is usually obtained by statistical analysis of historical fault data.

[0077] It should be noted that the fault modeling module further supports reachability analysis of the fault propagation path, which is used to determine the set of all possible reachable faulty states in the system from the initial state M0. The reachability analysis is calculated by the state transition matrix AAA, which includes the forward transition matrix A + and the reverse transition matrix A - , and the specific calculation process is as follows:

[0078] A=A + -A -

[0079] where:

[0080] A + denotes the forward transition, i.e. the fault triggering process;

[0081] A- represents a reverse transition, i.e. a fault recovery process.

[0082] In an exemplary implementation, the fault modeling module generates a fault propagation path graph by traversing all possible state paths. In the path graph, each node represents a fault state of a component, and each edge represents a path and probability of fault propagation.

[0083] In some embodiments, to improve the flexibility and reusability of the modeling, the fault modeling module supports the combined modeling of complex fault scenarios. For example, the system can combine multiple independent fault scenarios into one overall scenario according to actual needs, and the combination methods include:

[0084] Concurrent scenarios, i.e. multiple components fail at the same time;

[0085] Nested scenarios, i.e. a fault triggers other fault propagation;

[0086] Sequential scenarios, i.e. faults are triggered step by step in a predetermined order.

[0087] As an option, the fault modeling module can also dynamically adjust the fault trigger weight W and the recovery weight W ′ based on historical fault data and monitoring indicators. For example, the system can update the weight values based on the frequency and impact of the fault occurrence, thereby optimizing the fault scenario model.

[0088] It should be noted that the output of the fault modeling module is a fault scenario model, which includes:

[0089] A Petri net model describing the state of the cloud platform components;

[0090] Fault propagation paths and their corresponding weights.

[0091] In some other embodiments, the fault scenario model of the fault modeling module can also be linked with real-time monitoring data to provide more accurate initial modeling results for the fault prediction module.

[0092] It can be understood that the modeling accuracy of the fault modeling module has an important influence on the fault prediction, scheduling and rehearsal of the entire system, and therefore in the present application, the Petri net is used for formal modeling, which has strong expression and analysis capabilities and can meet the needs of complex fault scenario modeling.

[0093] For example, assume that there are three service components A, B and C in the cloud platform, which have dependency relationships. The fault propagation path modeling results are as follows:

[0094] Initial state: A normal ,B normal ,C normal

[0095] Fault trigger:

[0096] Fault recovery:

[0097] At this time, the Petri net model and the fault propagation path output by the fault modeling module can be taken as input for subsequent analysis and execution by the fault prediction module and the orchestration module.

[0098] At this time, the Petri net model and the fault propagation path output by the fault modeling module can be taken as input for subsequent analysis and execution by the fault prediction module and the orchestration module.

[0099] In this embodiment, the fault prediction module is used to dynamically infer the fault propagation path based on the Bayesian network, calculate the probability path of the fault occurrence, and determine the optimal fault trigger sequence. The main function of this module is to dynamically evaluate the state probability in the fault propagation process, and provide accurate path and weight information for the fault orchestration module.

[0100] In one possible implementation, the fault prediction module models the fault propagation as a Bayesian network, where the Bayesian network is composed of nodes, edges, and conditional probability distributions, which are described as follows:

[0101] Node: represents the state of each component in the cloud platform, including normal state, fault state and recovery state. For example, node X i represents the state of the i-th component.

[0102] Directed edge: represents the fault propagation relationship between nodes, for example, the fault propagation path from component X i to component X j .

[0103] Conditional probability distribution: represents the probability of a node's state occurring under the state of its parent nodes, i.e. P(X j |Pa(X j )), where Pa(X j ) represents the parent node set of node X j .

[0104] Specifically, the implementation of the fault prediction module includes the following steps:

[0105] In some embodiments, the probability calculation of fault propagation is realized based on the Bayes theorem, which is described as:

[0106]

[0107] Where:

[0108] P(Xj |X i ): Indicates that the fault originates from node X. i Propagate to node X j The conditional probability;

[0109] P(X i ): Represents node X i The prior probability;

[0110] P(X j ): Represents node X j The prior probability.

[0111] In one possible implementation, the total probability of fault propagation can be determined by summing the effects of all parent nodes, as calculated below:

[0112]

[0113] Among them, P fault (X j ) is the parent node X i The probability of failure, P(X) j |X i ) is the propagation probability from parent node to child node.

[0114] Alternatively, the fault prediction module can learn and train node probabilities and conditional probabilities using historical fault data. For example, the system can perform statistical analysis on data from historical fault logs to calculate the frequency of faults in each component and the degree of correlation between its faults and those of other components, thereby determining the initial conditional probability distribution in the Bayesian network.

[0115] It should be noted that during fault prediction, the dynamic reasoning of the fault propagation path is achieved through real-time updates of node states. For example, when the system detects a fault in a component X... i When a fault occurs, the fault prediction module will infer the probability of other nodes failing based on the Bayesian network and determine the optimal path for fault propagation.

[0116] In one exemplary implementation, assuming the cloud platform comprises three components A, B, and C, the fault propagation path is as follows:

[0117] P(A fault ) = 0.2 represents the prior probability that component A will fail;

[0118] P(B fault |A fault =0.8 represents the probability that component B will fail under the condition that component A fails;

[0119] P(C fault |B fault=0.6 represents the probability that component C will fail under the condition that component B fails.

[0120] At this point, the total probability of fault propagation is:

[0121] P(C fault )=P(A fault )·P(B fault |A fault )·P(C fault |B fault ) = 0.2·0.8·0.6

[0122] =0.096

[0123] Specifically, based on the above results, the fault prediction module will select the path with the highest probability of fault propagation as the optimal path, which will then be used by the subsequent fault orchestration module for fault simulation and triggering.

[0124] In some embodiments, to improve the real-time performance and accuracy of fault prediction, the fault prediction module supports dynamically updating the conditional probability distribution. For example, after the script execution module simulates a fault scenario, the system will adjust the conditional probabilities in the Bayesian network based on the feedback results of actual monitoring data, so as to better adapt to the real fault propagation path.

[0125] As an alternative, the fault prediction module can also support dynamic reasoning for multiple scenarios, such as the prediction of concurrent and cascading faults. In concurrent fault scenarios, multiple components may fail simultaneously, and the joint probability of fault propagation can be calculated using the joint distribution of a Bayesian network.

[0126] It should be noted that the output of the fault prediction module includes:

[0127] The probability of each component failing;

[0128] Fault propagation paths and their probability weights;

[0129] Optimal fault triggering sequence.

[0130] These outputs will be used as input to the fault orchestration module to automatically generate the optimal fault triggering script.

[0131] Understandably, through the dynamic reasoning and conditional probability updates of Bayesian networks, the fault prediction module can effectively assess the path and probability of fault propagation, providing accurate data support for subsequent modules.

[0132] In other embodiments, the fault prediction module can also be linked with real-time monitoring data to achieve early prediction and warning of faults. For example, when a sharp increase in the resource utilization of a certain component is detected, the system can predict the probability of the fault occurring in advance and generate the corresponding fault propagation path in advance for verification and drills in subsequent stages.

[0133] In summary, in this embodiment, the fault prediction module effectively calculates the probability of fault propagation paths and selects the optimal path through dynamic reasoning based on Bayesian networks, providing an accurate data foundation for subsequent fault orchestration and execution. It should be noted that the fault prediction module in this invention is not only applicable to single fault scenarios, but also supports flexible prediction of multiple fault combinations and dynamic scenarios.

[0134] In this embodiment, the fault orchestration module is used to automatically generate executable fault orchestration scripts based on the fault scenario model generated by the fault modeling module and the fault propagation probability path provided by the fault prediction module. This module is mainly responsible for the automated combination and logical orchestration of fault scenarios and the dynamic generation of scripts, so as to realize the execution and reproduction of fault scenarios in the cloud platform environment.

[0135] In one possible implementation, the fault orchestration module first receives a Petri net model output by the fault modeling module, which includes the states of each component and the fault propagation relationships between them. Simultaneously, the fault orchestration module receives the optimal fault propagation path and path weights calculated by the fault prediction module; these weights are used to determine the fault triggering order and execution priority.

[0136] Specifically, the execution logic of the fault orchestration module includes the following aspects:

[0137] I. Fault Scenario Combinations

[0138] The fault orchestration module dynamically combines fault scenarios for each component based on the optimal fault propagation path. Alternatively, these combinations of fault scenarios can include the following forms:

[0139] Concurrent scenarios: These are scenarios where failures in multiple components are triggered simultaneously. For example, components A and B can be injected with failures at the same time.

[0140] Sequential scenario: Faults are triggered in a certain order. For example, after component A fails, component B fails.

[0141] Nested scenario: A failure in one component triggers a cascading failure in other components. For example, a failure in component A affects the network state, which in turn causes failures in components B and C.

[0142] For example, if the optimal path output by the fault prediction module is A fault →B fault →Cfault The fault orchestration module will combine fault scenarios according to the path order and assign a trigger priority to each fault scenario.

[0143] II. Generation of Fault Execution Logic

[0144] In one possible implementation, the fault orchestration module combines fault scenarios into executable scripts that can be executed in containerized environments such as Kubernetes or Docker Compose.

[0145] It should be noted that the fault orchestration module uses a standardized description language, such as YAML or JSON, to express the logical definition of fault scenarios, which facilitates subsequent parsing and execution by the execution engine.

[0146] Specifically, the script generation process includes:

[0147] Component Fault Definition: Specify the component and fault type that need to be triggered, such as high CPU load, network interruption, service crash, etc.

[0148] Fault parameter configuration: Set execution parameters for fault scenarios, including fault injection time, duration, and resource utilization.

[0149] Execution order configuration: Set the order in which faults are triggered based on the fault propagation path and weight.

[0150] Example fault orchestration scripts such as Figure 2 As shown in the script:

[0151] Component A triggers a network interruption fault and restricts memory resources.

[0152] After component A completes its failure, component B triggers a high CPU load failure, which lasts for 60 seconds.

[0153] After component B fails, component C simulates a service crash.

[0154] III. Fault Triggering and Dynamic Parameter Adjustment

[0155] In some embodiments, to improve the flexibility of fault scenario execution, the fault orchestration module supports dynamic adjustment of parameters in the fault script. For example, parameters such as fault duration and resource consumption threshold can be dynamically optimized based on real-time monitoring data.

[0156] As an alternative, the fault orchestration module can dynamically update the fault triggering sequence and script content by combining the inference results from the fault prediction module. For example, when real-time monitoring detects that the resource utilization of a certain component is approaching the fault threshold, the fault orchestration module can trigger the fault scenario of that component in advance to verify the system's response capability to faults.

[0157] IV. Fault Script and Environment Adaptation

[0158] It should be noted that the scripts generated by the fault orchestration module support multiple runtime environments, including but not limited to:

[0159] Kubernetes clusters: Automatically orchestrate and execute fault scenarios through YAML configuration files.

[0160] Docker Compose: Enables fault simulation in multi-container environments through container orchestration definition files.

[0161] Adaptation to the domestic IT environment: Adjustments were made to the fault scripts to ensure compatibility with domestic operating systems and hardware platforms.

[0162] As an alternative, the fault orchestration module can also automatically verify and validate scripts to ensure that the execution logic of fault scenarios matches the actual deployment environment.

[0163] V. Script Output and Storage

[0164] The script files generated by the fault orchestration module can be stored in a unified script management library, which facilitates reuse in subsequent drills and fault scenario backtracking.

[0165] In the exemplary implementation, the script is stored along with the following metadata:

[0166] Fault Scenario ID

[0167] Fault Trigger Path

[0168] Fault parameter configuration

[0169] Generation time and version number

[0170] Understandably, the fault orchestration module, through standardized script generation and dynamic adjustment mechanisms, enables the automated combination and flexible execution of fault scenarios, providing technical support for fault simulation and troubleshooting on the cloud platform.

[0171] In summary, in this embodiment, the fault orchestration module achieves automated orchestration and execution of fault scenarios through combined fault scenario modeling, execution logic generation, and script output. It should be noted that this module is not only applicable to single fault scenarios but also supports complex fault combinations and multi-environment adaptation, ensuring flexible simulation and efficient triggering of fault scenarios.

[0172] In this embodiment, the script execution module executes the fault scripts generated by the fault orchestration module to simulate fault scenarios in the cloud platform and perform real-time monitoring and data collection on the fault drill process. The script execution module decouples the fault scenarios from the execution environment through containerization technology, enabling automatic triggering and status monitoring of fault scenarios.

[0173] In one possible implementation, the script execution module receives a script file output by the fault orchestration module. This script file is in a standardized YAML format, suitable for execution in Kubernetes or Docker Compose environments. Specifically, the script execution module calls the corresponding fault injection tool or simulation command to simulate a fault in the target component based on the fault triggering logic described in the script.

[0174] Alternatively, the script execution module can dynamically launch fault-simulating containers within a container orchestration environment and execute fault injection commands within those containers. For example, based on a Kubernetes cluster, the script execution module can inject faults into the target component using the following commands:

[0175] kubectl apply-f fault-scenario.yaml

[0176] Among them, fault-scenario.yaml is a script file generated by the fault orchestration module.

[0177] Specifically, the script execution module supports the following fault injection types:

[0178] Resource exhaustion failure: For example, simulate a high load failure by setting CPU utilization or memory consumption ratio.

[0179] Network failures: For example, network latency or outages can be simulated by limiting network speed or causing network packet loss.

[0180] Service crash / failure: Simulate service unavailability scenarios by stopping the service or exiting abnormally.

[0181] In some embodiments, the execution of a resource exhaustion failure can be achieved through the container's resource limit configuration. For example, it can be configured to limit the container's memory resources to 512MB when simulating a failure, in order to simulate a memory exhaustion scenario.

[0182] As an alternative, the script execution module also provides real-time status monitoring during the fault injection process, specifically including:

[0183] Resource utilization monitoring: Real-time monitoring of the CPU, memory, network bandwidth and other resources of the target component.

[0184] Log data collection: Collect service log information during the fault triggering process, including the time point of the fault occurrence, error code, and service output information.

[0185] Fault propagation monitoring: By monitoring the state changes of related components, the propagation path of faults in the cloud platform is recorded.

[0186] For example, in a Kubernetes environment, the script execution module can use monitoring tools such as Prometheus to monitor resource status. Below is an example expression for retrieving monitoring metrics:

[0187] rate(container_cpu_usage_seconds_total{namespace="fault-test"}[5m])

[0188] The above expression represents the calculation of the container's CPU utilization in the namespace fault-test.

[0189] It should be noted that the script execution module also supports time synchronization and recording of data during the fault execution process to ensure an accurate correspondence between the fault occurrence and the system response. For example, the script execution module records the following key time points:

[0190] Fault trigger time T start ;

[0191] Fault duration T duration ;

[0192] Fault recovery time T recovery .

[0193] In one possible implementation, the data from the fault execution process can be stored and replayed through a unified log file, which can be used in the fault review and analysis module to support the replay of fault drills and cause localization.

[0194] Alternatively, the script execution module can also work in conjunction with the fault prediction module to dynamically adjust the fault triggering sequence and parameters. For example, when the script execution module detects that a component has reached a fault threshold, it can trigger the corresponding recovery operation in advance to ensure the integrity of the fault simulation and the stability of the system.

[0195] In some embodiments, the script execution module also supports adaptation to the domestic IT innovation environment, that is, executing fault scripts on domestic operating systems and hardware platforms to simulate fault scenarios. For example, for high-load faults under domestic operating systems (such as Kylin OS), CPU usage simulation can be achieved through command-line tools:

[0196] stress -- CPU 4 -- Timeout 60

[0197] The above command simulates a 4-core CPU under high load for 60 seconds.

[0198] Understandably, the script execution module automates and makes the fault simulation process observable through automatic triggering and real-time monitoring of fault scenarios, providing data support for the review and optimization of fault drills.

[0199] In summary, this embodiment utilizes containerization technology and standardized script execution to achieve automatic triggering and monitoring of fault scenarios. Simultaneously, this module provides resource monitoring, log collection, and data recording functions, supporting dynamic observation and precise time-based analysis of fault propagation paths. It should be noted that the data generated by this module can also be used for retrospective analysis and model optimization, thereby further improving the accuracy and effectiveness of fault simulation.

[0200] In this embodiment, the debriefing and optimization module is used to optimize the fault scenario model, fault propagation path and its parameters based on the data and results of the fault drill process, generate a debriefing report, and feed it back to the fault modeling module and the fault prediction module, thereby achieving closed-loop optimization.

[0201] In one possible implementation, the review and optimization module first receives fault simulation data provided by the script execution module, including information such as fault trigger time, fault propagation path, component state changes, resource usage, and fault repair process.

[0202] Specifically, the review and optimization module performs the following operations:

[0203] I. Collection and Analysis of Fault Execution Data

[0204] The debriefing and optimization module retrieves fault drill logs and monitoring data from the script execution module, and performs data cleaning, classification, and correlation analysis. The collected data includes:

[0205] Fault trigger time T start The point in time when the fault script begins execution;

[0206] Fault propagation path: the order in which a fault propagates between components;

[0207] Changes in resource usage: such as changes in CPU utilization and memory utilization curves;

[0208] Fault recovery time T recovery The point in time when the fault repair was completed;

[0209] Log data includes text information such as service error codes and exception logs.

[0210] As an alternative, the review and optimization module uses time series analysis technology to analyze the relationship between fault trigger time and fault propagation time, and generates a fault propagation delay curve.

[0211] In one exemplary implementation, the fault propagation delay T delay It can be represented as:

[0212] T delay =T propagate -T trigger

[0213] in:

[0214] T propagate This is the time when the fault begins to propagate in the next component;

[0215] T trigger This represents the time when the fault was triggered in the current component.

[0216] II. Generation of the Post-mortem Report

[0217] In some embodiments, the review and optimization module generates a review report based on fault execution data, which includes the following:

[0218] Fault triggering path: A detailed record of the fault propagation process and propagation time;

[0219] Cause of the failure: Analyze the root cause of the failure by combining log data;

[0220] Fault repair steps: Describe the fault repair measures taken in the exercise and their effects;

[0221] Resource impact analysis: Analyze the impact of the failure on the resource usage of each component, such as peak CPU utilization and memory usage;

[0222] Time analysis: including fault trigger time, propagation time, and recovery time.

[0223] For example, the generated debriefing report can be stored as structured data, such as JSON format.

[0224] III. Optimization of Fault Scenario Model

[0225] Specifically, the review and optimization module utilizes data feedback during the fault execution process to optimize the fault propagation path weights in the Petri network and the conditional probability distribution parameters in the Bayesian network.

[0226] Alternatively, the weight W in the fault propagation path can be dynamically adjusted using the following formula:

[0227]

[0228] in:

[0229] W new The updated path weights;

[0230] ΔT delay This represents the change in fault propagation delay;

[0231] T threshold This is the preset propagation time threshold.

[0232] In one possible implementation, the conditional probability distribution P(X) in the Bayesian network j |X i This can be corrected using the following methods:

[0233]

[0234] in:

[0235] count(X j ∩X i ) represents component X j In component X i The number of times the fault occurred after the initial failure;

[0236] count(X i ) represents component X i Total number of failures.

[0237] It should be noted that the optimized weights and conditional probability distributions will be fed back to the fault modeling module and the fault prediction module to improve the accuracy of subsequent fault scenario modeling and prediction.

[0238] IV. Closed-loop feedback and iterative optimization

[0239] As an option, the debriefing and optimization module can perform multiple rounds of iterative optimization on the fault scenario model and exercise results, forming a closed-loop feedback mechanism. For example, after the initial optimization, the debriefing and optimization module can generate a new debriefing report and continuously update the Petri net and Bayesian network parameters to improve the accuracy of fault propagation path prediction and the effectiveness of fault exercise.

[0240] Specifically, the review and optimization module stores the optimization results in a unified model database, which contains the following data:

[0241] The revised Petri net model;

[0242] Optimized fault propagation path weights;

[0243] Updated Bayesian network conditional probability distribution parameters.

[0244] Understandably, the review and optimization module achieves continuous optimization of the fault simulation process through data analysis, report generation, and dynamic correction of model parameters, providing more accurate technical support for subsequent fault simulation and troubleshooting.

[0245] In summary, in this embodiment, the debriefing and optimization module generates a debriefing report based on fault drill data, optimizes the fault propagation path and conditional probability distribution, and achieves closed-loop feedback and iterative optimization of the fault scenario model. It should be noted that the data processing and optimization mechanism of the debriefing and optimization module ensures continuous improvement of the fault scenario and the ongoing enhancement of the fault drill effect.

[0246] In summary, this invention's fault modeling module models the state of cloud platform components and fault propagation paths based on Petri nets; the fault prediction module performs dynamic reasoning through Bayesian networks to calculate the probability path and optimal triggering sequence of fault propagation; the fault orchestration module automatically generates fault orchestration scripts based on the fault model and prediction results; the script execution module executes the scripts, simulates fault scenarios, and monitors the status; and the debriefing and optimization module generates debriefing reports based on the data and results of the fault drill process, and optimizes the fault propagation paths and model parameters to achieve closed-loop feedback and continuous iterative optimization of fault scenarios. This invention improves the accuracy and efficiency of operational fault handling in cloud environments through automated modeling, fault simulation, and debriefing analysis, and is suitable for complex, multi-component fault drill requirements.

[0247] Please see the appendix Figure 3 This invention also provides a cloud-based method for operational fault training and automatic fault scenario orchestration, comprising the following steps:

[0248] S1. Based on Petri net modeling, cloud platform component status and fault propagation paths are modeled to generate fault scenario models;

[0249] In this step, the fault modeling module formally models the states of various components and fault propagation paths of the cloud platform. By abstracting each component in the cloud platform as a state node and defining the state transition paths for fault triggering and recovery, a fault scenario model is generated. This model describes the behavior and dependencies of each component in the fault scenario, providing foundational data for subsequent fault propagation analysis and execution.

[0250] S2. Calculate the probability of fault propagation paths based on Bayesian networks to determine the optimal fault triggering path;

[0251] In this step, the fault prediction module, based on the fault scenario model output by the fault modeling module, uses a Bayesian network to dynamically infer the fault propagation path, calculates the probability weight of each path, and determines the optimal fault triggering path. This optimal path will serve as the basis for subsequent fault scenario execution, ensuring high accuracy and coverage in the fault simulation.

[0252] S3. Transform the fault scenario model and the optimal fault triggering path into a containerized fault orchestration script;

[0253] In this step, the fault orchestration module automatically generates containerized fault orchestration scripts based on the fault scenario model and the optimal trigger path provided by the fault prediction module. The fault orchestration scripts are described using a standardized format (such as Kubernetes YAML or Docker Compose) and include fault type, triggering components, execution order, and parameter configurations. The generated scripts can be executed directly on the cloud platform to trigger fault simulations.

[0254] S4. Execute the fault orchestration script to trigger faults and monitor the status of each component of the cloud platform;

[0255] In this step, the script execution module receives the script file generated by the fault orchestration module and executes the script on the cloud platform. During execution, the script execution module triggers corresponding fault scenarios, such as resource exhaustion, network interruption, or service crash. Simultaneously, the script execution module monitors the status of each component on the cloud platform in real time, including resource utilization, service logs, and status changes, and records various data during the fault triggering and propagation process.

[0256] S5. Generate a debriefing report based on the data from the fault drill process, and provide feedback to optimize the fault scenario model and fault propagation probability path;

[0257] In this step, the debriefing and optimization module analyzes the fault drill data collected by the script execution module and generates a fault debriefing report. The report includes the fault propagation path, fault trigger time, changes in resource usage, and the fault repair process. Simultaneously, based on the actual data feedback from the fault drills, the debriefing and optimization module optimizes the fault propagation path weights in the fault scenario model and dynamically adjusts the fault propagation probability parameters, making the fault scenario model more closely resemble the actual fault behavior of the cloud platform, thus achieving closed-loop optimization of the system.

[0258] Understandably, this method realizes the modeling, execution, and review optimization process of fault scenarios through the above steps. The data flow and logical connection between each module are close, ensuring the accuracy of fault scenario simulation and the continuity of system optimization.

[0259] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system, characterized in that, The method comprises the following steps: a fault modeling module for modeling the component state and fault propagation path of the cloud platform to generate a fault scenario model; a fault prediction module for dynamically reasoning the fault propagation path based on a Bayesian network, calculating the probability path of fault propagation, and optimizing the fault trigger sequence; a fault arrangement module for automatically generating an executable fault arrangement script according to the fault scenario model and the optimized fault trigger sequence; a script execution module for executing the fault arrangement script, simulating the fault scenario in the cloud platform, and triggering and monitoring the fault state; a review optimization module for generating a fault review report based on the data and results of the fault exercise process, and feeding back the optimized fault scenario model and fault propagation probability path; The review optimization module optimizes the fault propagation path weight in the Petri net according to the fault exercise results, and dynamically adjusts the conditional probability distribution parameters in the Bayesian network; The dynamic adjustment of the conditional probability distribution parameters in the Bayesian network comprises: receiving the monitoring data provided by the script execution module during the fault exercise process; updating the weight of the fault propagation path in the Petri net based on the actual data of the fault propagation path, and dynamically adjusting the weight value according to the fault trigger frequency and the probability of path occurrence; recomputing the conditional probability distribution of each node in the Bayesian network according to the monitoring data and the actual results of the fault propagation path; feeding back the dynamically adjusted conditional probability distribution to the fault prediction module for optimizing the calculation accuracy of the fault propagation probability; updating the optimized fault propagation path and the conditional probability result to the fault modeling module to further improve the fault scenario model and ensure that the subsequent fault exercise is closer to the actual fault state of the cloud platform. 2.The cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system according to claim 1, characterized in that, The fault modeling module models the fault propagation path through a Petri net, and the Petri net comprises: a set of libraries for representing the state of cloud platform components; a transition set for representing fault triggering and recovery operations; a weight function for describing the state transition weight in the fault propagation path; a set of initial tags for describing the initial state of fault propagation. 3.The cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system according to claim 1, characterized in that, The fault prediction module dynamically reasons the fault propagation probability through a Bayesian network, and the Bayesian network comprises: a node set for representing the state of each component of the cloud platform; a directed edge set for describing the dependency relationship of fault propagation between components; a conditional probability distribution for representing the probability of a node state under its parent node state. 4.The cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system according to claim 1, characterized in that, The fault arrangement module converts the fault scenario model and the fault propagation probability path into a containerized script, which supports execution in a Kubernetes cluster or a Docker Compose environment. 5.The cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system according to claim 1, characterized in that, The script execution module monitors the resource utilization, network status and service log of the cloud platform components during the fault exercise process, and records the time points of fault propagation and recovery. 6.The cloud environment-based operation and maintenance fault simulation and fault scenario automatic arrangement system according to claim 1, characterized in that, The review optimization module generates a fault review report based on the monitoring data during the fault propagation process, and the review report comprises: fault trigger time and fault recovery time; fault propagation path and propagation probability; fault repair operation and execution result.

7. The cloud environment-based operation and maintenance fault simulation and automatic fault scenario arrangement method is applied to the system as claimed in any one of claims 1-6, characterized in that, The method comprises the following steps: Based on Petri net modeling cloud platform component state and fault propagation path, generate fault scenario model; Based on Bayesian network, calculate the probability of fault propagation path, and determine the optimal fault trigger path; Convert the fault scenario model and the optimal fault trigger path into containerized fault orchestration scripts; Execute the fault orchestration script, trigger the fault and monitor the status of each component of the cloud platform; Based on the data of the fault exercise process, generate a review report and feedback to optimize the fault scenario model and the fault propagation probability path.

Images