Operation and maintenance fault training and fault scene automatic arrangement system based on cloud environment

Abstract

The invention relates to the field of cloud computing operation and maintenance management, and discloses an operation and maintenance fault training and fault scene automatic arrangement system based on a cloud environment, and the system comprises a fault modeling module, a fault prediction module, a fault arrangement module, a script execution module, and a redisk optimization module. The fault modeling module performs modeling on the component state and the fault propagation path of the cloud platform to generate a fault scene model; the fault prediction module calculates the probability of a fault propagation path and dynamically determines an optimal fault triggering path; the fault arrangement module automatically generates a containerized fault arrangement script; the script execution module executes a fault arrangement script and triggers a fault; and the recovery optimization module generates a recovery report based on the data of the fault drilling process. According to the method, through automatic fault modeling, dynamic prediction, script arrangement and redisk optimization, the accuracy and efficiency of fault drilling are improved, and the method is suitable for complex and multi-component cloud environment operation and maintenance fault simulation and training requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing operation and maintenance management, and in particular to an operation and maintenance fault training and fault scenario automatic orchestration system based on a cloud environment. Background Art

[0002] With the rapid development of cloud computing and microservice architecture, cloud platforms have become the infrastructure support for modern information technology. However, the large-scale deployment of cloud platforms and the complex component interactions have made the system structure more complex, the coupling and dependency between components have continued to increase, and the types and forms of failures have gradually diversified. Failures in cloud platforms not only involve abnormal status of a single component, but may also spread to other components through multiple channels such as network, resource or service calls, forming complex fault links and cascading effects. This complexity poses great challenges to operation and maintenance personnel in the process of troubleshooting and repairing.

[0003] Most existing operation and maintenance fault management systems focus on fault detection and recovery, for example, by using monitoring tools to identify fault alarms and perform repair operations, but these technologies only stay at the response level after the fault occurs, lacking the preventive function of fault simulation and drills. In addition, traditional fault drills mainly rely on static scenario design and manual fault simulation, which have problems such as insufficient flexibility, limited scenario coverage, and low execution efficiency, and cannot effectively respond to the actual operation and maintenance needs of complex and multi-fault combinations in cloud platforms.

[0004] Furthermore, it is difficult for existing technologies to achieve automatic modeling, dynamic prediction, orchestration execution and replay optimization of faults in an integrated platform. Especially in the process of modeling fault scenarios, existing methods usually rely on static rules and cannot flexibly respond to dynamic changes in the status of multiple components and fault propagation paths. At the same time, the analysis of fault propagation paths lacks intelligent means, and the probability of fault occurrence and the scope of impact are difficult to accurately assess, resulting in deviations between fault simulation results and actual conditions. In addition, the lack of a mechanism for automated script generation and execution requires operation and maintenance personnel to invest a lot of manpower costs in fault simulation and drills, and the accuracy and reusability of fault execution are poor.

[0005] Therefore, how to achieve dynamic simulation and continuous optimization of complex fault scenarios in a cloud platform environment through precise fault modeling technology, efficient fault propagation analysis methods, automated fault orchestration and execution mechanisms, and systematic review and optimization functions has become a technical problem that needs to be urgently solved in the current operation and maintenance management field. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides an operation and maintenance fault training and fault scenario automatic orchestration system based on a cloud environment, which solves the problems in the prior art of inaccurate cloud platform fault scenario modeling, unintelligent fault propagation path analysis, low fault simulation and execution efficiency, and lack of replay optimization mechanism.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cloud-based operation and maintenance fault training and fault scenario automatic orchestration system, comprising:

[0008] The fault modeling module is used to model the component status and fault propagation path of the cloud platform and generate a fault scenario model;

[0009] The fault prediction module is used to dynamically infer the fault propagation path based on the Bayesian network, calculate the probability path of fault propagation, and optimize the fault triggering sequence;

[0010] Fault orchestration module, which is used to automatically generate executable fault orchestration scripts based on fault scenario models and optimized fault triggering sequences;

[0011] The script execution module is used to execute the fault orchestration script, simulate the fault scenario in the cloud platform, and perform fault triggering and status monitoring;

[0012] The replay optimization module is used to generate a fault replay report based on the data and results of the fault drill process, and to provide feedback to optimize the fault scenario model and fault propagation probability path.

[0013] Preferably, the fault modeling module models the fault propagation path through a Petri net, and the Petri net includes:

[0014] A collection of places, used to represent the status of cloud platform components;

[0015] Transition set, used to represent fault triggering and recovery operations;

[0016] Weight function, used to describe the state transition weight in the fault propagation path;

[0017] An initial tag set used to describe the initial state of fault propagation.

[0018] Preferably, the fault prediction module performs dynamic reasoning of the fault propagation probability through a Bayesian network, and the Bayesian network includes:

[0019] A collection of nodes, used to represent the status of each component of the cloud platform;

[0020] A set of directed edges that describe the dependencies between components for fault propagation;

[0021] Conditional probability distribution, used to represent the probability of a node state being in the state of its parent node.

[0022] Preferably, the fault orchestration module converts the fault scenario model and the fault propagation probability path into a containerized script, and the containerized script supports execution in a Kubernetes cluster or Docker Compose environment.

[0023] Preferably, the containerized script includes fault triggering instructions, and the fault triggering instructions are used to simulate fault scenarios such as high load, network interruption, and service crash.

[0024] Preferably, during the fault drill, the script execution module monitors the resource usage, network status and service logs of the cloud platform components, and records the time points of fault propagation and recovery.

[0025] Preferably, the replay optimization module generates a fault replay report based on the monitoring data during the fault propagation process, and the replay report includes:

[0026] Fault trigger time and fault recovery time;

[0027] Fault propagation path and propagation probability;

[0028] Troubleshooting operations and execution results.

[0029] Preferably, the replay optimization module optimizes the fault propagation path weights in the Petri network according to the fault drill results, and dynamically adjusts the conditional probability distribution parameters in the Bayesian network.

[0030] Preferably, the dynamically adjusting the conditional probability distribution parameters in the Bayesian network includes:

[0031] Receive monitoring data provided by the script execution module during the fault drill;

[0032] Based on the actual data of the fault propagation path, the weight of the fault propagation path in the Petri network is updated, and the weight value is dynamically adjusted according to the fault triggering frequency and the probability of path occurrence;

[0033] Recalculate the conditional probability distribution of each node in the Bayesian network based on the actual results of the monitoring data and the fault propagation path;

[0034] Feedback the dynamically adjusted conditional probability distribution to the fault prediction module to optimize the calculation accuracy of the fault propagation probability;

[0035] The optimized fault propagation path and conditional probability results are synchronously updated to the fault modeling module to further improve the fault scenario model and ensure that subsequent fault drills are closer to the actual fault status of the cloud platform.

[0036] The present invention also provides a method for automatically arranging operation and maintenance fault training and fault scenarios based on a cloud environment, comprising the following steps:

[0037] Model the cloud platform component status and fault propagation path based on Petri network to generate fault scenario model;

[0038] Calculate the probability of fault propagation paths based on Bayesian networks and determine the optimal fault triggering path;

[0039] Convert the fault scenario model and the optimal fault trigger path into a containerized fault orchestration script;

[0040] Execute fault orchestration scripts, trigger faults, and monitor the status of each component of the cloud platform;

[0041] Generate a replay report based on the data from the fault drill process, and provide feedback to optimize the fault scenario model and fault propagation probability path.

[0042] The present invention provides an operation and maintenance fault training and fault scenario automatic arrangement system based on a cloud environment. It has the following beneficial effects:

[0043] 1. The present invention introduces Petri nets to formally model the component states and fault propagation paths of the cloud platform, which can accurately describe the state changes and interdependencies of each component. Compared with the traditional manual fault modeling method, Petri nets have stronger expression capabilities and can support multiple fault scenario combinations, including complex fault scenarios such as concurrency, sequence, and nesting, providing an accurate fault scenario model for subsequent fault propagation analysis and execution.

[0044] 2. The present invention uses Bayesian networks to perform probabilistic reasoning on fault propagation paths, dynamically evaluates the propagation probability of each component fault by calculating the conditional probability distribution between nodes, and selects the optimal fault triggering path. Unlike static path analysis methods, Bayesian networks can be updated and optimized in real time based on actual drill data, significantly improving the accuracy and applicability of fault propagation path prediction.

[0045] 3. The present invention converts the fault scenario model and the optimal trigger path into a standardized containerized script through the fault orchestration module, and automatically performs fault simulation in the cloud platform environment. This process does not require manual intervention and can automatically generate reusable fault scripts based on fault type, trigger sequence and parameter configuration. In addition, it supports multiple execution environments, ensuring the flexibility and convenience of fault scenario execution.

[0046] 4. The script execution module monitors the status of cloud platform components in real time during the fault triggering process, and collects fault drill data including resource usage, service logs, status changes, etc. These data provide a reliable basis for fault propagation analysis, resource impact assessment, and replay optimization, ensuring that the fault simulation process is observable and traceable, which helps to quickly locate and analyze problems.

[0047] 5. The present invention analyzes the data of the fault drill process through the replay optimization module to generate a detailed replay report, including the fault propagation path, fault cause, resource impact and repair process, etc. At the same time, the system optimizes the fault propagation path weight and Bayesian network conditional probability distribution based on the drill data feedback, thereby continuously iterating and optimizing the fault scenario model, forming a closed-loop optimization mechanism, and improving the system's fault simulation accuracy and prediction effect.

[0048] 6. The present invention realizes the automation of the whole process of fault drills through functions such as automatic modeling, dynamic prediction, automatic arrangement, real-time execution and closed-loop optimization, reduces manual operation and script maintenance work, and improves the efficiency of operation and maintenance fault drills. At the same time, the system supports the rapid construction of complex, multi-fault combination scenarios, which helps operation and maintenance personnel to efficiently carry out fault investigation and emergency drills.

[0049] 7. The present invention supports a variety of fault scenario types and drill environments, and can adapt to different cloud platform architectures, including domestic operating systems and hardware platforms in the trusted computing environment. At the same time, fault scripts and models are highly reusable, supporting the flexible expansion of new fault types and scenarios according to actual needs to meet diversified operation and maintenance needs.

[0050] 8. The replay reports and drill data generated by the system can be used to build and maintain the fault knowledge base, helping operation and maintenance personnel accumulate fault handling experience. At the same time, through replay analysis and model optimization, the system can continuously improve the fault scenario design and execution strategy, providing strong technical support and decision-making basis for subsequent fault detection and repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0052] Figure 2 This is a schematic diagram of an example of a fault arrangement script of the present invention;

[0053] Figure 3 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Please refer to the attached Figure 1 -Attached Figure 2 The present invention provides a cloud-based operation and maintenance fault training and fault scenario automatic arrangement system, which is mainly used to simulate fault scenarios, optimize fault propagation paths, implement fault triggering drills, and perform replay analysis and model optimization on the drill results. The following is a detailed description of each module of the system of the present invention in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, the cloud-based operation and maintenance fault training and fault scenario automatic orchestration system may include a fault modeling module, a fault prediction module, a fault orchestration module, a script execution module, and a replay optimization module.

[0057] In this embodiment, the fault modeling module is used to model the component status and fault propagation path of the cloud platform based on the Petri network to form a fault scenario model that can be used for fault scenario simulation. This module describes the status, fault triggering and propagation path of each component in the cloud platform through formal methods, providing a basic model for subsequent fault prediction, orchestration and execution.

[0058] In one possible implementation, a Petri net is defined by the following five elements:

[0059] The place set P is used to represent the different states of each component in the cloud platform;

[0060] The transition set T is used to describe the transition operations between states, including fault triggering and fault recovery;

[0061] The directed arc set F is used to connect places and transitions, indicating the flow relationship between states;

[0062] The weight function W is used to describe the weight on the directed arc and represents the weight of the fault propagation path;

[0063] The initial mark M0 is used to describe the initial state of fault propagation.

[0064] Specifically, in the cloud platform, the state of each component can be abstracted into different locations, including but not limited to the following states:

[0065] Normal state normal Indicates that the component is in normal working condition;

[0066] Fault status p fault Indicates a component failure;

[0067] Restore state p recovery Indicates that the component has recovered from a failure.

[0068] In some embodiments, the fault triggering and recovery process can be described by a set of transitions. For example, the process of a component switching from a normal state to a fault state can be expressed as:

[0069]

[0070] Wherein, W represents the weight of the triggered fault, which is used to describe the probability or impact of the fault triggering.

[0071] As an option, the process of a component recovering from a faulty state to a normal working state can be expressed as:

[0072]

[0073] Among them, W ′ The weight of the recovery operation, which is used to measure the cost or time of failure recovery.

[0074] In one possible implementation, the fault propagation path is modeled through directed arcs of the Petri net. Directed arcs connect places with transitions to form a fault propagation path. For example, a fault in component A can propagate through the network to affect component B, and the path can be expressed as:

[0075]

[0076] Among them, W A→B Represents the weight of fault propagation, which is usually obtained from historical fault data statistics.

[0077] It should be noted that the fault modeling module further supports the reachability analysis of the fault propagation path, which is used to determine the set of all possible fault states in the system starting from the initial state M0. The reachability analysis is calculated through the state transfer matrix AAA, which includes the forward transition matrix A + and the inverse transition matrix A - , the specific calculation process is as follows:

[0078] A=A + -A -

[0079] in:

[0080] A + Indicates positive transition, i.e., fault triggering process;

[0081] A- Indicates the reverse transition, that is, the fault recovery process.

[0082] In an exemplary implementation, the fault modeling module generates a fault propagation path graph by traversing all possible state paths. In the path graph, each node represents a fault state of a component, and each edge represents a path and probability of fault propagation.

[0083] In some embodiments, in order to improve the flexibility and reusability of modeling, the fault modeling module supports combined modeling of complex fault scenarios. For example, the system can combine multiple independent fault scenarios into an overall scenario according to actual needs. The combination methods include:

[0084] Concurrent scenarios, where multiple components fail at the same time;

[0085] Nested scenarios, where a certain fault triggers the propagation of other faults;

[0086] Sequential scenario, where faults are triggered step by step in a predetermined order.

[0087] As an option, the fault modeling module can also combine historical fault data with monitoring indicators to calculate the fault trigger weight W and recovery weight W. ′ Dynamic adjustments can be made. For example, the system can update the weight value based on the frequency and impact of the failure, thereby optimizing the failure scenario model.

[0088] It should be noted that the output of the fault modeling module is a fault scenario model, including:

[0089] Petri net model describing the status of cloud platform components;

[0090] Fault propagation paths and their corresponding weights.

[0091] In some other embodiments, the fault scenario model of the fault modeling module may also be linked with real-time monitoring data to provide a more accurate initial modeling result for the fault prediction module.

[0092] It can be understood that the modeling accuracy of the fault modeling module has an important impact on the fault prediction, scheduling and drill effects of the entire system. Therefore, in the present invention, Petri nets are used for formal modeling, which has strong expression and analysis capabilities and can meet the needs of complex fault scenario modeling.

[0093] For example, suppose there are three service components A, B, and C in the cloud platform, and there is a dependency relationship between them. The fault propagation path modeling results are as follows:

[0094] Initial state: A normal ,B normal ,C normal

[0095] Fault trigger:

[0096] Failure recovery:

[0097] At this point, the Petri net model and fault propagation path output by the fault modeling module can be used as input for subsequent fault prediction module and orchestration module to analyze and execute.

[0098] At this point, the Petri net model and fault propagation path output by the fault modeling module can be used as input for subsequent fault prediction module and orchestration module to analyze and execute.

[0099] In this embodiment, the fault prediction module is used to dynamically infer the fault propagation path based on the Bayesian network, calculate the probability path of the fault occurrence, and determine the optimal fault triggering sequence. The main function of this module is to dynamically evaluate the state probability during the fault propagation process and provide accurate path and weight information for the fault scheduling module.

[0100] In a possible implementation, the fault prediction module models fault propagation as a Bayesian network, where the Bayesian network consists of nodes, edges, and conditional probability distributions, as described below:

[0101] Node: represents the status of each component in the cloud platform, including normal status, fault status, and recovery status. For example, node X i Represents the state of the i-th component.

[0102] Directed edge: represents the fault propagation relationship between nodes, for example, the fault propagation path from component X i To component X j .

[0103] Conditional probability distribution: represents the probability of a node's state occurring in its parent node's state, that is, P(X j |Pa(X j )), where Pa(X j ) represents node X j The parent node collection of .

[0104] Specifically, the implementation of the fault prediction module includes the following steps:

[0105] In some embodiments, the probability calculation of fault propagation is implemented based on Bayes' theorem, which is described as:

[0106]

[0107] in:

[0108] P(Xj |X i ): indicates a faulty slave node X i Propagate to node X j The conditional probability of

[0109] P(X i ): represents node X i The prior probability of

[0110] P(X j ): indicates node X j The prior probability of .

[0111] In one possible implementation, the total probability of fault propagation can be determined by summing the impact of all parent nodes, as follows:

[0112]

[0113] Among them, P fault (X j ) is the parent node X i The probability of failure, P(X j |X i ) is the propagation probability from parent node to child node.

[0114] As an option, the fault prediction module can learn and train node probabilities and conditional probabilities through historical fault data. For example, the system performs statistical analysis on the data in the historical fault log, calculates the fault frequency of each component and its correlation with other components, and thus determines the initial conditional probability distribution in the Bayesian network.

[0115] It should be noted that in the fault prediction process, the dynamic reasoning of the fault propagation path is achieved through the real-time update of the node status. i When a fault occurs, the fault prediction module will infer the probability of failure of other nodes based on the Bayesian network and determine the optimal path for fault propagation.

[0116] In an exemplary implementation, assuming that the cloud platform includes three components A, B, and C, the fault propagation path is as follows:

[0117] P(A fault ) = 0.2 indicates the prior probability of component A failing;

[0118] P(B fault |A fault ) = 0.8 represents the probability that component B fails under the condition that component A fails;

[0119] P(C fault |B fault)=0.6 represents the probability of component C failing under the condition that component B fails.

[0120] At this time, the total probability of fault propagation is:

[0121] P(C fault )=P(A fault )·P(B fault |A fault )·P(C fault |B fault )=0.2·0.8·0.6

[0122] =0.096

[0123] Specifically, based on the above results, the fault prediction module will select the path with the highest probability of fault propagation as the optimal path for the subsequent fault orchestration module to simulate and trigger the fault.

[0124] In some embodiments, in order to improve the real-time and accuracy of fault prediction, the fault prediction module supports dynamic updating of conditional probability distribution. For example, after the script execution module simulates the fault scenario, the system will correct the conditional probability in the Bayesian network based on the actual monitoring data feedback results to better adapt to the actual fault propagation path.

[0125] As an option, the fault prediction module can also support dynamic reasoning for multiple scenarios, such as prediction of concurrent and cascading failures. In a concurrent failure scenario, multiple components may fail at the same time, and the joint probability of fault propagation can be calculated through the joint distribution of the Bayesian network.

[0126] It should be noted that the output results of the fault prediction module include:

[0127] The probability of failure of each component;

[0128] Fault propagation paths and their probability weights;

[0129] Optimal fault triggering sequence.

[0130] These output results will be provided as input to the fault orchestration module to automatically generate the optimal fault triggering script.

[0131] It can be understood that through the dynamic reasoning and conditional probability update of the Bayesian network, the fault prediction module can effectively evaluate the path and probability of fault propagation and provide accurate data support for subsequent modules.

[0132] In some other embodiments, the fault prediction module can also be linked with real-time monitoring data to achieve early prediction and early warning of faults. For example, when monitoring a sharp increase in the resource usage of a component, the system can predict the probability of fault occurrence in advance and generate the corresponding fault propagation path in advance for verification and drills in subsequent stages.

[0133] In summary, in this embodiment, the fault prediction module effectively realizes the probability calculation of the fault propagation path and the optimal path selection through dynamic reasoning based on the Bayesian network, providing an accurate data basis for subsequent fault scheduling and execution. It should be noted that the fault prediction module in the present invention is not only applicable to single fault scenarios, but also supports flexible prediction of multiple fault combinations and dynamic scenarios.

[0134] In this embodiment, the fault orchestration module is used to automatically generate an executable fault orchestration script based on the fault scenario model generated by the fault modeling module and the fault propagation probability path provided by the fault prediction module. This module is mainly responsible for the automatic combination of fault scenarios, logical orchestration, and dynamic generation of scripts to achieve the execution and reproduction of fault scenarios in the cloud platform environment.

[0135] In a possible implementation, the fault scheduling module first receives the Petri net model output by the fault modeling module, which contains the status of each component and the fault propagation relationship between them. The fault scheduling module also receives the optimal fault propagation path and path weight calculated by the fault prediction module, which are used to determine the fault triggering order and execution priority.

[0136] Specifically, the execution logic of the fault orchestration module includes the following aspects:

[0137] 1. Fault scenario combination

[0138] The fault orchestration module dynamically combines the fault scenarios of each component based on the optimal fault propagation path. As an option, the combination of these fault scenarios can include the following forms:

[0139] Concurrent scenario: multiple component failures are triggered at the same time. For example, component A and component B can be injected with failures at the same time.

[0140] Sequential scenario: Faults are triggered step by step in a certain order. For example, after component A fails, the failure of component B is triggered.

[0141] Nested scenario: The failure of a component triggers cascading failures of other components. For example, the failure of component A affects the network status, which leads to the failure of components B and C.

[0142] For example, if the optimal path output by the fault prediction module is A fault →B fault →Cfault ,The fault orchestration module will combine the paths in order and assign a trigger priority to each fault scenario.

[0143] 2. Fault Execution Logic Generation

[0144] In a possible implementation, the fault orchestration module converts the fault scenario combination into an executable script, and the script supports execution in a containerized environment (such as Kubernetes or Docker Compose).

[0145] It should be noted that the fault orchestration module uses a standardized description language to express the logical definition of the fault scenario, such as YAML or JSON format, to facilitate subsequent parsing and execution by the execution engine.

[0146] Specifically, the script generation process includes:

[0147] Component fault definition: specifies the component that needs to trigger the fault and the fault type, such as high CPU load, network interruption, service crash, etc.

[0148] Fault parameter configuration: Set execution parameters for fault scenarios, including fault injection time, duration, and resource usage.

[0149] Execution order configuration: Set the order of fault triggering according to the fault propagation path and weight.

[0150] An example fault orchestration script is Figure 2 As shown, in this script:

[0151] ComponentA triggers a network outage failure and limits memory resources.

[0152] After componentA fails, componentB triggers a high CPU load failure, which lasts for 60 seconds.

[0153] After componentB fails, componentC simulates a service crash.

[0154] 3. Fault triggering and dynamic parameter adjustment

[0155] In some embodiments, to improve the flexibility of fault scenario execution, the fault orchestration module supports dynamic adjustment of parameters in the fault script. For example, parameters such as fault duration and resource occupancy threshold can be dynamically optimized based on real-time monitoring data.

[0156] As an option, the fault orchestration module can dynamically update the fault triggering sequence and script content in combination with the reasoning results of the fault prediction module. For example, when real-time monitoring finds that the resource usage of a component is close to the fault threshold, the fault orchestration module can trigger the fault scenario of the component in advance to verify the system's ability to respond to faults.

[0157] 4. Fault Script and Environment Adaptation

[0158] It should be noted that the scripts generated by the fault orchestration module support multiple operating environments, including but not limited to:

[0159] Kubernetes cluster: Automatically orchestrate and execute fault scenarios through YAML configuration files.

[0160] Docker Compose: Implements fault simulation in a multi-container environment through container orchestration definition files.

[0161] Adaptation to the trusted computing environment: Compatibility adjustments for fault scripts are made for domestic operating systems and hardware platforms.

[0162] As an option, the fault orchestration module can also automatically verify and validate the scripts to ensure that the execution logic of the fault scenario matches the actual deployment environment.

[0163] 5. Script output and storage

[0164] The script files generated by the fault orchestration module can be stored in a unified script management library to facilitate subsequent drill reuse and fault scenario backtracking.

[0165] In an exemplary implementation, the script is stored with the following metadata:

[0166] Fault scenario ID

[0167] Fault trigger path

[0168] Fault parameter configuration

[0169] Build time and version number

[0170] It is understandable that the fault orchestration module realizes the automated combination and flexible execution of fault scenarios through standardized script generation and dynamic adjustment mechanisms, providing technical support for fault simulation and troubleshooting of the cloud platform.

[0171] In summary, in this embodiment, the fault orchestration module realizes the automatic orchestration and execution of fault scenarios through combined modeling of fault scenarios, execution logic generation and script output. It should be noted that this module is not only applicable to single fault scenarios, but also supports complex fault combinations and multi-environment adaptation, ensuring flexible simulation and efficient triggering of fault scenarios.

[0172] In this embodiment, the script execution module is used to execute the fault script generated by the fault orchestration module to simulate the fault scenario in the cloud platform and perform real-time monitoring and data collection on the fault drill process. The script execution module decouples the fault scenario from the execution environment through containerization technology to achieve automatic triggering and status monitoring of the fault scenario.

[0173] In a possible implementation, the script execution module receives the script file output by the fault orchestration module, which is in a standardized YAML format and is suitable for execution in a Kubernetes or Docker Compose environment. Specifically, the script execution module calls the corresponding fault injection tool or simulation command according to the fault trigger logic described in the script to simulate the fault of the target component.

[0174] As an option, the script execution module can dynamically start a fault simulation container in a container orchestration environment and execute the fault injection command in the container. For example, the script execution module can inject the fault into the target component based on the Kubernetes cluster through the following command:

[0175] kubectl apply-f fault-scenario.yaml

[0176] Among them, fault-scenario.yaml is the script file generated by the fault orchestration module.

[0177] Specifically, the script execution module supports the following fault injection types:

[0178] Resource exhaustion failure: For example, by setting the CPU usage or memory consumption ratio, a high load failure can be simulated.

[0179] Network failure: For example, network speed limiting or packet loss can be used to simulate network delays or outages.

[0180] Service crash failure: Simulate service unavailability scenarios by stopping services or abnormally exiting commands.

[0181] In some embodiments, the execution of the resource exhaustion failure can be implemented by configuring the resource limit of the container. For example, the configuration can limit the container memory resources to 512MB when simulating the failure to simulate the memory resource exhaustion scenario.

[0182] As an option, the script execution module also provides real-time status monitoring during the fault injection process, including:

[0183] Resource usage monitoring: Real-time monitoring of the target component's CPU, memory, network bandwidth and other resource usage.

[0184] Log data collection: Collect service log information during the fault triggering process, including the time when the fault occurred, error code, and service output information.

[0185] Fault propagation monitoring: By monitoring the status changes of related components, the propagation path of faults in the cloud platform is recorded.

[0186] For example, in a Kubernetes environment, the script execution module can use monitoring tools such as Prometheus to monitor resource status. The following is an example of an expression for obtaining monitoring indicators:

[0187] rate(container_cpu_usage_seconds_total{namespace="fault-test"}[5m])

[0188] The above expression calculates the CPU usage of the container in the fault-test namespace.

[0189] It should be noted that the script execution module also supports time synchronization and recording of data during the fault execution process to ensure that the time points between the fault occurrence and the system response accurately correspond. For example, the script execution module records the following key time points:

[0190] Fault trigger time T start ;

[0191] Fault duration T duration ;

[0192] Fault recovery time T recovery .

[0193] In one possible implementation, the data of the fault execution process can be stored and replayed through a unified log file, which can be used in the fault review and analysis module to support the playback and cause location of fault drills.

[0194] As an option, the script execution module can also work with the fault prediction module to dynamically adjust the fault triggering sequence and parameters. For example, when the script execution module detects that a component has reached the fault threshold, it can trigger the corresponding recovery operation in advance to ensure the integrity of the fault simulation and system stability.

[0195] In some embodiments, the script execution module also supports the adaptation of the trusted innovation environment, that is, executing fault scripts on domestic operating systems and hardware platforms to simulate fault scenarios. For example, for high-load faults under domestic operating systems (such as Kylin OS), CPU occupancy simulation can be achieved through command line tools:

[0196] stress --cpu 4 --timeout 60

[0197] The above command simulates a high load on a 4-core CPU for 60 seconds.

[0198] It can be understood that the script execution module realizes the automation and observability of the fault simulation process through automatic triggering and real-time monitoring of fault scenarios, providing data support for the review and optimization of fault drills.

[0199] In summary, in this embodiment, the script execution module realizes the automatic triggering and monitoring of fault scenarios through containerization technology and standardized script execution. At the same time, the module provides resource monitoring, log collection and data recording functions, and supports dynamic observation and time-accurate analysis of fault propagation paths. It should be noted that the data generated by this module can also be used for replay analysis and model optimization, thereby further improving the accuracy and effectiveness of fault simulation.

[0200] In this embodiment, the replay optimization module is used to optimize the fault scenario model, fault propagation path and its parameters based on the data and results of the fault drill process, generate a replay report, and feed it back to the fault modeling module and the fault prediction module, thereby achieving closed-loop optimization.

[0201] In one possible implementation, the replay optimization module first receives the fault drill data provided by the script execution module, including information such as fault trigger time, fault propagation path, component status changes, resource usage, and fault repair process.

[0202] Specifically, the replay optimization module performs the following operations:

[0203] 1. Collection and Analysis of Fault Execution Data

[0204] The replay optimization module obtains the fault drill log and monitoring data from the script execution module, and cleans, classifies and analyzes the data. The collected data includes:

[0205] Fault trigger time T start : The time point when the fault script starts to execute;

[0206] Fault propagation path: the order in which the fault propagates between components;

[0207] Resource usage changes: such as CPU usage and memory usage change curves;

[0208] Fault recovery time T recovery : The time point when the fault repair is completed;

[0209] Log data: including service error codes, exception logs and other text information.

[0210] As an option, the replay optimization module uses time series analysis technology to analyze the relationship between fault trigger time and fault propagation time and generate a fault propagation delay curve.

[0211] In one exemplary implementation, the fault propagation delay T delay It can be expressed as:

[0212] T delay =T propagate -T trigger

[0213] in:

[0214] T propagate is the time when the fault starts to propagate to the next component;

[0215] T trigger The time when the fault is triggered in the current component.

[0216] 2. Generation of replay report

[0217] In some embodiments, the replay optimization module generates a replay report based on the fault execution data, and the report includes the following contents:

[0218] Fault trigger path: record the fault propagation process and propagation time in detail;

[0219] Cause of the fault: Analyze the root cause of the fault in combination with log data;

[0220] Fault recovery steps: describe the fault recovery measures taken during the drill and their effectiveness;

[0221] Resource impact analysis: Analyze the impact of the fault on the resource usage of each component, such as CPU peak usage, memory usage, etc.

[0222] Time analysis: including fault trigger time, propagation time and recovery time, etc.

[0223] Exemplarily, the generated review report may be stored as structured data, such as in JSON format.

[0224] 3. Optimization of Fault Scenario Model

[0225] Specifically, the replay optimization module uses data feedback during the fault execution process to optimize the fault propagation path weights in the Petri network and the conditional probability distribution parameters in the Bayesian network.

[0226] As an option, the weight W in the fault propagation path can be dynamically adjusted by the following formula:

[0227]

[0228] in:

[0229] W new is the updated path weight;

[0230] ΔT delay is the change in fault propagation delay;

[0231] T threshold is the preset propagation time threshold.

[0232] In one possible implementation, the conditional probability distribution P(X j |X i ) can be corrected by:

[0233]

[0234] in:

[0235] count(X j ∩X i ) represents component X j In component X i the number of failures that occurred after the failure;

[0236] count(X i ) represents component X i The total number of failures.

[0237] It should be noted that the optimized weights and conditional probability distribution will be fed back to the fault modeling module and the fault prediction module to improve the modeling and prediction accuracy of subsequent fault scenarios.

[0238] 4. Closed-loop feedback and iterative optimization

[0239] As an option, the replay optimization module can perform multiple rounds of iterative optimization on the fault scenario model and the drill results to form a closed-loop feedback mechanism. For example, after the initial optimization, the replay optimization module can generate a new replay report again and continuously update the Petri net and Bayesian network parameters to improve the accuracy of fault propagation path prediction and the effect of fault drills.

[0240] Specifically, the replay optimization module stores the optimization results in a unified model database. The data in the database includes:

[0241] Modified Petri net model;

[0242] Optimized fault propagation path weights;

[0243] Updated Bayesian network conditional probability distribution parameters.

[0244] It is understandable that the review and optimization module achieves continuous optimization of the fault drill process through data analysis, report generation and dynamic correction of model parameters, providing more accurate technical support for subsequent fault simulation and troubleshooting.

[0245] In summary, in this embodiment, the replay optimization module generates a replay report based on the fault drill data, optimizes the fault propagation path and conditional probability distribution, and realizes closed-loop feedback and iterative optimization of the fault scenario model. It should be noted that the data processing and optimization mechanism of the replay optimization module ensures the continuous improvement of the fault scenario and the continuous improvement of the fault drill effect.

[0246] In general, the fault modeling module of the present invention models the cloud platform component status and fault propagation path based on the Petri network; the fault prediction module performs dynamic reasoning through the Bayesian network to calculate the probability path of fault propagation and the optimal triggering order; the fault scheduling module automatically generates a fault scheduling script according to the fault model and the prediction results; the script execution module executes the script, simulates the fault scenario and performs status monitoring; the replay optimization module generates a replay report based on the data and results of the fault drill process, and optimizes the fault propagation path and model parameters to achieve closed-loop feedback and continuous iterative optimization of the fault scenario. The present invention improves the accuracy and efficiency of operation and maintenance fault handling in a cloud environment through automated modeling, fault simulation and replay analysis, and is suitable for complex, multi-component fault training needs.

[0247] Please refer to the attached Figure 3 The present invention also provides a method for automatic operation and maintenance fault training and fault scenario arrangement based on a cloud environment, comprising the following steps:

[0248] S1. Model the cloud platform component status and fault propagation path based on Petri network to generate fault scenario model;

[0249] In this step, the fault modeling module formally models the states of each component of the cloud platform and the fault propagation path. By abstracting each component in the cloud platform into a state node and defining the state transition path of fault triggering and recovery, a fault scenario model is generated. This model is used to describe the behavior and dependency of each component in the fault scenario, providing basic data for subsequent fault propagation analysis and execution.

[0250] S2. Calculate the probability of fault propagation path based on Bayesian network and determine the optimal fault triggering path;

[0251] In this step, the fault prediction module uses the Bayesian network to dynamically reason about the fault propagation path based on the fault scenario model output by the fault modeling module, calculates the probability weight of each fault propagation path, and determines the optimal fault triggering path. The optimal path will serve as the basis for subsequent fault scenario execution to ensure that the fault simulation has high accuracy and coverage.

[0252] S3, convert the fault scenario model and the optimal fault triggering path into a containerized fault orchestration script;

[0253] In this step, the fault orchestration module automatically generates a containerized fault orchestration script based on the fault scenario model and the optimal trigger path provided by the fault prediction module. The fault orchestration script is described in a standardized format (such as Kubernetes YAML or Docker Compose), including fault type, trigger component, execution order, and parameter configuration. The generated script can be directly executed in the cloud platform to trigger fault simulation.

[0254] S4. Execute the fault orchestration script to trigger the fault and monitor the status of each component of the cloud platform;

[0255] In this step, the script execution module receives the script file generated by the fault orchestration module and executes the script in the cloud platform. During the execution process, the script execution module triggers the corresponding fault scenario, such as resource exhaustion, network interruption, or service crash. At the same time, the script execution module monitors the status of each component of the cloud platform in real time, including information such as resource utilization, service logs, and status changes, and records various data during the fault triggering and propagation process.

[0256] S5. Generate a replay report based on the data of the fault drill process, and provide feedback to optimize the fault scenario model and fault propagation probability path;

[0257] In this step, the replay optimization module analyzes the fault drill data collected by the script execution module and generates a fault replay report. The replay report includes the fault propagation path, fault trigger time, resource usage changes, and fault repair process. At the same time, based on the actual data feedback of the fault drill, the replay optimization module optimizes the fault propagation path weight in the fault scenario model and dynamically adjusts the fault propagation probability parameters, so that the fault scenario model is closer to the actual cloud platform fault behavior and realizes the closed-loop optimization of the system.

[0258] It can be understood that this method realizes the modeling, execution and review optimization process of the fault scenario through the above steps, and the data flow and logic between the modules are closely connected, ensuring the simulation accuracy of the fault scenario and the continuity of system optimization.

[0259] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. The cloud-based operation and maintenance fault training and fault scenario automatic orchestration system is characterized by: include: The fault modeling module is used to model the component status and fault propagation path of the cloud platform and generate a fault scenario model; The fault prediction module is used to dynamically infer the fault propagation path based on the Bayesian network, calculate the probability path of fault propagation, and optimize the fault triggering sequence; Fault orchestration module, which is used to automatically generate executable fault orchestration scripts based on fault scenario models and optimized fault triggering sequences; The script execution module is used to execute the fault orchestration script, simulate the fault scenario in the cloud platform, and perform fault triggering and status monitoring; The replay optimization module is used to generate a fault replay report based on the data and results of the fault drill process, and to provide feedback to optimize the fault scenario model and fault propagation probability path.

2. The cloud-based operation and maintenance fault training and fault scenario automatic arrangement system according to claim 1 is characterized in that: The fault modeling module models the fault propagation path through a Petri net, and the Petri net includes: A collection of places, used to represent the status of cloud platform components; Transition set, used to represent fault triggering and recovery operations; Weight function, used to describe the state transition weight in the fault propagation path; An initial tag set used to describe the initial state of fault propagation.

3. The cloud-based operation and maintenance fault training and fault scenario automatic arrangement system according to claim 1 is characterized in that: The fault prediction module performs dynamic reasoning of the fault propagation probability through a Bayesian network, and the Bayesian network includes: A collection of nodes, used to represent the status of each component of the cloud platform; A set of directed edges that describe the dependencies between components for fault propagation; Conditional probability distribution, used to represent the probability of a node state being in the state of its parent node.

4. The cloud-based operation and maintenance fault training and fault scenario automatic arrangement system according to claim 1 is characterized in that: The fault orchestration module converts the fault scenario model and the fault propagation probability path into a containerized script, and the containerized script supports execution in a Kubernetes cluster or Docker Compose environment.

5. The cloud-based operation and maintenance fault training and fault scenario automatic arrangement system according to claim 4 is characterized in that: The containerized script includes fault triggering instructions, which are used to simulate fault scenarios such as high load, network interruption, and service crash.

6. The cloud-based operation and maintenance fault training and fault scenario automatic orchestration system according to claim 1, characterized in that: During the fault drill, the script execution module monitors the resource usage, network status and service logs of the cloud platform components, and records the time points of fault propagation and recovery.

7. The cloud-based operation and maintenance fault training and fault scenario automatic orchestration system according to claim 1, characterized in that: The replay optimization module generates a fault replay report based on the monitoring data during the fault propagation process. The replay report includes: Fault trigger time and fault recovery time; Fault propagation path and propagation probability; Troubleshooting operations and execution results.

8. The cloud-based operation and maintenance fault training and fault scenario automatic orchestration system according to claim 1, characterized in that: The replay optimization module optimizes the fault propagation path weights in the Petri net according to the fault drill results, and dynamically adjusts the conditional probability distribution parameters in the Bayesian network.

9. The cloud-based operation and maintenance fault training and fault scenario automatic arrangement system according to claim 8, characterized in that: The dynamically adjusting the conditional probability distribution parameters in the Bayesian network includes: Receive monitoring data provided by the script execution module during the fault drill; Based on the actual data of the fault propagation path, the weight of the fault propagation path in the Petri net is updated, and the weight value is dynamically adjusted according to the fault triggering frequency and the probability of path occurrence; Recalculate the conditional probability distribution of each node in the Bayesian network based on the actual results of the monitoring data and the fault propagation path; Feedback the dynamically adjusted conditional probability distribution to the fault prediction module to optimize the calculation accuracy of the fault propagation probability; The optimized fault propagation path and conditional probability results are synchronously updated to the fault modeling module to further improve the fault scenario model and ensure that subsequent fault drills are closer to the actual fault status of the cloud platform.

10. A method for automatic operation and maintenance fault training and fault scenario arrangement based on a cloud environment, applied to a system as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: Model the cloud platform component status and fault propagation path based on Petri nets to generate fault scenario models; Calculate the probability of fault propagation paths based on Bayesian networks and determine the optimal fault triggering path; Convert the fault scenario model and the optimal fault trigger path into a containerized fault orchestration script; Execute fault orchestration scripts, trigger faults, and monitor the status of each component of the cloud platform; Generate a replay report based on the data from the fault drill process, and provide feedback to optimize the fault scenario model and fault propagation probability path.

Images