Chaotic engineering based on domestic operating system and method thereof
By introducing chaos engineering technology into domestic operating systems, real-time monitoring and fault injection, identifying weak links of the system and optimizing configuration, the problems of traditional fault processing time and insufficient prediction are solved, and the system stability and fault processing efficiency are improved.
Patent Information
- Application Number
- CN202411192573.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-13
AI Technical Summary
The troubleshooting of traditional domestic operating systems depends on historical data and experience, and cannot predict and deal with new fault types in a timely manner. The recovery process takes a long time and is prone to errors.
Using chaos engineering technology based on domestic operating systems, including risk perception and monitoring alarm module, fault injection and experimental plan design module, experimental execution and result analysis module, fault recovery and replication analysis module, collaboration module and management platform module, through real-time Monitor and fault injection, automated recovery and optimization of alarm strategies, identify weak links in the system and improve resilience.
Identify potential risks by simulating fault scenarios, optimize system configuration and fault processing processes, improve system stability and reliability, reduce fault processing time, and enhance business stability.
Smart Images
Figure CN119127608A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of chaos engineering technology, and in particular to a chaos engineering method based on a domestic operating system. Background Art
[0002] Chaos engineering is an emerging technical discipline. It is a set of methodologies that proactively identify the weak links in the system by conducting experiments on the system infrastructure. Through experimental methods, potential weak links in the system that may cause disasters or damage users are discovered, and R&D is encouraged to independently fix problems and optimize codes, ultimately building a truly resilient architecture to increase users' ability and confidence to resist emergencies. Fault handling in domestic operating systems is a complex process involving multiple aspects. When a fault occurs, the first step is to identify the fault. The system status and indicator data are collected in real time through the monitoring system, and the fault impact and system response are recorded. Then, the fault is located, the cause of the fault is analyzed, and the specific location of the fault is determined. Next, the fault is recovered. According to the preset fault recovery plan, automatic, semi-automatic or manual recovery operations are performed to ensure that the system resumes normal operation as soon as possible. At the same time, fault analysis is required to summarize the fault causes and handling experience to provide a basis for subsequent system improvements. In addition, fault reports are required to record the fault handling process and results to provide a reference for subsequent fault handling. After the fault handling is completed, a fault summary is required to reflect on and summarize the fault handling process, find out the shortcomings, and provide improvement measures for subsequent fault handling.
[0003] Traditional domestic operating system fault handling relies on historical data and experience, and is often unable to predict and respond to new types of faults in a timely manner. At the same time, the recovery process is time-consuming and prone to errors during the processing. Summary of the invention
[0004] To solve the above problems, the present invention provides a chaos engineering and method based on a domestic operating system.
[0005] In the first aspect, the present application provides a chaos engineering based on a domestic operating system, which adopts the following technical solutions: A chaos engineering based on a domestic operating system, including: risk perception and monitoring alarm module, fault injection and experimental scheme design module, experimental execution and result analysis module, fault recovery and replay analysis module, collaboration module, security assurance module and management platform module, among which, Risk perception and monitoring alarm module: For domestic operating systems, a monitoring indicator system is established and alarms are issued. The monitoring indicators cover key indicators of system resources, processes, services, and kernel parameters; Fault injection and experimental scheme design module: establish a fault type library covering system, process, network and storage dimensions, and design experimental schemes for the target system; Experiment execution and result analysis module: executes the experimental plan on the target system, injects preset faults, and observes system status and indicator changes in real time through the risk perception and monitoring alarm module, and records the impact of faults and system responses; Fault recovery and replay analysis module: includes fault recovery tools and replay analysis tools. The fault recovery tool is provided for the fault recovery function to restore the system status, and the replay analysis tool provides a visual interface for users to conduct replay summary. Collaboration module: When the monitoring system detects abnormal indicators, it automatically triggers the chaos engineering experiment, feeds the experimental results back to the monitoring system, and optimizes the alarm strategy and indicator threshold; Security assurance module: set permissions for different roles, operations, and resources; Management platform module: used for experiment management, resource management and data management.
[0006] In a possible implementation, the risk perception and monitoring alarm module establishes a monitoring indicator system for the domestic operating system and issues an alarm, including: Threshold-based alarm: set a threshold and trigger an alarm when the indicator exceeds the threshold; Trend-based alarm: monitor the trend of indicator changes and trigger an alarm when the indicator fluctuates abnormally; Alarm based on correlation: Analyze the correlation between indicators and trigger an alarm when multiple indicators are abnormal at the same time; Baseline-based alarm: Establish a system baseline and trigger an alarm when the system status deviates from the baseline; Alarm notification: supports multiple alarm notification methods, including SMS, email, WeChat, and DingTalk, to ensure timely notification of relevant personnel; Alarm visualization: Provides a visualization interface for alarm information.
[0007] In a possible implementation, the fault injection and experimental scheme design module designs an experimental scheme for the target system, including: Requirements analysis: Communicate with the business team to understand business requirements and system architecture, and determine system components and functions for business scenarios and test objectives; Fault type selection: Select the appropriate fault type based on the business scenario and test objectives, including system fault, process fault, network fault, and storage fault; Experimental design: Use experimental design tools to design experimental plans for test objectives, including setting experimental objectives, fault types, fault parameters, and expected results; Experimental plan review: Relevant personnel will review the experimental plan to ensure its scientificity and feasibility; Experiment plan publishing: Publish the experiment plan to the experiment orchestration engine to prepare for experiment execution.
[0008] In a possible implementation, the experiment execution and result analysis module executes the experiment plan on the target system, injects a preset fault, and observes the system status and indicator changes in real time through the risk perception and monitoring alarm module, and records the fault impact and system response, including: S1: Experiment execution: The experiment orchestration engine injects preset faults into the target system according to the experiment plan; S2: Result observation: The monitoring system collects system status and indicator data in real time and sends the data to the result analysis tool; S3: Result analysis: The result analysis tool analyzes the experimental results, evaluates the system resilience, and identifies weak links and potential risks; S4: Result visualization: The result analysis tool visualizes the analysis results; S5: Anomaly detection: Use machine learning technology to detect anomalies in experimental results.
[0009] In a possible implementation, the fault recovery tool provides a fault recovery function including: Automated recovery: Automatically perform recovery operations through scripts or automation tools, including restarting the system, restarting services, and restarting processes; Semi-automatic recovery: The operation and maintenance personnel manually perform some recovery operations, including restarting services and restarting processes; Manual recovery: All recovery operations are performed manually by the operation and maintenance personnel.
[0010] In one possible implementation, the review analysis tool provides a visual interface, through which users can review and analyze the experimental process and experimental results, summarize lessons learned, and optimize the experimental plan and troubleshooting process, including: Review of the experimental process: Review the experimental process and analyze whether the experimental steps are reasonable and whether the experimental operation is standardized; Experimental result analysis: Analyze the experimental results, evaluate the system resilience, and identify weak links and potential risks; Development of improvement measures: Development of improvement measures, including optimization of system architecture, optimization of system configuration, and optimization of fault handling process.
[0011] In one possible implementation, the collaborative module optimizes the alarm strategy and indicator thresholds, including: Adjust the alarm threshold: Adjust the alarm threshold based on the experimental results to avoid false positives and false negatives.
[0012] Add alarm rules: Add new alarm rules based on experimental results, including adding associated alarm rules, adding trend alarm rules, etc.
[0013] Modify the alarm level: Modify the alarm level based on the experimental results, including upgrading low-level alarms to high-level alarms.
[0014] In one possible implementation, the security assurance module sets permissions for different roles, different operations, and different resources, including: Role permissions: Set permissions for different roles, including administrators, operation and maintenance personnel, and developers. Different roles have different operation permissions; Operation permissions: Set permissions for different operations, including creating experiments, executing experiments, and viewing results. Different operations have different permissions; Resource permissions: Set permissions for different resources, including target systems and fault injection tools. Different resources have different permissions.
[0015] In one possible implementation, the management platform module is used for performing experiment management, resource management, and data management, including: Experiment management: Provides experimental plan management, experimental execution management, and experimental result management functions; Resource management: Provide experimental resource management, authority management, and audit management functions; Data management: Provides experimental data storage, analysis, and visualization functions.
[0016] In the second aspect, this application provides a chaos engineering method based on a domestic operating system, which adopts the following technical solutions: A chaos engineering method based on a domestic operating system, including: Step S101, demand analysis and goal setting: determine the test objectives and risk identification results according to business needs; Step S102, experimental scheme design, designing an experimental scheme based on the test objectives and the risk identification results; Step S103, experimental environment preparation: simulating the experimental environment based on the test objectives and the risk identification results; Step S104, experiment execution: execute the experiment plan in the simulated experiment environment and monitor the system status and indicator changes through indicator thresholds and alarm strategies; Step S105, result analysis: analyzing the system status, indicator changes and system recovery capability after executing the experimental plan to obtain analysis results; Step S106, review and improvement: improve the experimental plan based on the analysis results; Step S107, continuous iteration: execute steps S101 to S106 on the test target according to a preset cycle.
[0017] In summary, this application includes the following beneficial technical effects: 1. In this application, chaos engineering can help us better understand and evaluate the resilience of the system. By injecting faults into a real environment, we can simulate real fault scenarios and observe the state and response capabilities of the system after the fault occurs. This helps us identify potential weaknesses and potential risks in the system, so that we can take measures to improve it and enhance the resilience of the system. For example, experiments have shown that when there is a delay in the network, the system service response time increases significantly, indicating that the network is a potential risk point in the system. By optimizing the network configuration and increasing the network bandwidth, the impact of network failures on the system can be effectively reduced, thereby improving the resilience of the system.
[0018] 2. In this application, chaos engineering can help us improve the stability and reliability of the system. By conducting chaos engineering experiments regularly, we can continuously test the resilience of the system, identify weak links and potential risks, and continuously improve the system based on the experimental results to enhance the resilience and security of the system. For example, experiments have shown that when the disk I / O is abnormal, the system performance decreases significantly, indicating that the disk is a potential risk point in the system. By optimizing disk configuration and improving disk performance, the impact of disk failures on the system can be effectively reduced, thereby improving the stability of the system.
[0019] 3. In this application, chaos engineering can also help us optimize the fault handling process and monitoring system. Through the analysis of experimental results, we can find the deficiencies in the fault handling process, thereby optimizing the fault handling process and improving the efficiency of fault handling. At the same time, chaos engineering can also help us find blind spots in the monitoring system, thereby optimizing the monitoring strategy and improving the accuracy of monitoring. In summary, chaos engineering and its methods based on domestic operating systems can help us better understand and evaluate the resilience of the system, improve the stability and reliability of the system, optimize the fault handling process and monitoring system, and thus ensure the stable operation of the business. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a system framework diagram of chaos engineering based on a domestic operating system in an embodiment of the present application. DETAILED DESCRIPTION
[0021] The present application is further described in detail below in conjunction with the accompanying drawings.
[0022] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.
[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0024] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.
[0025] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.
[0026] The embodiment of the present application provides a chaos engineering and method based on a domestic operating system, wherein: Figure 1 As shown in the figure, chaos engineering includes: risk perception and monitoring alarm module, fault injection and experimental scheme design module, experimental execution and result analysis module, fault recovery and replay analysis module, collaboration module, security assurance module and management platform module. Among them, the experimental execution and result analysis module, the fault recovery and replay analysis module and the collaboration module are respectively connected to the risk perception and monitoring alarm module, the fault injection and experimental scheme design module is connected to the experimental execution and result analysis module, the fault recovery and replay analysis module and the security assurance module are respectively connected to the management platform module, the fault injection and experimental scheme design module, the experimental execution and result analysis module and the fault recovery and replay analysis module are respectively connected to the monitoring and chaos engineering system module. The detailed function description of each module is as follows: 1. Risk Perception and Monitoring Alarm Module Including a monitoring system, for domestic operating systems, a complete set of monitoring indicator systems is established and alarms are issued through the monitoring system, covering key indicators of system resources, processes, services, and kernel parameters. Specifically, in the embodiment of the present application, the monitoring system in the risk perception and monitoring alarm module selects the Prometheus monitoring tool compatible with the domestic operating system. More specifically, the risk perception and monitoring alarm module establishes a monitoring indicator system for the domestic operating system and issues alarms, including: Threshold-based alarm: set a threshold and trigger an alarm when the indicator exceeds the threshold; Trend-based alarm: monitor the trend of indicator changes and trigger an alarm when the indicator fluctuates abnormally; Alarm based on correlation: Analyze the correlation between indicators and trigger an alarm when multiple indicators are abnormal at the same time; Baseline-based alarm: Establish a system baseline and trigger an alarm when the system status deviates from the baseline; Alarm notification: supports multiple alarm notification methods, including SMS, email, WeChat, and DingTalk, to ensure timely notification of relevant personnel; Alarm visualization: Provides a visualization interface for alarm information.
[0027] 2. Fault Injection and Experimental Design Module Establish a complete set of fault type libraries, covering system, process, network and storage dimensions, and design experimental plans for the target system. The specific process of designing experimental plans includes: Requirements analysis: Communicate with the business team to understand business requirements and system architecture, and determine system components and functions for business scenarios and test objectives; Fault type selection: Select the appropriate fault type based on the business scenario and test objectives, including system fault, process fault, network fault, and storage fault; Experimental design: Use experimental design tools to design experimental plans for test objectives, including setting experimental objectives, fault types, fault parameters, and expected results; Experimental plan review: Relevant personnel will review the experimental plan to ensure its scientificity and feasibility; Experiment plan publishing: Publish the experiment plan to the experiment orchestration engine to prepare for experiment execution.
[0028] 3. Experiment Execution and Result Analysis Module Execute the experimental plan on the target system, inject the preset faults, and observe the system status and indicator changes in real time through the monitoring system in the risk perception and monitoring alarm module, and record the fault impact and system response. Specifically, the operation process of the experiment execution and result analysis module is as follows: S1: Experiment execution: The experiment orchestration engine injects preset faults into the target system according to the experiment plan, including: Script injection: Simulate failures through scripts, including killing processes, interrupting network connections, etc.
[0029] Simulation injection: Use simulation tools to simulate failures, including simulated network delays, simulated disk damage, etc.
[0030] Real injection: Inject faults into a real environment, including unplugging network cables and hard drives.
[0031] S2: Result observation: The monitoring system collects system status and indicator data in real time and sends the data to the result analysis tool. Including: System status: including whether the system has crashed, whether the system services are normal, etc.
[0032] Metrics data: including CPU usage, memory usage, disk I / O, network traffic, etc.
[0033] Application status: including whether the application crashes, whether the application responds, etc.
[0034] Log information: including system logs, application logs, etc.
[0035] S3: Result analysis: The result analysis tool analyzes the experimental results, evaluates system resilience, and identifies weak links and potential risks, including: Fault impact scope: Analyze the impact scope of the fault on the system, including which components and services are affected by the fault.
[0036] System recovery time: Analyze the time required for the system to recover from a faulty state to a normal state.
[0037] System performance degradation: Analyze the impact of the fault on system performance.
[0038] System self-recovery capability: Analyze whether the system can recover automatically and how effective the recovery is.
[0039] Fault handling process: Analyze whether the fault handling process is reasonable and whether there is room for improvement. ; S4: Result visualization: The result analysis tool visualizes the analysis results, including: Charts: including line charts, bar charts, pie charts, etc.
[0040] Curves: including system indicator change curve, fault impact curve, etc.
[0041] Maps: including system topology maps, fault impact maps, etc.
[0042] S5: Anomaly detection: Use machine learning technology to detect anomalies in experimental results and identify potential system vulnerabilities and weaknesses, including: Abnormal pattern recognition: Identify abnormal system behavior patterns, including frequent crashes, frequent freezes, etc.
[0043] Abnormal correlation analysis: Analyze the correlation between abnormal events, including the crash of a process causing the crash of the entire system.
[0044] Abnormality prediction: predict possible abnormalities in the system in the future and take preventive measures in advance.
[0045] 4. Fault recovery and replay analysis module The fault recovery and replay analysis module includes fault recovery tools and replay analysis tools.
[0046] 4.1 Fault recovery tools are provided for fault recovery functions to quickly restore system status; Specifically, in the embodiment of the present application, the fault recovery tool is provided to the fault recovery function according to the preset fault recovery plan after the experiment is completed, so as to quickly restore the system status, including: Automated recovery: automatically performing recovery operations through scripts or automation tools, including restarting the system, restarting services, restarting processes, etc. Semi-automated recovery: partial recovery operations are manually performed by operation and maintenance personnel, including restarting services, restarting processes, etc. Manual recovery: all recovery operations are manually performed by operation and maintenance personnel.
[0047] 4.2 The review analysis tool provides a visual interface for users to review and summarize experiences and lessons, optimize experimental plans and troubleshooting processes.
[0048] Specifically, users can review and analyze the experimental process and results based on the visual interface provided by the review analysis tool, summarize lessons learned, and optimize the experimental plan and troubleshooting process, including: Review of the experimental process: Review the experimental process and analyze whether the experimental steps are reasonable and whether the experimental operation is standardized; Experimental result analysis: Analyze the experimental results, evaluate the system resilience, and identify weak links and potential risks; Development of improvement measures: Development of improvement measures, including optimization of system architecture, optimization of system configuration, and optimization of fault handling process.
[0049] 5. Collaboration Module When the monitoring system detects abnormal indicators, the collaborative module automatically triggers a chaos engineering experiment (a chaos engineering experiment refers to the execution and implementation process of an experimental plan) to verify the system's ability to respond to abnormal situations, evaluate the system's self-recovery capabilities, and feed back the experimental results to the monitoring system to optimize the alarm strategy and indicator thresholds. The optimization of the alarm strategy and indicator thresholds includes: (1) adjusting the alarm threshold: adjusting the alarm threshold based on the experimental results to avoid false positives and false negatives; (2) adding alarm rules: adding new alarm rules based on the experimental results, including adding associated alarm rules, adding trend alarm rules, etc.; (3) modifying the alarm level: modifying the alarm level based on the experimental results, including upgrading low-level alarms to high-level alarms.
[0050] Furthermore, when the monitoring system detects abnormal indicators, the collaborative module automatically triggers the chaos engineering experiment, including: When the CPU usage, memory usage, or disk usage is too high, the system crash experiment (one of the experimental schemes designed by the fault injection and experimental scheme design module) is automatically triggered to verify whether the system can recover automatically; When a process crashes, process resources are exhausted, or process communication is abnormal, the process crash experiment is automatically triggered (the process crash experiment is one of the experimental schemes designed by the fault injection and experimental scheme design module) to verify whether the process can automatically restart or return to normal status; When the network is interrupted, delayed, or lost, the network interruption experiment is automatically triggered (the network interruption experiment is one of the experimental schemes designed by the fault injection and experimental scheme design module) to verify whether the system can automatically switch to the backup network or restore the normal network connection; When the disk is damaged, the disk I / O is abnormal, or the file system is damaged, the disk damage experiment is automatically triggered (the disk damage experiment is one of the experimental schemes designed by the fault injection and experimental scheme design module) to verify whether the system can automatically switch to the backup storage or restore the normal storage state; When the application crashes, the application resources are exhausted, or the application is abnormal, the application crash experiment is automatically triggered (the application crash experiment is one of the experimental schemes designed by the fault injection and experimental scheme design module) to verify whether the application can automatically restart or return to normal.
[0051] Furthermore, the collaboration module is also equipped with a visual analysis platform to visualize monitoring data and chaos engineering experiment results, helping operation and maintenance personnel to quickly locate problems and analyze system status, and providing multiple analysis dimensions, including: Fault impact scope: Analyze the impact scope of the fault on the system, including which components and services are affected by the fault; System recovery time: The time required for the analysis system to recover from a faulty state to a normal state; System performance degradation: Analyze the impact of the fault on system performance; System self-recovery capability: Analyze whether the system can recover automatically and how effective the recovery is; Troubleshooting process: Analyze whether the troubleshooting process is reasonable and whether there is room for improvement.
[0052] 6. Security Assurance Module The security assurance module establishes a strict experimental authority management system to ensure that only authorized personnel can perform experiments; isolates the experimental environment from the production environment to prevent the experiment from affecting the production environment; and formulates a complete fault recovery plan to ensure that the system status can be restored in time when problems occur during the experiment.
[0053] The experimental permission management system includes: role permission, operation permission and resource permission. Among them, (1) Role permission: setting permissions for different roles, including administrators, operation and maintenance personnel, developers, etc. Different roles have different operation permissions; (2) Operation permission: setting permissions for different operations, including creating experiments, executing experiments, viewing results, etc. Different operations have different permissions; (3) Resource permission: setting permissions for different resources, including target systems, fault injection tools, etc. Different resources have different permissions.
[0054] Furthermore, after the security assurance module has formulated a complete fault recovery plan, the fault recovery and replay analysis module executes the fault recovery plan to perform automatic, semi-automatic or manual recovery of the domestic operating system.
[0055] 7. Management Platform Module Used to create experimental plans and perform experimental management, resource management, and data management. The experimental plan includes information such as the experiment name, experimental objectives, experimental time, and experimental personnel.
[0056] 7.1 Experiment management: Provides experimental plan management, experimental execution management, and experimental result management functions; 7.2 Resource Management: Provides experimental resource management, authority management, and audit management functions, including: 7.2.1 Experimental resource management includes: Add target system: Add the target system to the management platform and set the target system information, including system name, system IP address, etc.
[0057] Delete target system: Delete the target system.
[0058] Modify target system information: Modify target system information, including modifying the system name, modifying the system IP address, etc.
[0059] Add a fault injection tool: Add the fault injection tool to the management platform and set the fault injection tool information, including the tool name and tool version.
[0060] Remove Fault Injection Tool: Removes the fault injection tool.
[0061] Modify fault injection tool information: Modify fault injection tool information, including modifying the tool name, modifying the tool version, etc.
[0062] 7.2.2 Permission management refers to the management of the experimental permission management system set up by the security assurance module.
[0063] 7.2.3 Audit management refers to the management of audit logs, including: Viewing user operation records: Viewing all operations of users on the management platform, including creating experiments, executing experiments, viewing results, etc.
[0064] View the experiment execution record: View the log information during the experiment execution, including fault injection log and fault recovery log.
[0065] 7.3 Data management: Provide experimental data storage, analysis, and visualization functions.
[0066] The above is an introduction to the functions of various modules of a chaos engineering based on a domestic operating system in an embodiment of the present application. For ease of understanding, the embodiment of the present application also provides a chaos engineering method based on a domestic operating system to illustrate the specific implementation process of the chaos engineering, including: Step S101, demand analysis and goal setting: determine the test objectives and risk identification results according to business requirements, where the test objectives are the target systems that need to be tested.
[0067] For the embodiments of the present application, the starting point of chaos engineering is to clarify the test objectives and risk identification. First, through in-depth business interviews, we can fully understand the business processes, key business systems, and key indicators that affect the business, so as to identify the potential weaknesses and potential risks of the system. Secondly, by analyzing the system architecture diagram, we can clearly understand the relationship between system components, system dependencies, and the working principle of the system, providing a basis for further experimental design. In addition, using methods such as fault tree analysis and failure mode and effect analysis, we can conduct in-depth fault analysis of the system and identify potential weaknesses and potential risks of the system. Finally, based on business needs and risk identification results, we can determine the chaos engineering test objectives, such as improving the system's fault tolerance, verifying the system's recovery capabilities, testing system performance, etc., to ensure that the chaos engineering experiment is targeted.
[0068] Step S102, experimental scheme design, designs the experimental scheme based on the test objectives and risk identification results.
[0069] For the embodiments of the present application, the design of the experimental scheme is the core link of chaos engineering. First, it is necessary to select appropriate fault types, such as system faults, process faults, network faults, and storage faults, according to the test objectives and risk identification results, to ensure that the experiment can effectively test the resilience of the system. Secondly, it is necessary to set fault parameters, such as fault duration, fault trigger conditions, etc., to ensure the authenticity and effectiveness of the fault simulation and avoid irreversible damage to the system. Thirdly, it is necessary to design experimental processes, such as fault injection, result observation, fault recovery, etc., to ensure the clarity and executability of the experimental process and facilitate the execution of the experiment and result analysis. Finally, it is necessary to formulate a fault recovery plan, such as automatic recovery, semi-automatic recovery, manual recovery, etc., to ensure that the system status can be restored in time when problems occur during the experiment to avoid affecting the business.
[0070] Step S103, experimental environment preparation: simulating the experimental environment based on the test objectives and risk identification results.
[0071] For the embodiments of the present application, the preparation of the experimental environment is the basis for ensuring the smooth progress of the chaos engineering experiment. First, it is necessary to deploy a monitoring system on the target system, such as Prometheus, Zabbix, etc., to collect system indicator data and alarm information, and provide a data basis for the observation and analysis of experimental results. Secondly, it is necessary to prepare fault injection tools, such as ChaosMesh, Toxiproxy, etc., to inject preset faults into the target system to ensure that the experiment can effectively simulate the real fault scenario. In addition, it is also necessary to configure the experimental environment, such as setting network isolation, setting system parameters, etc., to ensure that the experimental environment is isolated from the production environment and avoid the impact of the experiment on the production environment.
[0072] Step S104, experiment execution: execute the experiment plan in the simulated experiment environment and monitor the system status and indicator changes through indicator thresholds and alarm strategies.
[0073] For the embodiments of the present application, the experimental execution is the practical link of the chaos engineering experiment. First, it is necessary to use a fault injection tool to inject preset faults on the target system, such as system crashes, process crashes, network interruptions, etc., to simulate real fault scenarios. Secondly, it is necessary to use a monitoring system to observe system status and indicator changes in real time, such as CPU usage, memory usage, disk I / O, network traffic, etc., record the impact of faults and system responses, and provide data support for subsequent result analysis. In addition, if an abnormal situation occurs during the experiment, such as a system crash, service unavailability, etc., it is necessary to handle it in a timely manner, such as restarting the system, restarting the service, restarting the process, etc., to ensure that the experiment proceeds smoothly and avoid affecting the business.
[0074] Step S105, result analysis: analyzing the system status, indicator changes and system recovery capability after executing the experimental plan to obtain analysis results.
[0075] For the embodiments of the present application, result analysis is a key link in the chaos engineering experiment. First, it is necessary to analyze the state of the system after the failure occurs, such as whether the system crashes, whether the system service is normal, etc., to evaluate the system's ability to respond to failures. Secondly, it is necessary to analyze the changing trend of system indicator data, such as CPU utilization, memory utilization, disk I / O, network traffic, etc., to evaluate the system's tolerance for failures. Thirdly, it is necessary to evaluate the system's recovery ability after a failure occurs, such as whether the system can recover automatically, the system recovery time, etc., to evaluate the system's self-recovery ability. Finally, it is necessary to identify potential weaknesses and potential risks in the system, such as unreasonable system architecture design, unreasonable system configuration, unclear system dependencies, etc., to provide a basis for subsequent system improvements.
[0076] Step S106, review and improvement: improve the experimental plan based on the analysis results.
[0077] For the embodiments of the present application, review and improvement are the continuous improvement links of chaos engineering experiments. First, it is necessary to summarize the lessons learned during the experiment, such as whether the experimental scheme design is reasonable, whether the experimental operation is standardized, etc., so as to make improvements in subsequent experiments. Secondly, it is necessary to optimize the experimental scheme according to the experimental results, such as adjusting the fault type, adjusting the fault parameters, etc., to ensure that the experiment can more effectively test the resilience of the system. Thirdly, it is necessary to improve the system architecture according to the experimental results, such as optimizing the system design, optimizing the system configuration, etc., to enhance the system's fault tolerance and recovery capabilities. Finally, it is necessary to formulate an improvement plan, such as optimizing the fault handling process, optimizing the monitoring system, etc., to ensure that the system can better respond to faults and ensure stable business operations.
[0078] Step S107, continuous iteration: execute steps S101 to S106 on the test target according to a preset cycle.
[0079] For the embodiments of the present application, chaos engineering is a process of continuous improvement. Chaos engineering experiments need to be conducted regularly to continuously test system resilience, identify weak links and potential risks, and continuously improve the system based on the experimental results to enhance system resilience and security. In addition, chaos engineering needs to be combined with the monitoring system to form a closed-loop management to ensure stable operation of the system. Through continuous iteration, the stability and reliability of the system can be continuously improved, providing a solid foundation for business development.
[0080] This application can help us better understand and evaluate the resilience of the system through chaos engineering. Specifically, by injecting faults into the real environment, we can simulate real fault scenarios and observe the state and response capabilities of the system after the fault occurs. This helps us identify potential weaknesses and potential risks in the system, so that we can take measures to improve and enhance the resilience of the system. For example, experiments have found that when there is a delay in the network, the system service response time increases significantly, which indicates that the network is a potential risk point in the system. By optimizing the network configuration and increasing the network bandwidth, the impact of network failures on the system can be effectively reduced, thereby improving the resilience of the system.
[0081] This application can also help us improve the stability and reliability of the system through chaos engineering. By conducting chaos engineering experiments regularly, we can continuously test the resilience of the system, identify weak links and potential risks, and continuously improve the system based on the experimental results to enhance the resilience and security of the system. For example, through experiments, it was found that when the disk I / O is abnormal, the system performance is significantly reduced, which indicates that the disk is a potential risk point in the system. By optimizing the disk configuration and improving the disk performance, the impact of disk failures on the system can be effectively reduced, thereby improving the stability of the system.
[0082] Furthermore, this application also achieves the effect of optimizing the fault handling process and monitoring system through chaos engineering. Through the analysis of experimental results, we can find the shortcomings in the fault handling process, thereby optimizing the fault handling process and improving the efficiency of fault handling. At the same time, chaos engineering can also help us find blind spots in the monitoring system, thereby optimizing the monitoring strategy and improving the accuracy of monitoring. In summary, chaos engineering and its methods based on domestic operating systems can help us better understand and evaluate the resilience of the system, improve the stability and reliability of the system, optimize the fault handling process and monitoring system, and thus ensure the stable operation of the business.
[0083] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0084] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A chaos engineering based on a domestic operating system, characterized in that: The chaos engineering includes: risk perception and monitoring alarm module, fault injection and experimental scheme design module, experimental execution and result analysis module, fault recovery and replay analysis module, collaboration module, security assurance module and management platform module, among which, Risk perception and monitoring alarm module: including monitoring system, establishing monitoring indicator system and monitoring system alarm for domestic operating system, the monitoring indicators cover key indicators of system resources, processes, services and kernel parameters; Fault injection and experimental scheme design module: establish a fault type library covering system, process, network and storage dimensions, and design experimental schemes for the target system; Experiment execution and result analysis module: executes the experimental plan on the target system, injects preset faults, and observes system status and indicator changes in real time through the risk perception and monitoring alarm module, and records the impact of faults and system responses; Fault recovery and replay analysis module: includes fault recovery tools and replay analysis tools. The fault recovery tool is provided for the fault recovery function to restore the system status, and the replay analysis tool provides a visual interface for users to conduct replay summary. Collaboration module: When the monitoring system detects abnormal indicators, it automatically triggers the chaos engineering experiment, feeds the experimental results back to the monitoring system, and optimizes the alarm strategy and indicator threshold; Security assurance module: set permissions for different roles, operations, and resources; Management platform module: used for experiment management, resource management and data management.
2. According to claim 1, a chaos engineering based on a domestic operating system is characterized in that: The risk perception and monitoring alarm module establishes a monitoring indicator system for domestic operating systems and issues alarms, including: Threshold-based alarm: set a threshold and trigger an alarm when the indicator exceeds the threshold; Trend-based alarm: monitor the trend of indicator changes and trigger an alarm when the indicator fluctuates abnormally; Alarm based on correlation: Analyze the correlation between indicators and trigger an alarm when multiple indicators are abnormal at the same time; Baseline-based alarm: Establish a system baseline and trigger an alarm when the system status deviates from the baseline; Alarm notification: supports multiple alarm notification methods, including SMS, email, WeChat, and DingTalk, to ensure timely notification of relevant personnel; Alarm visualization: Provides a visualization interface for alarm information.
3. According to claim 1, a chaos engineering based on a domestic operating system is characterized in that: The fault injection and experimental scheme design module designs an experimental scheme for the target system, including: Requirements analysis: Communicate with the business team to understand business requirements and system architecture, and determine system components and functions for business scenarios and test objectives; Fault type selection: Select the appropriate fault type based on the business scenario and test objectives, including system fault, process fault, network fault, and storage fault; Experimental design: Use experimental design tools to design experimental plans for test objectives, including setting experimental objectives, fault types, fault parameters, and expected results; Experimental plan review: Relevant personnel will review the experimental plan to ensure its scientificity and feasibility; Experiment plan publishing: Publish the experiment plan to the experiment orchestration engine to prepare for experiment execution.
4. According to claim 1, a chaos engineering based on a domestic operating system is characterized in that: The experiment execution and result analysis module executes the experiment plan on the target system, injects the preset fault, observes the system status and indicator changes in real time through the risk perception and monitoring alarm module, and records the fault impact and system response, including: S1: Experiment execution: The experiment orchestration engine injects preset faults into the target system according to the experiment plan; S2: Result observation: The monitoring system collects system status and indicator data in real time and sends the data to the result analysis tool; S3: Result analysis: The result analysis tool analyzes the experimental results, evaluates the system resilience, and identifies weak links and potential risks; S4: Result visualization: The result analysis tool visualizes the analysis results; S5: Anomaly detection: Use machine learning technology to detect anomalies in experimental results.
5. According to claim 1, a chaos engineering based on a domestic operating system is characterized in that: Fault recovery tools are provided for fault recovery functions, including: Automated recovery: Automatically perform recovery operations through scripts or automation tools, including restarting the system, restarting services, and restarting processes; Semi-automatic recovery: The operation and maintenance personnel manually perform some recovery operations, including restarting services and restarting processes; Manual recovery: All recovery operations are performed manually by the operation and maintenance personnel.
6. The chaos engineering based on a domestic operating system according to claim 1 is characterized in that: The review analysis tool provides a visual interface. Users can review and analyze the experimental process and experimental results based on the visual interface, summarize lessons learned, and optimize the experimental plan and troubleshooting process, including: Review of the experimental process: Review the experimental process and analyze whether the experimental steps are reasonable and whether the experimental operation is standardized; Experimental result analysis: Analyze the experimental results, evaluate the system resilience, and identify weak links and potential risks; Development of improvement measures: Development of improvement measures, including optimization of system architecture, optimization of system configuration, and optimization of fault handling process.
7. The chaos engineering based on a domestic operating system according to claim 1 is characterized in that: The collaborative module optimizes alarm strategies and indicator thresholds, including: Adjust the alarm threshold: According to the experimental results, adjust the alarm threshold to avoid false positives and false negatives; Add alarm rules: Add new alarm rules based on experimental results, including adding correlation alarm rules, adding trend alarm rules, etc.; Modify the alarm level: Modify the alarm level based on the experimental results, including upgrading low-level alarms to high-level alarms.
8. The chaos engineering based on a domestic operating system according to claim 1 is characterized in that: Security Assurance Module: Set permissions for different roles, operations, and resources, including: Role permissions: Set permissions for different roles, including administrators, operation and maintenance personnel, and developers. Different roles have different operation permissions; Operation permissions: Set permissions for different operations, including creating experiments, executing experiments, and viewing results. Different operations have different permissions; Resource permissions: Set permissions for different resources, including target systems and fault injection tools. Different resources have different permissions.
9. The chaos engineering based on a domestic operating system according to claim 1 is characterized in that: Management platform module: used for experiment management, resource management and data management, including: Experiment management: Provides experimental plan management, experimental execution management, and experimental result management functions; Resource management: Provide experimental resource management, authority management, and audit management functions; Data management: Provides experimental data storage, analysis, and visualization functions.
10. A chaos engineering method based on a domestic operating system, characterized in that: include: Step S101, demand analysis and goal setting: determine the test objectives and risk identification results according to business needs; Step S102, experimental scheme design, designing an experimental scheme based on the test objectives and the risk identification results; Step S103, experimental environment preparation: simulating the experimental environment based on the test objectives and the risk identification results; Step S104, experiment execution: execute the experiment plan in the simulated experiment environment and monitor the system status and indicator changes through indicator thresholds and alarm strategies; Step S105, result analysis: analyzing the system status, indicator changes and system recovery capability after executing the experimental plan to obtain analysis results; Step S106, review and improvement: improve the experimental plan based on the analysis results; Step S107, continuous iteration: execute steps S101 to S106 on the test target according to a preset cycle.