Failure mode analysis method, system, electronic device, and medium

Abstract

The application relates to the field of electronic technology and discloses a fault mode analysis method, a fault mode analysis system, an electronic device and a medium. The method comprises the following steps: acquiring components of a system to be analyzed and parameters of the components; acquiring external components interacting with the components of the system to be analyzed and parameters of the external components; acquiring a topological structure between the components of the system to be analyzed and the external components; acquiring corresponding fault modes from a preset fault mode library according to the parameters of the components of the system to be analyzed, the parameters of the external components and the topological structure, and generating a fault mode list of the system to be analyzed. In this way, the efficiency and completeness of fault mode analysis are improved, and when the system to be analyzed changes, the updating of the fault mode list can be completed by making corresponding modifications to the topological structure of the system to be analyzed and the like, thereby improving the maintenance efficiency of the fault mode list.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a failure mode analysis method, system, electronic device, and medium. Background Technology

[0002] Chaos engineering, which involves conducting experience-guided controlled experiments to observe system behavior and identify weaknesses, has gradually become a primary means of verifying the reliability of live network services and improving the efficiency of drills. For example... Figure 1A As shown, the implementation of chaos engineering mainly includes steps such as failure mode analysis (FMA), task scheduling, task drills, and result analysis. Among these, FMA involves analyzing the system's subsystems, modules, and workflows one by one to identify potential failure modes and their possible impact, thus pinpointing the system's weaknesses and potential risks. The efficiency of FMA and the completeness of scenario coverage directly affect the threshold and effectiveness of chaos engineering implementation.

[0003] In existing technologies, failure mode analysts perform failure mode analysis on live network systems using web (World Wide Web) pages or spreadsheets, such as... Figure 1B As shown, failure mode analysts determine the failure modes of sub-modules involved in the system under analysis, and then summarize the failure modes of sub-modules to generate a failure mode set for the system under analysis. It is evident that failure mode analysts perform failure mode analysis on web pages or spreadsheets, and the quality of the analysis depends heavily on the analyst's experience, raising the barrier to entry for chaos engineering and making it difficult to guarantee the completeness of failure mode coverage. Furthermore, web pages or Excel spreadsheets cannot hold data such as system component dependencies and access chains, resulting in high maintenance costs when system call relationships and access chains change. In addition, when the system's network topology or external interfaces change, failure mode analysis needs to be performed again, which is inefficient and incurrs high costs for modifying and maintaining failure modes. Summary of the Invention

[0004] In view of this, embodiments of this application provide a fault mode analysis method, system, electronic device, and medium. By analyzing the topology of the system under analysis, a fault mode list for the system under analysis is automatically generated for fault simulation in chaos engineering. This improves the efficiency and completeness of scenario coverage in fault mode analysis, lowers the barrier to entry for using chaos engineering, and allows for updating the fault mode list simply by modifying the topology of the system under analysis when changes occur in component dependencies, access chains, network topology, or external interfaces, thus improving maintenance efficiency and reducing maintenance costs.

[0005] In a first aspect, embodiments of this application provide a fault mode analysis method, which includes: acquiring components of a system to be analyzed and parameters of the components; acquiring external components that interact with the components of the system to be analyzed and parameters of the external components; acquiring the topology between the components of the system to be analyzed and the external components; acquiring corresponding fault modes from a preset fault mode library based on the parameters of the components of the system to be analyzed, the parameters of the external components that interact, and the topology, generating a fault mode list for the system to be analyzed, and the fault mode list is used to perform fault drills on the system to be analyzed in chaos engineering.

[0006] In this way, by constructing the topology of the system to be analyzed, a list of failure modes of the system to be analyzed can be obtained, which improves the efficiency and completeness of failure mode analysis, thereby improving the efficiency of using chaos engineering for failure mode simulation.

[0007] In one possible implementation of the first aspect above, the parameters of the components of the system to be analyzed include at least one of the following parameters: the type of the component, the number of nodes included in the component, the deployment method of the nodes in the component, the IP address corresponding to the component and the component nodes, and the process corresponding to the component.

[0008] In one possible implementation of the first aspect above, the topology includes at least one of the following: upstream components and access methods accessed by components of the system to be analyzed; downstream components and access methods accessed by components of the system to be analyzed; and service components and access methods accessed by components of the system to be analyzed.

[0009] In one possible implementation of the first aspect above, the above-mentioned method of obtaining corresponding fault modes from a pre-set fault mode library based on the parameters of the components of the system to be analyzed, the parameters of the external components with which the interaction occurs, and the topology, and generating a fault mode list of the system to be analyzed, includes at least one of the following methods: generating basic fault classes based on the parameters of the components, including host fault classes, resource fault classes, and process fault classes; generating network fault classes and other fault classes based on the upstream components accessed by the components; generating network fault classes based on the downstream components accessed by the components; and generating service fault classes based on the service components accessed by the components.

[0010] In one possible implementation of the first aspect above, the method further includes: acquiring parameter changes of the system to be analyzed; and updating the fault mode list based on the parameter changes.

[0011] In this solution, when the parameters of the components of the system to be analyzed change (such as changes in the deployment method of nodes in the components, the number of nodes, or the IP addresses of the nodes), components are added or removed, or external components that interact with the components of the system to be analyzed change, the failure mode list of the system to be analyzed can be updated according to the changes in the topology by making corresponding modifications to the topology. This improves the maintenance efficiency of the failure mode list.

[0012] In one possible implementation of the first aspect above, the parameter changes include at least one of the following: changes in the parameters of the components of the system to be analyzed; changes in the parameters of external components with which there is interaction; and changes in the topology.

[0013] In one possible implementation of the first aspect above, the aforementioned host-type failures include host offline or host restart.

[0014] In one possible implementation of the first aspect above, the aforementioned process-related faults include at least one of the following faults: process suspension, process failure to start, and process exit.

[0015] In one possible implementation of the first aspect above, the aforementioned resource-related faults include at least one of the following faults: host processor overload, host memory overload, host disk I / O high, and host disk full.

[0016] In one possible implementation of the first aspect above, the aforementioned network-related faults include at least one of the following faults: component access failure, component access packet loss, component access delay, component access return error, network card failure, network packet corruption, network link packet loss, network link delay, and network rate limiting.

[0017] In one possible implementation of the first aspect described above, the other types of faults mentioned above include external request overload.

[0018] In one possible implementation of the first aspect above, the aforementioned service failures include at least one of the following: service master node failure, service backup node failure, service availability zone failure, service data loss, and password expiration.

[0019] Secondly, embodiments of this application provide a fault mode analysis system, which includes: an application editor, a topology editor, and a fault mode analyzer; wherein: the application editor is used to obtain the components of the system to be analyzed and the parameters of the components; the topology editor is used to obtain the topology between the components of the system to be analyzed obtained by the application editor and the external components that interact with the system to be analyzed; the fault mode analyzer includes a fault mode library, and the fault mode analyzer is used to obtain the corresponding fault modes from the fault mode library according to the parameters of the components, the parameters of the external components that interact with the system, and the topology, and generate a fault mode list of the system to be analyzed.

[0020] Thirdly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform any of the fault mode analysis methods described in the first aspect above.

[0021] Fourthly, embodiments of this application provide a program product for implementing the fault mode analysis method described in the second aspect above.

[0022] Fifthly, embodiments of this application provide an electronic device, the electronic device comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the processors of the electronic device, for executing the instructions stored in the memory to implement any of the fault mode analysis methods described in the first aspect above. Attached Figure Description

[0023] Figure 1A A flowchart illustrating the workflow of chaos engineering is shown.

[0024] Figure 1B A flowchart of a failure mode analysis is shown;

[0025] Figure 2 According to some embodiments of this application, a schematic diagram of an application scenario for a fault mode analysis method is shown;

[0026] Figure 3A According to some embodiments of this application, a schematic diagram of the structure of a fault mode analysis system is shown;

[0027] Figure 3B According to some embodiments of this application, a schematic diagram of fault types in a fault mode library is shown;

[0028] Figure 4 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown;

[0029] Figure 5 According to some embodiments of this application, a flowchart of a failure mode analysis is shown;

[0030] Figure 6 According to some embodiments of this application, a schematic diagram of the interface of a component editor is shown;

[0031] Figure 7A According to some embodiments of this application, a schematic diagram of an interface for editing components in a component editor is shown;

[0032] Figure 7B According to some embodiments of this application, a schematic diagram of an interface for editing components in a component editor is shown;

[0033] Figure 8 According to some embodiments of this application, a schematic diagram of the interface of a topology editor is shown;

[0034] Figure 9 According to some embodiments of this application, a schematic diagram of the topology of system 10 in a topology editor is shown;

[0035] Figure 10A According to some embodiments of this application, a schematic diagram of a list of failure modes is shown;

[0036] Figure 10B According to some embodiments of this application, a schematic diagram of a list of failure modes is shown;

[0037] Figure 10C According to some embodiments of this application, a schematic diagram of a list of failure modes is shown;

[0038] Figure 10D According to some embodiments of this application, a schematic diagram of a list of failure modes is shown;

[0039] Figure 10E According to some embodiments of this application, a schematic diagram of a list of failure modes is shown;

[0040] Figure 10F According to some embodiments of this application, a schematic diagram of a list of failure modes is shown;

[0041] Figure 11A According to some embodiments of this application, a schematic diagram of an interface for editing a fault mode is shown;

[0042] Figure 11B According to some embodiments of this application, a schematic diagram of an interface for adding a fault mode is shown;

[0043] Figure 12A According to some embodiments of this application, a schematic diagram of the structure of a system 10 is shown;

[0044] Figure 12BAccording to some embodiments of this application, a schematic diagram of the topology of system 10 in a topology editor is shown;

[0045] Figure 12C According to some embodiments of this application, a schematic diagram of a list of failure modes is shown. Detailed Implementation

[0046] The illustrative embodiments of this application include, but are not limited to, failure mode analysis methods, systems, electronic devices, and media.

[0047] The technical solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0048] Figure 2 According to some embodiments of this application, a schematic diagram of an application scenario for a fault mode analysis method is shown. For example... Figure 2 As shown, system 10 includes component 100, component 200, and database 300. Component 100 is deployed in a cluster configuration including a master node, comprising node servers (hereinafter referred to as nodes) 101, node 102 (master node), node 103, node 104, and node 105. Component 200 is deployed in a master-slave configuration, comprising node 201 (master node) and node 202. System 10 can access database 300 and external component 400 through component 100. To improve the reliability of system 10, the impact of component failures on system 10 can be analyzed using a chaos engineering exercise platform 600. Based on the analysis results, the weak points of system 10 can be identified, and corresponding contingency plans can be developed. However, this requires determining a list of failure modes for each component in system 10. In this embodiment, the fault mode analysis system 500 acquires parameters of components in system 10, such as the types and deployment methods of components 100 and 200, external components interacting with system 10 and their parameters, such as the parameters of external component 400, and the topology between components in system 10 and components interacting with system 10. Based on the aforementioned parameters of components in system 10, parameters of components interacting with system 10, and the aforementioned topology, it automatically generates a fault mode list for system 10. This improves the efficiency and completeness of fault mode analysis, thereby enhancing the effectiveness of the fault mode simulation results of the chaos engineering simulation platform 600 on system 10. In some embodiments, when components of system 10 change, such as when components are added to system 10, the fault mode analysis system 500 only needs to update the topology of system 10. The fault mode analysis system 500 can then automatically update the fault mode list based on the change in the topology of system 10, improving the maintenance efficiency of the fault mode list.

[0049] further, Figure 3A According to some embodiments of this application, it is shown that Figure 2A schematic diagram of the software architecture of the Fault Mode Analysis System 500. (See diagram for example.) Figure 3A As shown, the Fault Mode Analysis System 500 includes an application editor 501, a topology editor 502, and a fault mode analyzer 503.

[0050] Application Editor 501 is used to obtain parameters of the components contained in the system to be analyzed, as well as the parameters of the components that interact with the system to be analyzed. The component parameters include, but are not limited to: the component type (e.g., server type and database type), the number of nodes in the component and their deployment method (e.g., cluster, master-slave, master-standby, etc.), and the parameters of each node in the component (e.g., the IP address of each node), and the processes corresponding to the component and nodes.

[0051] It is understood that in some embodiments, the application editor 501 may further include a component library from which the user can select components that match the components in the system to be analyzed. In other embodiments, the user may also customize components based on the characteristics of the components in the system to be analyzed. This application does not impose limitations on the embodiments described herein.

[0052] The topology editor 502 is used to obtain the topological relationships between components in the system to be analyzed and components that interact with the system to be analyzed, including but not limited to network connection relationships and data access relationships (such as read and / or write).

[0053] It is understood that in some embodiments, the system to be analyzed already has a complete topology structure during the design phase. In this case, the user can obtain the topology structure of the system to be analyzed from the designer of the system and import it into the fault mode analysis system 500 through the topology editor 502. In other embodiments, the user can also construct the topology structure of the system to be analyzed through the topology editor 502 based on the connection relationships of the system to be analyzed. This application does not impose any limitations on the embodiments described.

[0054] The fault mode analyzer 503 provides the function of automatically generating fault modes. It can obtain the corresponding fault modes of the components from the fault mode library 5031 according to the component type, component parameters and topology of the system to be analyzed, and generate a fault mode list.

[0055] The following are some exemplary scenarios for generating a list of failure modes, but this application does not limit this to any particular method.

[0056] Scenario 1: Generate basic fault types for components based on their type and parameters, such as host faults, resource faults, process faults, etc.

[0057] Scenario 2: When a component has an upstream component it accesses, it generates network-related faults of its own component and faults of overloaded requests to the upstream component.

[0058] Scenario 3: When a component accesses a downstream component, a network-related fault is generated, such as a network packet loss fault or a network latency fault when accessing a downstream component.

[0059] Scenario 4: When a component uses a service component, a service failure occurs, such as database data loss.

[0060] further, Figure 3B According to some embodiments of this application, schematic diagrams of some fault modes in the fault mode library 5031 are shown. For example... Figure 3B As shown, the fault mode library 5031 includes:

[0061] Host-related failures: host offline, host restart, etc.;

[0062] Other types of failures: External request overload, etc.;

[0063] Network-related faults: Component XX cannot access Service XX, Component XX loses packets when accessing Service XX, Component XX experiences delays when accessing Service XX, Component XX returns an error when accessing Service XX, network card failure, corrupted network packets, network link packet loss, network link delay, network speed limiting, etc.

[0064] Resource-related faults: host processor overload, host memory overload, high host disk I / O (input and output), host disk full, etc.

[0065] Process-related failures: process suspension, process failure to start, process exit, etc.

[0066] Service-related failures: XX service master node failure, XX service single AZ (availability zone) failure, XX service backup node failure, XX service data loss, password expiration, etc.

[0067] Understandable. Figure 3B The fault mode types shown in the fault mode library 5031 are merely an example. In other embodiments, the fault mode library 5031 may also include other fault types, which is not limited in this application embodiment.

[0068] It is understandable that when the component types, component parameters, and topology of the system to be analyzed change, the fault mode analyzer 503 can automatically update the fault mode list according to the aforementioned changes, thereby improving the maintenance efficiency of the fault mode list.

[0069] It is understood that in some embodiments, users can also customize fault categories in the fault mode library according to the characteristics of the system to be analyzed, or add specific fault modes to the fault mode list to further improve the completeness of the fault mode list.

[0070] further, Figure 4 According to some embodiments of this application, a schematic diagram of an electronic device 700 that can be used to implement the fault mode analysis system shown in FIG3 is illustrated. Figure 4 As shown, the electronic device 700 may include one or more processors 701, system memory 702, non-volatile memory (NVM) 703, network interface 704, input / output (I / O) devices 705, and system control logic 706 for coupling the processor 701, system memory 702, non-volatile memory (NVM) 703, network interface 704, and input / output (I / O) devices 705. Wherein:

[0071] Processor 701 may include one or more single-core or multi-core processors. In some embodiments, processor 701 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, processor 701 may execute instructions for implementing a failure mode analysis method.

[0072] System memory 702 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions. For example, in some embodiments, system memory 702 can be used to temporarily store instructions for implementing a fault mode analysis method, and can also be used to temporarily store the results of fault mode analysis.

[0073] The non-volatile memory 703 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 703 may include any suitable non-volatile memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 703 may also be a removable storage medium, such as a secure digital storage (SD) card. In other embodiments, the non-volatile memory 703 may also be a network memory, accessible by the electronic device 700 via a network interface 704. The non-volatile memory 703 may be used to store instructions for implementing a fault mode analysis method, and may also be used to store the results of fault mode analysis, such as a list of fault modes.

[0074] Specifically, system memory 702 and non-volatile memory 703 may each include a temporary copy and a permanent copy of instruction 707. Instruction 707 may include, when executed by at least one of processors 701, causing electronic device 700 to implement... Figure 4 The failure mode analysis method and / or failure mode analysis system shown in Figure 12.

[0075] Network interface 704 may include a transceiver for providing a wireless communication interface for electronic device 700, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, network interface 704 may be integrated into other components of electronic device 700, such as processor 701. In some embodiments, electronic device 700 may communicate with other devices via network interface 704, for example, retrieving instructions from network memory for implementing a fault mode analysis method.

[0076] Input / output (I / O) device 705 may include a user interface that enables a user to interact with electronic device 700. For example, in some embodiments, input / output (I / O) device 705 may include a display for displaying the user interface of fault mode analysis system 500. In other embodiments, input / output (I / O) device 705 may also include input devices such as a keyboard, mouse, or touchscreen, through which the user can input the topology and / or parameters of the components of the system to be analyzed.

[0077] System control logic 706 may include any suitable interface controller to provide any suitable interface to other modules of electronic device 700. For example, in some embodiments, system control logic 706 may include one or more memory controllers to provide an interface to system memory 702 and non-volatile memory 703.

[0078] In some embodiments, at least one of the processors 701 may be packaged together with the logic of one or more controllers for system control logic 706 to form a system in package (SiP). In other embodiments, at least one of the processors 701 may also be integrated on the same chip with the logic of one or more controllers for system control logic 706 to form a system-on-chip (SoC).

[0079] It is understood that the electronic device 700 can be any electronic device capable of executing instructions to implement the failure mode analysis method, including but not limited to computers, servers, mobile phones, tablets, handheld computers, and personal digital assistants (PDAs). This application embodiment does not limit the scope of the device.

[0080] It is understood that the structure of the electronic device 700 shown in the embodiments of this application does not constitute a specific limitation on the electronic device 700. In other embodiments of this application, the electronic device 700 may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0081] To more clearly describe the technical solutions of the embodiments of this application, the following is combined with... Figure 2 The application scenarios shown in Figure 3, and the Fault Mode Analysis System 500 shown in Figure 3, Figure 4 The electronic device 700 shown will be introduced.

[0082] Figure 5 According to some embodiments of this application, it is shown that by... Figure 4 The flowchart of the fault mode analysis system shown in Figure 3 illustrates the operation of electronic device 700 during fault mode analysis. Figure 5 As shown, failure mode analysis includes the following steps:

[0083] Step S501: Obtain the components and parameters of the system to be analyzed. Electronic device 700 obtains the components and parameters of the system to be analyzed, including but not limited to the component type, deployment method, and node parameters (e.g., IP address) within the component. It is understood that in some embodiments, users can interact with electronic device 700 through its input / output devices and input components of system 10 through application editor 501.

[0084] Specifically, Figure 6 According to some embodiments of this application, an interface diagram of an application editor 501 displayed on the display of an electronic device 700 is shown. For example... Figure 6 As shown, for Figure 2 The system 10 shown may include an application editor that includes an "Add Component" control 5011, an "Edit Application" control 5012, a "Component Edit" control 5013, and a "Component Edit" control 5014.

[0085] It is understood that in some embodiments, if there are other components in the system to be analyzed, the user can add them by clicking the "Add Component" control 5011. In other embodiments, the user can modify the name of the system to be analyzed by clicking the "Edit Application" control 5012. For example, when the user clicks the "Edit Application" control 4012, the "Application Edit" control 50121 pops up, and the user can modify the name of the system to be analyzed in the "Application Edit" control 50121.

[0086] The "Component Editing" controls 5013 and 5014 correspond to components 100 and 200 in system 10, respectively. Users can modify the parameters of components 100 and 200 using the "Component Editing" controls 5013 and 5014. For example, users can modify the process names of components 100 and 200 using the "Component Editing" controls 5013 and 5014.

[0087] In some embodiments, a user can use the mouse to click the "Edit Component" control 50131 within the "Component Edit" control 5013 to edit the properties of component 100. It can be understood that when the user clicks the "Edit Component" control 50131, the application editor 501 pops up the "Edit Component" window 501311, as shown below. Figure 7A As shown. Users can determine the parameters of component 100 based on its deployment method in system 10. For example, the deployment method of component 100 can be set to "cluster (with master node)", and IP addresses 1.1.1.1 to 5.5.5.5 can be assigned to nodes 101 to 105, with node 102 as the master node. Figure 7AAs shown in the “Edit Components” window 501311, the IP address 2.2.2.2 of node 102 is at the very beginning of the IP list, meaning that node 102 is the master node.

[0088] In other embodiments, the user can use the mouse to click the "Edit Component" control 50141 within the "Component Edit" control 5014 to edit the properties of component 200. It can be understood that when the user clicks the "Edit Component" control 50141, the application editor 501 pops up the "Edit Component" window 501411. For example... Figure 7B As shown. Users can determine the parameters of component 200 based on the deployment method of component 200 in system 10. For example, they can determine the parameters based on... Figure 2 The parameters of component 200 in system 10 are set to "primary / backup" deployment mode, and IP addresses 6.6.6.6 to 7.7.7.7 are assigned to nodes 201 and 202, with node 202 as the primary node. Figure 7B As shown in the “Edit Components” window 501411, the IP address of node 201, 6.6.6.6, is at the very beginning of the IP list, meaning that node 202 is the master node.

[0089] It is understandable that after the user determines the parameters of the components in System 10, they can exit the application editor 501.

[0090] Step S502: Obtain the external components that interact with the system under analysis and their parameters. For example, in some embodiments, the user can add components that interact with the system under analysis through the topology editor 502 of the fault mode analysis system. Specifically, for Figure 2 The system 10 shown includes components that interact with it, such as external components 400 and service components like the database 300. Figure 8 As shown, users can add external components 400 and database 300 in the topology editor 502.

[0091] It is understood that the component library of the fault mode analysis system 500 contains pre-built models of external components and service components (such as the database 300). Users can select the corresponding components from the component library and add them to the topology editor 502 of the fault mode analysis system 500. In some other embodiments, users can also customize the parameters of external components in the topology editor 502, which is not limited in this application embodiment.

[0092] Step S503: Obtain the topology of the components and external components of the system to be analyzed. For example, in some embodiments, the user can edit the topology of the system to be analyzed through the topology editor 502 of the Failure Mode and Effects Analysis system 500. Specifically, Figure 9 The topology of system 10 is shown, such as Figure 9 As shown, external component 400 is coupled to component 100, database 300 is coupled to component 100, and component 100 is coupled to component 200, with the coupling method being that component 100 writes data to component 200.

[0093] It is understood that in some embodiments, users can modify the coupling method between components by double-clicking the connection line between components, including but not limited to "read", "write", "read and write", etc. In other embodiments, users can also modify the coupling relationship between components in other ways, which is not limited in the embodiments of this application.

[0094] Step S504: Determine the fault mode list based on the parameters of the components in the system to be analyzed, the parameters of the external components, and the topology. For example, the fault mode analyzer 503 of the fault mode analysis system 500 obtains the corresponding fault modes from the fault mode library 5031 based on the parameters of the components of the system 10 obtained above, the external components 400 and database 300 that interact with the system 10, and the topology between components 100, 200, database 300, and external components 400, and generates a fault mode list.

[0095] It is understood that in some embodiments, before generating the list of failure modes, the failure mode analyzer 503 can also detect the integrity of the system 10 topology, i.e., whether there are any unconnected components. If there are unconnected components, the failure mode analyzer 503 can prompt the user to complete the topology.

[0096] The following exemplifies the process by which the fault mode analyzer 503 generates lists of different categories of fault modes in system 10 under different circumstances.

[0097] First, the fault mode analyzer 503 generates basic fault types for the components based on the types and parameters of the components in the system 10, such as host faults, resource faults, and process faults.

[0098] For example, in some embodiments, the fault mode analyzer 503 generates host-class and resource-class faults in the basic fault categories of components 100 and 200 based on the type and deployment method of components 100 and 200. In some embodiments, host-class faults may include off-site failures, and resource-class faults may include host processor overload faults.

[0099] Figure 10A According to some embodiments of this application, a schematic diagram of host class and resource class fault lists for components 100 and 200 of system 10 is shown. Figure 10AAs shown, component 100 is a cluster deployment with 5 nodes. Host offline failures include 20% node offline, 50% node offline, and 80% node offline. Resource failures include 20% node host processor overload, 50% node host processor overload, and 80% node host processor overload. Component 200 is a master-slave deployment with 2 nodes. Host failures include both the master and slave nodes being offline. Resource failures include both the master and slave nodes experiencing processor overload. It is understood that for cluster-deployed components, such as component 100, the division of host failures and resource failures based on 20%, 50%, and 80% of nodes is merely an example. Other ratios may be used in other embodiments, and this application does not limit the scope of such divisions.

[0100] It is understandable that host-related failures and resource-related failures can also include other failures. For example, host-related failures can also include host restarts, and resource-related failures can also include node host memory overload, node disk overload, etc., which will not be elaborated here.

[0101] In other embodiments, the fault mode analyzer 503 generates process-type faults for components 100 and 200 based on process parameters of components 100 and 200. In some embodiments, process-type faults may include process suspension. Figure 10B According to some embodiments of this application, a schematic diagram of a process-type fault list for system 10 is shown. For example... Figure 10B As shown, component 100 is a cluster deployment with 5 nodes, the process name is Pid_A, and process suspension failures include 20% of nodes' Pid_A processes being suspended, 50% of nodes' Pid_A processes being suspended, and 80% of nodes' Pid_A processes being suspended; component 200 is a master-slave deployment with 2 nodes, the process name is Pid_B, and process suspension failures include the master node's Pid_B process being suspended and the slave node's Pid_B process being suspended. It is understood that for cluster-deployed components, such as component 100, dividing the process failures into 20%, 50%, and 80% of nodes is merely an example; other ratios may be used in other embodiments, and this application does not limit this.

[0102] It is understandable that process-related failures can also include other failures, such as a process failing to start or a process exiting, which will not be elaborated here.

[0103] Secondly, the fault mode analyzer 503 generates network-related faults of the component itself and overload faults of the upstream component request based on the upstream component accessed by the component in system 10.

[0104] For example, the fault mode analyzer 503 generates network-related faults and other request overload faults for components 100 and 200 themselves, based on the parameters of components 100 and 200 and the upstream components accessed. In some embodiments, network-related faults may include network interface card (NIC) faults. Figure 10C According to some embodiments of this application, a schematic diagram showing a list of network-related faults and request overload faults of components in System 1 is illustrated. For example... Figure 10C As shown, component 100 is a cluster deployment with 5 nodes and has an upstream component (external component 400) for access. Network interface card (NIC) failures in component 100 include 20% node NIC failure, 50% node NIC failure, and 80% node NIC failure. Request overload failures in component 100 include request overload in external component 400. Component 200 is a master-slave deployment with 2 nodes and has an upstream component (component 100) for access. NIC failures in component 200 include master node NIC failure and slave node NIC failure. Request overload failures include request overload in component 100. It is understood that for cluster-deployed components, such as component 100, dividing the network interface card failures into 20%, 50%, and 80% node failures is merely an example. Other ratios may be used in other embodiments, and this application does not limit the scope of the embodiments.

[0105] It is understood that network-related faults when accessing upstream components may also include other faults, such as network rate limiting, corrupted network data packets, etc., which will not be elaborated upon in the embodiments of this application.

[0106] Furthermore, the fault mode analyzer 503 generates network-type faults based on the downstream components accessed by the components in system 10.

[0107] The fault mode analyzer 503 generates network-related faults, including network outage faults and access packet loss faults, based on the downstream components accessed by components 100 and 200. Figure 10D According to some embodiments of this application, a schematic diagram of the network failure and access packet loss failure list of system 10 is shown. As shown in Figure 10, in system 10, component 100 has access to downstream components, component 200 and database 300. The network failure of component 100 includes component 100's inability to access component 200 and component 100's inability to access database 300. The access packet loss failure of component 100 includes packet loss when component 100 accesses component 200 and packet loss when component 100 accesses database 300.

[0108] It is understood that network failures when accessing downstream components may include other failures, such as network rate limiting, network link damage, and network packet data corruption, which will not be elaborated upon in the embodiments of this application.

[0109] Finally, the fault mode analyzer 503 generates service-type faults based on the usage of service-type components by the components in system 10.

[0110] For example, in some embodiments, component 100 of system 10 may access service-type component database 300, that is, it may use database services, and database 300 may be a MySQL database deployed as a primary or backup. Figure 10E According to some embodiments of this application, a schematic diagram of the fault mode list of the database 300 in system 10 is shown, such as... Figure 10E As shown, database 300 service-related failures include primary node failure, backup node failure, data loss, and password expiration.

[0111] It is understood that for different service components, such as different types of databases, the service failures included may be different, and may include more types of failures. This application embodiment does not limit this.

[0112] further, Figure 10F According to some embodiments of this application, a schematic diagram of a list of failure modes obtained by performing failure mode analysis on system 10 using failure mode analysis system 500 is shown. For example... Figure 10F As shown, System 10 includes a total of 94 faults (only the first page of the fault mode list is shown in the figure). The fault mode list includes the serial number, component, fault type, exercise summary, fault name, injection method, fault level, fault flag, whether it was exercised, the most recent exercise time, whether it is enabled, fault source, and the operation on the fault (including editing, disabling, deleting, editing exercise, etc.). Among them:

[0113] The exercise log is a record of the exercise conducted using the Chaos Engineering Exercise Platform 600 for the corresponding fault item. It can be added by the user or automatically generated by the Chaos Engineering Exercise Platform 600. It is understandable that the exercise log will be empty for fault items that have not undergone an exercise.

[0114] The injection method refers to the method used when components of the virtual system 10 experience corresponding failures during a simulation of fault items in the system 10 by the chaos engineering simulation platform 600. It is understood that the injection method can vary depending on the chaos engineering simulation platform 600. It may be pre-set in the fault mode library 5031 of the fault mode analysis system 500 according to different fault types. In other embodiments, users can also modify it as needed, for example, by modifying the injection method list for different types of faults provided by the chaos engineering simulation platform 600.

[0115] Fault marking refers to the marking of the characteristics of a fault, including but not limited to being monitorable, self-healing, and manually recoverable. It is understood that in some embodiments, users can modify the fault mode markings as needed, and this application does not limit this.

[0116] The operations include deciding whether to conduct a drill for the current fault item, editing the fault, deleting the fault, and editing the fault drill. For example, editing the drill summary. For instance, when the user clicks the "Edit" control 1001 for fault item number 1, the editing page for fault item number 1 pops up, as shown in 11A. The user can edit the drill summary, fault level, whether to conduct a drill, fault injection method, etc. for fault item number 1.

[0117] It is understood that in some embodiments, users can also manually add fault items as needed. For example, users can add fault items on their mobile phones by clicking the "Add Fault" control 1002. Figure 11B According to some embodiments of this application, a schematic diagram of the interface for users to manually add fault items is shown. Users can enter the fault name, corresponding component, fault category, whether it is a drill, fault level, injection method, fault flag, etc. on the "Add Fault" page to complete the addition of fault items.

[0118] It is understood that in some embodiments, users can export a list of failure modes of system 10 so that the chaos engineering exercise platform 600 can use the aforementioned list of failure modes to conduct exercises, thereby analyzing the weaknesses of system 10 and developing solutions. For example, in some embodiments, users can store the list of failure modes of system 10 in the non-volatile memory 703 of the aforementioned electronic device 700 by clicking the "Export Fault" control 1003.

[0119] In this way, the Fault Mode Analysis System 500 can analyze the list of fault modes of System 10 by using the topology and component parameters of System 10, which improves the efficiency of the analysis. Furthermore, users only need to input the topology and parameters of each component of System 10 according to the actual structure of System 10, without relying on the user's experience in fault mode analysis, which lowers the threshold for conducting fault mode analysis and avoids the influence of the user's experience on the validity of the fault mode analysis results.

[0120] Specifically, when the topology of System 10 changes, the user can make corresponding modifications in the topology editor of Fault Mode Analysis System 500 to automatically update the fault mode list. For example, as Figure 12A As shown, a database 800 has been added to system 10 for access by component 200. At this time, the user adds database 800 in the topology editor of the fault mode analysis system 500, as follows: Figure 12BAs shown, the user adds database 800 to the topology of system 10 and establishes the topology relationship between database 800 component 200 and the network cable. It can be understood that when the user clicks the "Save and Update Fault Modes" control 1201, the fault mode analysis system 500 can update the fault mode list of system 10. For example, Figure 12C According to some embodiments of this application, a fault mode analyzer 503 of a fault mode analysis system 500 is shown as generating a fault list of network-related faults based on downstream components accessed by component 100 and component 200 in system 10. (Comparison) Figure 12C and Figure 10D As can be seen, after adding database 800, such as Figure 12C As indicated by the dashed box, two new fault items have been added: "Component 200 fails to access database 800" and "Component 200 loses packets when accessing database 800". It's understandable that after adding database 800, corresponding service-related faults will be automatically updated after the user updates the topology of System 10, and will not be elaborated further here.

[0121] Thus, when the topology of system 10 changes, the maintenance of the fault mode list of system 10 can be achieved by updating the topology of system 10 in the fault mode analysis system 500, thereby improving the maintenance efficiency of the fault mode list.

[0122] It is understood that in some other embodiments, the fault mode analysis system 500 can also be integrated with the chaos engineering exercise platform 600 in the same system. In this case, the fault mode list obtained by the fault mode analysis system 500 can be directly used by the chaos engineering exercise platform 600 to conduct exercises and further analyze the efficiency of the weak links of the system 10.

[0123] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0124] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0125] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0126] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0127] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0128] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0129] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0130] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A failure mode analysis method, characterized in that, The method includes: Obtain the components of the system to be analyzed and the parameters of the components of the system to be analyzed; Obtain the external components that interact with the components of the system to be analyzed, and the parameters of the external components that interact with the system. Obtain the components of the system to be analyzed and the topology between the external components; Based on the parameters of the components of the system to be analyzed, the parameters of the external components with interaction, and the topology, the corresponding fault modes are obtained from the preset fault mode library to generate a fault mode list of the system to be analyzed. The fault mode list is used to conduct fault drills on the system to be analyzed in chaos engineering. The step of obtaining corresponding fault modes from a pre-set fault mode library based on the parameters of the components of the system to be analyzed, the parameters of the external components with interaction, and the topology, and generating a fault mode list of the system to be analyzed, includes at least one of the following methods: Based on the parameters of the component, basic fault types are generated, including host fault types, resource fault types, and process fault types. Based on the upstream components accessed by the aforementioned components, network-related faults and other types of faults are generated; Based on the downstream components accessed by the aforementioned component, generate network-related faults; Based on the service-type components accessed by the aforementioned components, service-type faults are generated.

2. The failure mode analysis method according to claim 1, characterized in that, The parameters of the components of the system to be analyzed include at least one of the following parameters: the type of the component, the number of nodes included in the component, the deployment method of the nodes in the component, the IP address corresponding to the component and the component node, and the process corresponding to the component.

3. The failure mode analysis method according to claim 1, characterized in that, The topology includes at least one of the following: The upstream components accessed by the components of the system to be analyzed and the access methods therein; The downstream components accessed by the components of the system to be analyzed and the access methods therein; The service components and access methods accessed by the components of the system to be analyzed.

4. The failure mode analysis method according to claim 1, characterized in that, The method further includes: Obtain the parameter changes of the system to be analyzed; The fault mode list is updated based on the changes in the parameters.

5. The failure mode analysis method according to claim 4, characterized in that, The parameter change includes at least one of the following: The changes in the parameters of the components of the system to be analyzed; The changes in the parameters of the interactive external components; The change in topology.

6. The failure mode analysis method according to claim 1, characterized in that, The host-related faults include host offline or host restart.

7. The failure mode analysis method according to claim 1, characterized in that, The process-related faults include at least one of the following: process suspension, process failure to start, and process exit.

8. The failure mode analysis method according to claim 1, characterized in that, The resource-related faults include at least one of the following: host processor overload, host memory overload, host disk I / O high, and host disk full.

9. The failure mode analysis method according to claim 1, characterized in that, The network-related faults include at least one of the following: component access failure, component access packet loss, component access delay, component access error return, network card failure, network packet corruption, network link packet loss, network link delay, and network speed limit.

10. The failure mode analysis method according to claim 1, characterized in that, Other types of failures include external request overload.

11. The failure mode analysis method according to claim 1, characterized in that, The service-related failures include at least one of the following: service master node failure, service backup node failure, service availability zone failure, service data loss, and password expiration.

12. A fault mode analysis system, characterized in that, The system includes: an application editor, a topology editor, and a fault mode analyzer; wherein: The application editor is used to obtain the components of the system to be analyzed and the parameters of the components; The topology editor is used to obtain the topology between the components of the system to be analyzed and the external components that interact with the system to be analyzed, as obtained by the application editor. The fault mode analyzer includes a fault mode library, and the fault mode analyzer is used to obtain the corresponding fault mode from the fault mode library according to the parameters of the component, the parameters of the external component with interaction and the topology, and generate a fault mode list of the system to be analyzed. The fault mode analyzer is used to retrieve corresponding fault modes from the fault mode library based on the parameters of the component, the parameters of the interacting external component, and the topology, and generate a fault mode list of the system to be analyzed, including: The failure mode analyzer is used to generate a list of failure modes of the system to be analyzed based on at least one of the following methods: Based on the parameters of the component, basic fault types are generated, including host fault types, resource fault types, and process fault types. Based on the upstream components accessed by the aforementioned components, network-related faults and other types of faults are generated; Based on the downstream components accessed by the aforementioned component, generate network-related faults; Based on the service-type components accessed by the aforementioned components, service-type faults are generated.

13. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the fault mode analysis method according to any one of claims 1 to 11.

14. A program product, characterized in that, The program product includes a computer program or program code that, when executed by at least one processor, implements the fault mode analysis method according to any one of claims 1 to 11.

15. An electronic device, characterized in that, include: Memory is used to store instructions executed by one or more processors of an electronic device; And a processor, one of the processors of the electronic device, for executing instructions stored in the memory to implement the fault mode analysis method according to any one of claims 1 to 11.

Images