Fault drilling method, device and system, medium and equipment

Abstract

The invention provides a fault drilling method, device and system, a medium and equipment, and the method comprises the steps: obtaining at least one piece of alarm data of the system based on a preset period; determining whether fault alarm data exists in the at least one piece of alarm data, wherein the fault alarm data is alarm data with a potential fault problem; in response to the existence of the at least one piece of fault alarm data, the fault type of at least one potential fault corresponding to the alarm type of the at least one piece of fault alarm data is obtained from a causal knowledge base, and the incidence relation between the alarm type and the fault type is stored in the causal knowledge base; generating a fault drilling case based on the fault type of the at least one potential fault and a drilling material library; and based on the fault drilling case, executing a fault drilling operation on the system. According to the technical scheme, targeted fault drilling can be performed on the system, missing of problem points with potential fault risks is avoided, and drilling resource consumption can be remarkably reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of system operation and maintenance management, and in particular, to a fault drill method, apparatus, system, medium, and device. Background Art

[0002] With the development and popularization of various architectures of systems (such as microservice architecture, distributed architecture), the complexity of the services provided by the system and the dependencies between services have also increased accordingly. When a fault occurs in a certain service or node in the system, it often affects the operation of other services in the service link, and may even lead to system paralysis. In order to improve the reliability and stability of the system to cope with various possible fault scenarios, chaos engineering has emerged.

[0003] Chaos engineering refers to a technical means of introducing abnormal states of software or hardware into the system through fault drills, observing the system behavior, and determining optimization strategies to improve the fault tolerance and recoverability of the system. Under related technologies, the method of conducting a full-scale fault drill on the system requires a large amount of machine resources and human resources, resulting in high costs. If only partial-dimensional fault drills are carried out, key risk points may be missed, and the stability and reliability of the entire system cannot be ensured. Summary of the Invention

[0004] To solve the above technical problems, embodiments of the present disclosure provide a fault drill method, apparatus, system, medium, and device.

[0005] According to a first aspect of the embodiments of the present disclosure, there is provided a fault drill method, including:

[0006] Based on a preset period, obtaining at least one piece of alarm data of the system, where the alarm data includes an alarm type;

[0007] Determining whether there is fault alarm data in the at least one piece of alarm data, where the fault alarm data is alarm data with potential fault problems;

[0008] In response to the existence of at least one piece of fault alarm data, obtaining at least one fault type of potential faults corresponding to the alarm type of the at least one piece of fault alarm data from a causal knowledge base, where the causal knowledge base stores the association relationship between the alarm type and the fault type;

[0009] Based on the fault type of the at least one potential fault and a drill material library, generating a fault drill use case, where the drill material library stores fault drill schemes corresponding to different fault types, and the fault drill use case includes at least one fault drill scheme;

[0010] Based on the fault drill use case, performing a fault drill operation on the system.

[0011] According to a second aspect of the embodiments of the present disclosure, a fault drill device is provided, including:

[0012] A first acquisition module, configured to acquire at least one piece of alarm data of the system based on a preset period, where the alarm data includes an alarm type;

[0013] A first determination module, configured to determine whether there is fault alarm data in the at least one piece of alarm data, where the fault alarm data is alarm data with potential fault problems;

[0014] A second acquisition module, configured to, in response to the existence of at least one piece of fault alarm data, acquire at least one fault type of potential faults corresponding to the alarm type of the at least one piece of fault alarm data from a causal knowledge base, where the causal knowledge base stores an association relationship between an alarm type and a fault type;

[0015] A use case generation module, configured to generate fault drill use cases based on the fault types of the at least one potential fault and a drill material library, where the drill material library stores fault drill schemes corresponding to different fault types, and the fault drill use cases include at least one fault drill scheme;

[0016] A fault drill module, configured to perform a fault drill operation on the system based on the fault drill use cases.

[0017] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, where the storage medium stores computer program instructions, and when the computer program instructions are executed, the above-mentioned fault drill method is implemented.

[0018] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, where the electronic device includes:

[0019] A memory, configured to store a computer program product;

[0020] A processor, configured to execute the computer program product stored in the memory, and when the computer program product is executed, the above-mentioned fault drill method is implemented.

[0021] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including computer program instructions, and when the computer program instructions are executed by a processor, the above-mentioned fault drill method is implemented.

[0022] Based on the above embodiments of the present disclosure, in view of the scenario of simulating various faults and abnormal conditions of the system through fault drills, at least one piece of alarm data (the alarm data includes alarm types) of the system is obtained based on a preset period; it is determined whether there is fault alarm data among the at least one piece of alarm data; in response to the existence of at least one piece of fault alarm data, at least one fault type of at least one potential fault corresponding to the alarm type of the at least one piece of fault alarm data is obtained from the cause-and-effect knowledge base, and the association relationship between the alarm type and the fault type is stored in the cause-and-effect knowledge base; based on the fault types of the at least one potential fault and the drill material library, a fault drill use case is generated; based on the fault drill use case, a fault drill operation is performed on the system. Thus, the technical solution of the present disclosure determines the fault alarm data with potential fault problems by analyzing the received alarm data, and then generates a fault drill use case according to the association relationship between the alarm type and the fault type in the cause-and-effect knowledge base, which helps to implement targeted fault drills on the system; in addition, the technical solution of the present disclosure determines the fault drill use case according to the alarm data received at a preset period, which helps to periodically identify the existing potential faults. The fault drill use cases formulated for the potential faults can ensure the comprehensiveness of the drills, avoid missing the problem points with potential fault risks, improve the stability and reliability of the system, and can significantly reduce the resource consumption of the drills and avoid resource waste.

[0023] The technical solution of the present disclosure will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By describing the embodiments of the present disclosure in more detail in conjunction with the drawings, the above and other objects, features, and advantages of the present disclosure will become more obvious. The drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation to the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 It is a flowchart of an embodiment of the fault drill method of the present disclosure;

[0026] Figure 2 It is a flowchart of step 102 of the fault drill method of the present disclosure;

[0027] Figure 3 It is a flowchart of generating and updating the cause-and-effect knowledge base in the fault drill method of the present disclosure;

[0028] Figure 4 It is a flowchart of step 104 of the fault drill method of the present disclosure;

[0029] Figure 5 It is a flowchart of another embodiment of the fault drill method of the present disclosure;

[0030] Figure 6 Structural schematic diagram of an embodiment of the fault drill device of the present disclosure;

[0031] Figure 7 Structural schematic diagram of another embodiment of the fault drill device of the present disclosure;

[0032] Figure 8 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. Detailed implementation manners

[0033] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0034] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0035] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they indicate an inevitable logical order between them.

[0036] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0037] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, without clear limitation or contrary indication in the context, it is generally understood as one or more.

[0038] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the associated objects before and after.

[0039] It should also be understood that the present disclosure emphasizes the differences between the various embodiments, and the same or similar parts can be referred to each other. For the sake of brevity, they will not be described in detail one by one.

[0040] At the same time, it should be understood that for the sake of convenience of description, the dimensions of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0041] The following description of at least one exemplary embodiment is merely illustrative and is in no way a limitation on the present disclosure, its application, or its use.

[0042] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and devices should be considered as part of the specification.

[0043] It should be noted that like reference numerals and letters denote like items in the following figures, and thus, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] Embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with numerous other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and so on.

[0045] Terminal devices, computer systems, servers, and other electronic devices can be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0046] Overview of the present disclosure

[0047] In the process of implementing the present disclosure, the inventors found through research that when conducting fault drills on a system, if a full-scale fault drill is performed on the system, a large amount of machine resources and human resources are required, and the cost is relatively high. However, if only partial-dimensional fault drills are performed, key risk points may be missed, and the stability and reliability of the entire system cannot be ensured.

[0048] In the fault drill method provided by the technical solution of the present disclosure, alarm data of the system can be periodically obtained, and fault alarm data with potential fault problems can be analyzed from the alarm data. Then, according to the association relationship between the alarm type and the fault type in the causal knowledge base, a fault drill case is generated to perform a targeted fault drill on the system. The fault drill case formulated for potential faults can ensure the comprehensiveness of the drill, avoid missing problem points with potential fault risks, improve the stability and reliability of the system, and significantly reduce the resource consumption of the drill, avoiding resource waste.

[0049] Exemplary method

[0050] Figure 1 It is a flowchart of an embodiment of the fault drill method of the present disclosure; this fault drill method can be applied to terminal devices (such as computer systems, servers), as Figure 1 shown, this fault drill method includes steps 101 - 105. Each step will be described separately below.

[0051] In step 101, based on a preset period, at least one piece of alarm data of the system is obtained, and the alarm data includes the alarm type.

[0052] Among them, the preset period is used to indicate the period for performing a fault drill on the system based on the alarm data of the system. Since the system complexities and risk levels of different systems are different, the periods for performing fault drills are usually different, and the preset period can be determined according to the actual scenario.

[0053] Among them, the alarm data may include information such as the alarm time, alarm type, and alarm content. The alarm time is the time point when the alarm occurs; the alarm type is used to indicate the location where the abnormality occurs. For example, database alarms, resource alarms, etc. The alarm type can be identified according to the status code. For example, an alarm with a status code of 499 represents an alarm type of request timeout; the alarm content is used to describe the specific information of the alarm. For example, an alarm for database query response delay, an alarm for too many database connections, an alarm for too high server memory usage, etc.

[0054] In this embodiment, through a preset system, such as a chaos system (kechaos system), alarm data is collected, and after preprocessing such as cleaning and formatting the alarm data, the alarm data can be stored in a preset location. For example, the alarm data is stored in a database system, such as a SQL Server database (relational database) system; or stored in a file system; or stored in a message queue, or cached in memory first. The terminal device can obtain at least one piece of alarm data from the location where the alarm data is stored according to the preset period.

[0055] Exemplarily, based on a preset period, at least one piece of alarm data that occurred within the current period of the system can be obtained. For example, if the preset period is 3 hours, then every 3 hours, the alarm data that occurred within the 3 hours before the current time point can be obtained.

[0056] In step 102, it is determined whether there is any fault alarm data among the at least one piece of alarm data, and the fault alarm data is alarm data with potential fault problems.

[0057] Among them, the potential fault means that if no preventive maintenance or adjustment measures are taken and it continues to be used, a system fault may occur. The fault alarm data with potential fault problems refers to the alarm data that appears before the actual fault of the system occurs. For example, if alarm data of database query response delay is frequently received, there may be a potential fault of database server overload.

[0058] In some embodiments, according to the alarm type corresponding to the alarm data, it can be determined whether the alarm data is fault alarm data with potential fault problems. For example, if the alarm type is database server downtime, it indicates that the alarm data is fault alarm data with potential fault problems.

[0059] In this implementation, the alarm types with potential fault problems can be preset and stored in advance, so that when the alarm data is obtained subsequently, it can be determined whether the alarm data is fault alarm data with potential fault problems according to the alarm type of the alarm data.

[0060] In some other embodiments, it can also be determined whether the alarm data is fault alarm data with potential fault problems according to the change trend of the alarm data. For example, among the alarm data obtained in the current period, if the number of alarms with the alarm type of connection timeout shows an abnormal upward trend, then according to this abnormal upward trend, it can be determined that the alarm of connection timeout is fault alarm data with potential fault problems.

[0061] Furthermore, if it is determined that there is at least one piece of fault alarm data, the fault type of the potential fault can be determined through step 103.

[0062] In step 103, in response to the existence of at least one piece of fault alarm data, at least one fault type of potential faults corresponding to the alarm type of the at least one piece of fault alarm data is obtained from the causal knowledge base, and the causal knowledge base stores the association relationship between the alarm type and the fault type.

[0063] Among them, the causal knowledge base is a storage structure for storing the association relationship between the alarm type and the fault type, and can be a database table. The association relationship between the alarm type and the fault type is obtained by analyzing historical alarm data and historical fault data. For details, please refer toFigure 3 The embodiments shown are not described in detail here for the time being.

[0064] In this embodiment, the association relationship between the alarm type and the fault type is used to characterize that the alarm of a certain alarm type is the cause of the fault associated with it occurring in the system. For example, if there is an association relationship between alarm type A and fault type A', it means that the alarm of alarm type A is the cause of the fault of fault type A' occurring in the system, and there is a causal association relationship between alarm type A and fault type A'.

[0065] In this embodiment, if there is at least one piece of fault alarm data and the alarm types of the at least one piece of fault alarm data are different, potential faults corresponding to different alarm types can be determined separately from the causal knowledge base. For example, the alarm types of at least one piece of fault alarm data are alarm type A and alarm type B respectively, and the corresponding fault types of potential faults that may be determined are A' and B'.

[0066] In step 104, based on the fault types of at least one potential fault and the drill material library, fault drill cases are generated.

[0067] Among them, the drill material library stores fault drill schemes corresponding to different fault types, and the fault drill cases include at least one fault drill scheme. Each fault drill scheme includes a fault scenario and a drill strategy. The drill strategy refers to the strategy that can quickly recover the fault when the fault occurs, and the fault scenario refers to the abnormal situation or problem corresponding to the fault. The fault scenario can include multiple types, such as single product fault, power failure and network disconnection, hardware fault, overload type fault, dependency type fault, etc.

[0068] In this embodiment, if there is only one potential fault in the current cycle, one fault drill case can be generated according to the fault drill scheme corresponding to the potential fault; if there are multiple potential faults in the current cycle, one fault drill case can be generated according to the multiple fault drill schemes corresponding to the multiple potential faults.

[0069] In this embodiment, the drill material library can be generated based on all possible risk nodes obtained by systematically analyzing the components of the system. The risk nodes include but are not limited to key components such as MySQL, Redis cache, Kafka message queue, RocketMQ message queue, and servers. The drill material library contains the fault scenarios and drill strategies of all possible fault types in the system.

[0070] In step 105, based on the fault drill cases, a fault drill operation is performed on the system.

[0071] In this embodiment, based on the fault scenarios in the fault drill cases, the test environment can be accurately drilled, fault injection can be simulated, and corresponding response measures can be taken according to the drill strategy to implement the fault drill operation on the system.

[0072] Through the above steps 101 - 105, for the scenarios of simulating various faults and abnormal conditions of the system through fault drills, by obtaining at least one piece of alarm data (the alarm data includes alarm types) of the system based on a preset period; determining whether there is fault alarm data in the at least one piece of alarm data; in response to the existence of at least one piece of fault alarm data, obtaining at least one fault type of at least one potential fault corresponding to the alarm type of the at least one piece of fault alarm data from the causal knowledge base, where the association relationship between the alarm type and the fault type is stored in the causal knowledge base; generating fault drill cases based on the at least one fault type of the potential fault and the drill material library; and performing a fault drill operation on the system based on the fault drill cases. Thus, the technical solution of the present disclosure analyzes the received alarm data to determine the fault alarm data with potential fault problems, and then generates fault drill cases according to the association relationship between the alarm type and the fault type in the causal knowledge base, which helps to implement targeted fault drills on the system; in addition, the technical solution of the present disclosure determines the fault drill cases according to the alarm data received at a preset period, which helps to periodically identify the existing potential faults. The fault drill cases formulated for the potential faults can ensure the comprehensiveness of the drills, avoid missing the problem points with potential fault risks, improve the stability and reliability of the system, and can significantly reduce the resource consumption of the drills and avoid resource waste.

[0073] Figure 2 It is a flowchart of step 102 of the fault drill method of the present disclosure. In the embodiments of the present disclosure, taking how to determine the fault alarm data with potential fault problems as an example for illustrative purposes, on the basis of the above Figure 1 shown embodiments, step 102 includes steps 121 - 123. Each step will be described separately below.

[0074] In step 121, based on the alarm types corresponding to each piece of alarm data in the at least one piece of alarm data, clustering is performed on the at least one piece of alarm data to obtain at least one alarm instance corresponding to at least one alarm type, and each alarm instance includes at least one piece of classified alarm data with the same alarm type.

[0075] In this embodiment, clustering the alarm data means clustering the alarm data with the same alarm type into one alarm instance, thereby obtaining at least one alarm instance.

[0076] In step 122, using the autoregressive integrated moving average model, based on at least one piece of classified alarm data in each alarm instance, the change trend of the classified alarm data of each alarm type is obtained.

[0077] Among them, the Autoregressive Integrated Moving Average Model (ARIMA for short) is a time series prediction model that can analyze the alarm data (alarm time, alarm type) in each alarm instance, identify the patterns, trends, and seasonal factors in the alarm data, and obtain the change trends of each classified alarm data.

[0078] Among them, the pattern in the alarm data refers to the periodic changes, peaks, and valleys of the alarm data; the trend refers to the trend of the alarm data in the current period, such as rising, falling, or stable; the seasonal factor refers to whether the change of the alarm data conforms to the fluctuations of holidays and annual cycles, and the trends in the off-season and peak season are different.

[0079] In step 123, based on the change trends of the classified alarm data of each alarm type, determine the classified alarm data with potential fault problems in at least one piece of alarm data.

[0080] Among them, the classified alarm data with potential fault problems is the fault alarm data with potential fault problems. According to the change trend of the alarm data, it can be determined whether the alarm data may lead to the occurrence of potential faults. For example, if the change trend is stable and basically unchanged, then the alarm data is very likely not to lead to the occurrence of potential faults. If the change trend is rising, then the alarm data may lead to the occurrence of potential faults.

[0081] Exemplarily, for the alarm data of alarm type A, its change trend is stable. For example, it alarms 1-2 times per hour, which can indicate that the alarm data of alarm type A is stable and will not lead to potential faults; if the alarm data of alarm type B has an upward change trend, it may indicate that the data of alarm type A is alarm data with potential fault problems.

[0082] Through the above steps 121-step 123, an implementation method for determining whether the alarm data is alarm data with potential fault problems according to the change trend of the alarm data is disclosed. By using the time series prediction model, the change trend of the alarm data of each alarm type in this period can be identified, and then the alarm data with potential fault problems can be determined, which helps to identify the risk points in the system.

[0083] In some other alternative implementation manners, the alarm data with potential fault problems can also be directly determined according to the alarm type of the alarm data. For example, the alarm type of poor sensor contact usually directly causes the occurrence of faults. Therefore, some alarm data with potential fault problems can be directly determined according to the alarm type.

[0084] Figure 3 This is a flowchart for generating and updating a cause-and-effect knowledge base in the fault drill method of the present disclosure. The embodiments of the present disclosure take how to generate / update the cause-and-effect knowledge base as an example for illustrative explanation, including steps 301-step 303. Each step will be described separately below.

[0085] In step 301, obtain the fault data corresponding to at least one historical fault and the alarm data corresponding to at least one historical alarm before each historical fault occurs. The fault data includes the fault type.

[0086] Among them, the historical fault is a fault that occurred in the system during a certain period in the past. The fault data of the historical fault includes information such as the fault time, fault type, and fault content. The fault time is the time point when the fault occurred; the fault type is used to indicate the location where the abnormality occurred. For example, hard disk or disk damage faults, file system faults, memory faults, central processing unit faults, graphics card faults, etc.; the fault content is used to describe the specific information of the fault, such as data loss, system loading failure, etc. The historical alarm before the fault occurs is an alarm before the fault that occurred in the system during a certain period in the past. For example, the alarm within 3 hours before the historical fault is the historical alarm corresponding to the historical fault.

[0087] In this embodiment, the historical faults, the corresponding fault data, and the alarm data corresponding to at least one historical alarm before each historical fault occurs can be obtained at regular time intervals, for example, every 3 months.

[0088] In this embodiment, the fault data corresponding to the historical fault can be obtained at a preset storage location.

[0089] In step 302, using the transition probability matrix of the Markov chain, based on the fault data corresponding to at least one historical fault and the alarm data corresponding to at least one historical alarm before each historical fault occurs, determine the association relationship between the alarm type and the fault type.

[0090] Among them, the Markov chain is a stochastic process with Markov properties in probability theory and mathematical statistics, and is applicable to discrete exponential sets and state spaces. The states in the state space of the Markov chain can be defined as the alarm type and the fault type. According to the fault type of the historical fault and the alarm type of the historical alarm that occurred before the historical fault, construct a transition probability matrix, analyze the transition frequency and probability between different alarm types and fault types, and by analyzing the frequency and probability of state transition, identify the alarms that can cause faults, and obtain the association relationship between the alarm type and the fault type.

[0091] In step 303, store the association relationship between the alarm type and the fault type in the cause-and-effect knowledge base.

[0092] In this embodiment, the association relationship between the alarm type and the fault type can be determined at regular time intervals, for example, every three months, based on the fault data of historical faults and the alarm data of historical alarms, and stored in the causal knowledge base correspondingly, or the association relationship between the alarm type and the fault type stored in the causal knowledge base can be updated.

[0093] Through the above steps 301 - 303, an implementation method for determining the association relationship between an alarm type and a fault type using a Markov chain is disclosed. By periodically updating the causal knowledge base according to historical faults and the alarm data before the historical faults, it helps to improve the accuracy of the causal relationship between faults and alarms, and further enhances the accuracy and effectiveness of fault drills.

[0094] In some other alternative implementation manners, other state transition models can also be used to determine the association relationship between the alarm type and the fault type. For example, hidden Markov models, conditional random fields, etc.

[0095] Among them, the hidden Markov model is a common probabilistic state transition model, and the transition probability between states and the probability of a state generating a specific observation value can be described by a state transition matrix and an observation probability matrix. The conditional random field is a probabilistic state transition model based on an undirected graph, which can be represented by an undirected graph. Each node represents a state or an observation variable, and each edge represents the relationship between nodes. The conditional random field uses feature functions and weights to describe the state transition probability.

[0096] Figure 4 It is a flowchart of step 104 of the fault drill method of the present disclosure; the embodiments of the present disclosure are exemplarily described by taking how to generate fault drill cases as an example. On the basis of the above Figure 1 shown embodiments, step 104 includes steps 141 - 142. Each step will be described separately below.

[0097] In step 141, at least one fault drill plan corresponding to the fault type of at least one potential fault is obtained from the drill material library.

[0098] Among them, the drill material library stores fault drill plans corresponding to different fault types, and the fault drill cases include at least one fault drill plan. Each fault drill plan includes a fault scenario and a drill strategy. The drill strategy refers to the strategy that can quickly recover the fault when the fault occurs. The fault scenario refers to the abnormal situation or problem corresponding to the fault. The fault scenario can include various types, such as single product fault, power failure and network disconnection, hardware fault, overload - type fault, dependency - type fault, etc.

[0099] In this implementation manner, the drill material library can be pre-generated based on all possible faults obtained through systematic analysis of the components of the system. The drill material library contains fault scenarios and drill strategies for all possible fault types of the system.

[0100] Among them, the fault drill plan is a plan for performing fault drills for potential faults, including simulating faults such as emergency response processes, resource allocation strategies, etc.

[0101] In this embodiment, if there are multiple fault types for at least one potential fault, multiple corresponding fault drill plans can be obtained from the drill material library; if there is one fault type for at least one potential fault, one corresponding fault drill plan can be obtained from the drill material library.

[0102] In step 142, based on the fault scenarios and drill strategies corresponding to each fault drill plan in at least one fault drill plan, fault drill cases are generated.

[0103] In this embodiment, if there are multiple fault types for at least one potential fault, the corresponding multiple fault drill plans can be combined to obtain one or more fault drill cases; if there is one fault type for at least one potential fault, the corresponding fault drill case can be generated according to this fault drill plan.

[0104] Among them, the fault drill case includes a fault scenario, a fault drill object, and a drill strategy.

[0105] Among them, the specific manner of generating fault drill cases according to the fault drill plans in the drill material library can refer to related technologies and will not be elaborated here.

[0106] Through the above steps 141 - step 143, an implementation manner of generating fault drill cases using the drill material library is disclosed. The method of generating fault drill cases through the drill material library is more efficient, and the drill plans in the drill material library are formulated based on the risk nodes identified after analyzing the system, with higher reliability.

[0107] Figure 5 It is a flowchart of another embodiment of the fault drill method of the present disclosure; the embodiments of the present disclosure are exemplarily described by taking how to perform fault drills as an example, as Figure 5 The method shown includes steps 501 - step 507. Each step will be described separately below.

[0108] In step 501, based on a preset period, at least one piece of alarm data of the system is obtained, and the alarm data includes an alarm type.

[0109] In step 502, it is determined whether there is any fault alarm data in at least one piece of alarm data, and the fault alarm data is alarm data with potential fault problems;

[0110] In step 503, in response to the existence of at least one piece of fault alarm data, at least one fault type of potential faults corresponding to the alarm type of the at least one piece of fault alarm data is obtained from the causal knowledge base.

[0111] Among them, the causal knowledge base stores the association relationship between the alarm type and the fault type.

[0112] In step 504, based on the fault types of at least one potential fault and the drill material library, a fault drill case is generated.

[0113] Among them, the drill material library stores fault drill schemes corresponding to different fault types, and the fault drill case includes at least one fault drill scheme.

[0114] In step 505, based on the fault drill case, a fault drill operation is performed on the system.

[0115] In some embodiments, the implementation manners of steps 501 - 505 can be referred to Figure 1 the embodiments shown, which will not be elaborated here.

[0116] In step 506, the drill index data during the fault drill operation is obtained, and the drill index data includes at least one of the following: response time, fault recovery time, and resource utilization efficiency.

[0117] Among them, the response time refers to the time taken for a device or system to start processing a fault or taking actions after a fault occurs. The recovery time refers to the time required for a system or service to recover from a fault state to a normal state after a fault occurs. The resource utilization efficiency is used to indicate the degree of use and utilization effect of resources.

[0118] In this embodiment, during the fault drill process after the simulated fault injection operation, the drill progress can be tracked through real-time monitoring of the system, the drill index data during the fault drill operation can be obtained, and the drill scheme can be dynamically adjusted according to the actual situation. After the fault drill ends, the fault drill effect and fault drill indexes are evaluated to measure the fault response speed, fault handling efficiency, and overall recovery time.

[0119] Furthermore, if it is determined according to the drill index data that the drill scheme needs to be optimized, step 507 can be executed.

[0120] In step 507, in response to the drill index data indicating that the drill material library needs to be updated, the fault drill schemes included in the fault drill case are updated based on the drill index data.

[0121] Among them, if the response time, fault handling efficiency, fault recovery time, etc. in the drill index data indicate that the fault drill plan in the fault drill case has a poor effect, for example, a long response time, a long fault recovery time, etc., it means that the drill plan in the drill material library has a poor effect, and the drill plan needs to be optimized and adjusted.

[0122] In this embodiment, the specific method for updating the fault drill plan included in the fault drill case according to the drill index data can refer to the related technology and will not be elaborated here.

[0123] Through the above steps 501 - 507, the implementation method for updating the drill material library is disclosed. Combining the drill index data in the drill process, the drill plan can be continuously improved, the drill material library can be updated regularly, ensuring that the drill content is always consistent with the actual risk status of the system, and improving the stability and reliability of the system.

[0124] 0Corresponding to the embodiment of the foregoing fault drill method, the present disclosure also provides an embodiment of a fault drill device.

[0125] Exemplary apparatus

[0126] Figure 6 It is a schematic structural diagram of an embodiment of the fault drill device of the present disclosure. As Figure 6 shown, the device includes:

[0127] A first acquisition module 61, configured to acquire at least one piece of alarm data of the system based on a preset period, where the alarm data includes an alarm type;

[0128] A first determination module 62, configured to determine whether there is fault alarm data in the at least one piece of alarm data, where the fault alarm data is alarm data with potential fault problems;

[0129] A second acquisition module 63, configured to, in response to the existence of at least one piece of fault alarm data, acquire at least one fault type of at least one potential fault corresponding to the alarm type of the at least one piece of fault alarm data from a causality knowledge base, where the causality knowledge base stores an association relationship between the alarm type and the fault type;

[0130] A use case generation module 64, configured to generate a fault drill use case based on the fault type of at least one potential fault and a drill material library, where the drill material library stores fault drill plans corresponding to different fault types, and the fault drill use case includes at least one fault drill plan;

[0131] A fault drill module 65, configured to perform a fault drill operation on the system based on the fault drill use case

[0132] Figure 7A structural diagram of another embodiment of the fault drill device disclosed herein. Figure 7 As shown, in Figure 6 Based on the illustrated embodiment, in some implementations of the present disclosure, the first determining module 62 includes:

[0133] A classification submodule 621 is configured to cluster the at least one piece of alarm data based on the alarm type corresponding to each alarm data in the at least one piece of alarm data to obtain at least one alarm instance corresponding to the at least one alarm type, wherein each alarm instance includes at least one classified alarm data of the same alarm type;

[0134] A first determining submodule 622 is configured to obtain a change trend of the classified alarm data of each alarm type based on at least one piece of classified alarm data in each alarm instance using an autoregressive differential moving average model;

[0135] The second determination submodule 623 is used to determine the classified alarm data with potential fault problems in at least one alarm data based on the change trend of the classified alarm data of each alarm type, wherein the classified alarm data with potential fault problems is the fault alarm data with potential fault problems.

[0136] In some embodiments of the present disclosure, further comprising:

[0137] A third acquisition module 66 is configured to acquire fault data corresponding to at least one historical fault and alarm data corresponding to at least one historical alarm before each historical fault occurs, wherein the fault data includes a fault type;

[0138] A second determination module 67 is configured to determine, using a transition probability matrix of a Markov chain, an association between an alarm type and a fault type based on fault data corresponding to at least one historical fault and alarm data corresponding to at least one historical alarm before each historical fault occurs;

[0139] The storage module 68 is used to store the association relationship between the alarm type and the fault type in the causal knowledge base.

[0140] In some embodiments of the present disclosure, the fault drill plan includes a fault scenario and a drill strategy;

[0141] The use case generation module 64 includes:

[0142] The solution acquisition submodule 641 is configured to acquire at least one fault drill solution corresponding to a fault type of at least one potential fault from a drill material library;

[0143] The use case generation submodule 642 is configured to generate a fault drill use case based on the fault scenario and drill strategy corresponding to each fault drill scheme in at least one fault drill scheme.

[0144] In some embodiments of the present disclosure, it further includes:

[0145] A third acquisition module 69, configured to acquire drill index data during the operation of a fault drill. The drill index data includes at least one of the following: response time, fault recovery time, and resource utilization efficiency;

[0146] An update module 70, configured to update the fault drill scenario included in the fault drill case based on the drill index data in response to an indication from the drill index data that the drill material library needs to be updated.

[0147] In some embodiments of the present disclosure, it further includes:

[0148] A preprocessing module 71, configured to perform data cleaning and preprocessing operations on at least one piece of alarm data. The preprocessing operations are used to unify the data format, and the preprocessing operations include at least one of the following: unifying the data format and unifying the data encoding.

[0149] The device according to the embodiments of the present disclosure can be used to implement the methods of the above various embodiments of the present disclosure. The specific implementations between the two correspond to each other, and the specific implementations of the relevant parts are mutually referred to and will not be elaborated herein.

[0150] Exemplary electronic device, computer program product, and computer-readable storage medium

[0151] The embodiments of the present disclosure further provide an electronic device, including: a memory, configured to store a computer program; a processor, configured to execute the computer program stored in the memory, and when the computer program is executed, implement the fault drill method of any one of the above embodiments of the present disclosure.

[0152] Next, with reference to Figure 8 describe the electronic device according to the embodiments of the present disclosure, in which a device for implementing the method according to the embodiments of the present disclosure can be integrated. Figure 8 is a structural diagram of an electronic device provided for an illustrative embodiment of the present disclosure. As Figure 8 shown, the electronic device includes one or more processors 81, a memory 82 of one or more computer-readable storage media, and a computer program stored on the memory and executable on the processor. When executing the program of the memory 82, the above-mentioned fault drill method can be implemented.

[0153] Specifically, in practical applications, the electronic device may further include components such as an input device 83 and an output device 84, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art can understand that Figure 8 the structure of the electronic device shown in

[0154] The processor 81 can be a central processing unit (CPU) or other forms of processing units with fault drill capabilities and / or instruction execution capabilities. By running or executing software programs and / or modules stored in the memory 82, and invoking the data stored in the memory 82, it performs various functions and processes data, thereby monitoring the electronic device as a whole.

[0155] The memory 82 can store one or more computer program products. The memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The above-mentioned volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. The above-mentioned non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor 81 can run the computer program products to implement the fault drill methods of the various embodiments of the present disclosure above and / or other desired functions.

[0156] The input device 83 can be used to receive input digital or character information. The input device 83 can include a keyboard, a mouse, a joystick, etc. related to user settings and function control.

[0157] The output device 84 can output various information to the outside, including the determined distance information, direction information, etc. The output device 84 can include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0158] The electronic device can also include a power supply for powering each component, which can be logically connected to the processor 81 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator, etc.

[0159] Of course, for simplicity, Figure 8 only some of the components related to the present disclosure in the electronic device are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device can also include any other appropriate components.

[0160] In addition to the above methods and devices, the embodiments of the present disclosure can also be computer program products, which include computer program instructions that, when run by a processor, cause the processor to execute the steps in the fault drill methods according to various embodiments of the present disclosure described in the "Exemplary Methods" section of this specification.

[0161] A computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0162] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the fault drill method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0163] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. In addition, the above-disclosed specific details are only for the purposes of illustration and facilitating understanding, and are not limitations. The above details do not limit the present disclosure to necessarily implement using the above specific details.

[0165] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments may be referred to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiments.

[0166] Those of ordinary skill in the art will understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including those of the above method embodiments. The aforementioned storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disks, or optical discs.

[0167] The methods and apparatuses of the present disclosure may be implemented in many ways. For example, the methods and apparatuses of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only. The steps of the method of the present disclosure are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0168] The description of the present disclosure is given for purposes of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations are obvious to those of ordinary skill in the art. The embodiments are chosen and described in order to better explain the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific purposes.

Claims

1. A fault drill method, characterized in that Including: Based on a preset period, obtain at least one piece of alarm data of the system, where the alarm data includes an alarm type; Determine whether there is any fault alarm data in the at least one piece of alarm data, where the fault alarm data is alarm data with potential fault problems; In response to the existence of at least one piece of fault alarm data, obtain at least one fault type of potential faults corresponding to the alarm type of the at least one piece of fault alarm data from a causal knowledge base, where the association relationship between the alarm type and the fault type is stored in the causal knowledge base; Based on the fault type of the at least one potential fault and a drill material library, generate fault drill cases, where different fault drill scenarios corresponding to different fault types are stored in the drill material library, and the fault drill cases include at least one fault drill scenario; Based on the fault drill cases, perform a fault drill operation on the system.

2. The method according to claim 1, characterized in that The determining whether there is any fault alarm data in the at least one piece of alarm data includes: Based on the alarm types corresponding to each piece of alarm data in the at least one piece of alarm data, cluster the at least one piece of alarm data to obtain at least one alarm instance corresponding to at least one alarm type, and each alarm instance includes at least one piece of classified alarm data with the same alarm type; Using an autoregressive integrated moving average model, based on at least one piece of classified alarm data in each alarm instance, obtain the change trend of the classified alarm data of each alarm type; Based on the change trend of the classified alarm data of each alarm type, determine the classified alarm data with potential fault problems in the at least one piece of alarm data, where the classified alarm data with potential fault problems is the fault alarm data.

3. The method according to any one of claims 1-2, characterized in that, Also including: Obtain fault data corresponding to at least one historical fault and alarm data corresponding to at least one historical alarm before the occurrence of each historical fault, where the fault data includes a fault type; Using the transition probability matrix of a Markov chain, based on the fault data corresponding to the at least one historical fault and the alarm data corresponding to at least one historical alarm before the occurrence of each historical fault, determine the association relationship between the alarm type and the fault type; Store the association relationship between the alarm type and the fault type in the causal knowledge base.

4. The method according to any one of claims 1-3, characterized in that The fault drill scenario includes a fault scenario and a drill strategy; The generating of the fault drill cases based on the fault type of the at least one potential fault and the drill material library includes: Obtain at least one fault drill scenario corresponding to the fault type of the at least one potential fault from the drill material library; Based on the fault scenario and the drill strategy corresponding to each fault drill scenario in the at least one fault drill scenario, generate the fault drill cases.

5. The method according to any one of claims 1-4, characterized in that, During the process of performing the fault drill operation on the system based on the fault drill cases, it also includes: Obtain drill index data during the fault drill operation, where the drill index data includes at least one of the following: response time, fault recovery time, resource utilization efficiency; In response to the drill index data indicating that the drill material library needs to be updated, update the fault drill scenarios included in the fault drill cases based on the drill index data.

6. The method according to any one of claims 1-5, characterized in that, After obtaining at least one piece of alarm data of the system based on a preset period, it further includes: Performing data cleaning and preprocessing operations on the at least one piece of alarm data, where the preprocessing operation is used to unify the data format, and the preprocessing operation includes at least one of the following: unifying the data format, unifying the data encoding.

7. A fault drill device, characterized in that, It includes: A first acquisition module, configured to obtain at least one piece of alarm data of the system based on a preset period, where the alarm data includes an alarm type; A first determination module, configured to determine whether there is any fault alarm data in the at least one piece of alarm data, where the fault alarm data is alarm data with potential fault problems; A second acquisition module, configured to, in response to the existence of at least one piece of fault alarm data, obtain at least one fault type of potential faults corresponding to the alarm type of the at least one piece of fault alarm data from a causal knowledge base, where the association relationship between the alarm type and the fault type is stored in the causal knowledge base; A use case generation module, configured to generate a fault drill use case based on the fault type of the at least one potential fault and a drill material library, where different fault drill scenarios corresponding to different fault types are stored in the drill material library, and the fault drill use case includes at least one fault drill scenario; A fault drill module, configured to perform a fault drill operation on the system based on the fault drill use case.

8. A computer-readable storage medium, where the storage medium stores computer program instructions, and when the computer program instructions are executed, the method described in any one of claims 1-6 above is implemented.

9. An electronic device, where the electronic device includes: A memory, configured to store a computer program product; A processor, configured to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in any one of claims 1-6 above is implemented.

10. A computer program product, comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, the method described in any one of claims 1-6 above is implemented.

Images