Interpretable dispatching automation system fault root cause positioning method and system
By collecting and analyzing the index data of the scheduling automation system and building software and hardware topology, using event inference tree for failure reasoning, the problem that the scheduling automation system is difficult to locate the root cause of the failure when facing massive alarms and complex topology is solved, and the problem of higher fault positioning interpretability and system stability is achieved.
Patent Information
- Application Number
- CN202411849388.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-16
- Publication Date
- 2025-04-22
AI Technical Summary
When facing massive alarms and complex software and hardware topologies, existing scheduling automation systems are difficult to quickly and accurately locate the root cause of failures, and the machine learning classification method is low, making it difficult for operation and maintenance personnel to understand the results of failure classification.
By collecting system indicator data and storing it into a timing database, building a system software and hardware topology, defining an event inference tree, using the event inference tree for indicator abnormality detection and system failure inference, and locate the root cause of the failure.
It solves the problem of massive alarm storms, improves the interpretability of the root cause positioning of the fault, assists operation and maintenance engineers to respond quickly to the fault, and ensures the stability and reliability of the system.
Smart Images
Figure CN119862057A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of power system automation, and in particular to an explainable method and system for locating the root cause of a dispatching automation system fault. Background Art
[0002] With the development of ICT technology, the dispatching automation system architecture has undergone major changes. The addition of new technologies and new requirements has greatly enhanced the correlation between various links in the dispatching automation system. The increase in the complexity of the system's software and hardware topology makes it difficult for operation and maintenance personnel to locate the root cause of the fault. When using traditional manual operation and maintenance to face the operation and maintenance needs of tens of thousands of operation and maintenance objects, it requires multiple R&D and operation and maintenance personnel such as applications, middleware, and hardware to jointly troubleshoot problems, which consumes a lot of manpower and time. Therefore, how to quickly and accurately locate the root cause of the fault is an urgent problem that needs to be solved.
[0003] At present, the dispatching automation system mainly performs anomaly detection on various indicators of the software and hardware in the system, and issues alarms for the results of anomaly detection. This solution will generate a large number of alarms, and the correlation between the alarms cannot be reflected, resulting in the unavailability of the operation and maintenance system. At the same time, the fault diagnosis solution based on machine learning classification methods lacks the interpretability of the results. The operation and maintenance personnel cannot understand the correlation between the fault classification results and the indicator performance. In addition, this type of method requires a large number of fault labels for the training of the classification model, which is difficult to put into practical use.
[0004] Therefore, there is an urgent need for an explainable method for locating the root cause of faults in dispatching automation systems. Summary of the invention
[0005] Purpose of the invention: The purpose of the present invention is to provide an explainable method and system for locating the root cause of faults in a dispatching automation system, to solve the problem of massive alarm storms in existing indicator abnormality alarm schemes and the problem of low interpretability of machine learning classification schemes, to improve the availability of the operation and maintenance system, and to ensure the safe and stable operation of the power grid dispatching automation system.
[0006] Technical solution: The present invention discloses an explainable method for locating the root cause of a dispatching automation system fault, comprising the following steps:
[0007] (1) Collect indicator data from the system, store the collected indicator data in a time series database, and construct an indicator set; build the system hardware and software topology based on the system architecture information and the call chain log generated by the system;
[0008] (2) defining an event reasoning tree, which includes indicator values, indicator abnormal events, system failure events, indicator abnormality detection operators, and system failure reasoning operators;
[0009] (3) Use the event inference tree to perform anomaly detection on the indicator set, infer the indicator abnormal events, and further infer the system failure events;
[0010] (4) Locate the root cause of the fault based on the system hardware and software topology and system fault events.
[0011] Furthermore, in step (1), an indicator collection period Δt is set, and the indicator data includes system indicators and business call chain indicators;
[0012] Collect system indicator data from the system once every Δt; the system indicator data includes system load, CPU, disk, memory, and network;
[0013] Every Δt, a business call chain index is extracted and counted from the call chain log generated by the process within the past Δt. The business call chain index includes the average duration of process call, the number of calls, and the call error rate.
[0014] Furthermore, in step (1), building the system hardware and software topology is to extract the hardware and software topology information from the system architecture information and the call chain log generated by the system, and the hardware and software topology information includes concepts and relationships, wherein the concepts include host, process, message middleware, and database; and the relationships include process calling process, process calling message middleware, process calling database, host deploying process, host deploying message middleware, and host deploying database.
[0015] Furthermore, the step (2) of defining the event inference tree includes:
[0016] Indicator value: the indicator set defined in the actual system and its value size;
[0017] Indicator anomaly detection operator: includes the calculation node of indicator anomaly detection rules, with the input being an indicator value and the output being an indicator anomaly event corresponding to the indicator;
[0018] Indicator abnormality event: Boolean value, indicating whether a certain indicator has a certain abnormality. The value is True or False. True indicates that the indicator has an abnormality, and False indicates that the abnormality does not exist.
[0019] System fault event inference operator: includes the computing nodes of the system fault inference rules, with inputs of multiple indicator abnormal events associated with the fault and outputs a system fault event;
[0020] System fault event: Boolean value, indicating whether a system fault exists. The value is True or False. True indicates that the fault exists, and False indicates that the fault does not exist.
[0021] Furthermore, the inference of abnormal indicator events in step (3) includes the following steps:
[0022] (3.1) For a single indicator m in the time series database, set the current time to time t, and the operation and maintenance indicator data at this time is m t , take the time T from the current time forward 1 The indicator sequence is constructed based on the data within minutes, that is, the time range is [tT 1 ,t]; index sequence S m It is expressed as follows:
[0023]
[0024] (3.2) According to the indicator anomaly detection operators corresponding to different indicators, it is inferred whether there is an indicator abnormal event for the indicator, and if so, the indicator abnormal event is output.
[0025] Furthermore, the indicator anomaly detection operator detects the indicator value in the event reasoning tree according to a preset algorithm. If the indicator value has such an indicator abnormal event, the corresponding indicator abnormal event value in the event reasoning tree is set to True, otherwise it is set to False.
[0026] Furthermore, in step (3), the system fault event is further inferred, and the system fault event inference operator calculates multiple related indicator abnormal events through Boolean logic operation expressions to obtain the value of the system fault event:
[0027] B e =F(a 1 , a 2 ,…,a n )
[0028] Where e represents a fault event, B e A Boolean value indicating whether a fault event e exists; F is a Boolean function, which is a Boolean logic operation expression used to infer whether a fault event e exists; a 1 ,a 2 ,…,a n A Boolean value representing the metric anomaly event associated with the failure event e.
[0029] Furthermore, the value of the system fault event indicates whether the system has such a fault, True indicates that such a fault exists, and False indicates that such a fault does not exist; the Boolean logic operation expression is set according to expert experience and preset in the system fault event reasoning operator.
[0030] Furthermore, the root cause location in step (4) includes the following steps:
[0031] (4.1) Infer and locate the bottom-level fault business process based on the process calling process, message middleware, database relationship and process failure events;
[0032] (4.2) Infer and locate the host where the bottom-level fault process is located based on the relationship between the process, message middleware, and database deployment host;
[0033] (4.3) Check whether there is a host failure event on the host where the faulty process is located. If so, the fault alarm issued includes the process failure event and node failure event of the current host. Otherwise, only the process failure event is alarmed.
[0034] The present invention also discloses a computer system, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned fault root cause locating method.
[0035] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: automatic extraction of the software and hardware topology of the dispatching automation system, definition of the rules for indicator anomalies and system failures, inference of the root cause of failures in the dispatching automation system, solving the problem of massive alarm storms, improving the explainability of root cause location of failures, assisting operation and maintenance engineering personnel to locate and quickly respond to failures, and ensuring the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A flowchart of an explainable method for locating the root cause of a fault in a dispatching automation system according to the present invention;
[0037] Figure 2 The present invention is a schematic diagram of an event reasoning tree of an explainable dispatching automation system fault root cause location method. DETAILED DESCRIPTION
[0038] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0039] The present invention provides an explainable method for locating the root cause of a dispatching automation system fault, comprising the following steps:
[0040] (1) Collect indicator data from the system, store the collected indicator data in a time series database, and construct an indicator set; build the system hardware and software topology based on the system architecture information and the call chain log generated by the system.
[0041] Collect system indicators from each server in the dispatching automation system. The indicator categories include system load, CPU, disk, memory, and network. Each indicator category contains several system indicators that need to be detected under this category. In this embodiment, the system indicator sampling interval Δt=1 is set, that is, the system indicator is collected once every minute. The indicators are collected and saved in a time series database.
[0042] Collect the call chain logs generated by the process from each server in the scheduling automation system. The call chain logs include the nodeName of the server where the process is located, the call chain identifier traceId, the current call identifier spanID, the previous call identifier parentID, the call duration cost, and the call success flag isSucess. Save the call chain logs in the document database.
[0043] In this embodiment, the call chain index and the system hardware and software are extracted at an extraction time interval Δt=1, and the call chain log generated by the process in the past minute is extracted and counted every 1 minute.
[0044] Building the system hardware and software topology is to extract the hardware and software topology information from the system architecture information and the call chain log generated by the system. The hardware and software topology information includes concepts and relationships, wherein the concepts include host, process, message middleware, and database; the relationships include process calling process, process calling message middleware, process calling database, host deploying process, host deploying message middleware, and host deploying database.
[0045] (2) Define an event reasoning tree, which includes indicator abnormal events, system failure events, indicator abnormality detection operators, and system failure reasoning operators. The event reasoning tree structure in this embodiment is as follows: Figure 2 Specifically including:
[0046] Index value: The index set and its value defined in the actual system are the leaf nodes of the inference tree;
[0047] Indicator anomaly detection operator: includes the calculation node of the indicator anomaly detection rule. The input is a certain indicator value, and the output is a certain indicator anomaly event corresponding to the indicator. It is the intermediate node of the inference tree.
[0048] Indicator abnormality event: Boolean value, indicating whether a certain indicator has a certain abnormality. The value is True or False. True indicates that the indicator has an abnormality, and False indicates that the abnormality does not exist. It is the middle node of the inference tree.
[0049] System fault event inference operator: includes the calculation node of the system fault inference rule, with multiple indicator abnormal events associated with the fault as input and a system fault event as output, which is the intermediate node of the inference tree;
[0050] System failure event: Boolean value, indicating whether a system failure exists. The value is True or False. True indicates that the failure exists, and False indicates that the failure does not exist. It is the root node of the inference tree.
[0051] The preset indicator anomaly detection operator in this embodiment is used to detect whether there is an indicator abnormal event in the indicator set. If so, the indicator abnormal event value corresponding to the indicator is set to True, and the indicator abnormality detection operator outputs the name of the existing indicator abnormal event. In this embodiment, the indicator abnormal event includes:
[0052] 1. Abnormal load indicator events: The load on a server system increases;
[0053] 2. Abnormal events of CPU indicators: CPU usage of a server increases, CPU_IO usage of a server increases, CPU_user usage of a server increases;
[0054] 3. Abnormal events of disk indicators: increased usage of a server's disk partition, increased inode usage of a server's disk partition, increased disk read response time of a server, increased disk write response events of a server, and increased disk IO usage of a server;
[0055] 4. Abnormal events of memory indicators: the memory usage of a server increases, the memory usage of a server increases;
[0056] 5. Abnormal events of network indicators: The inbound traffic of a server’s network card decreases, the outbound traffic of a server’s network card decreases, the number of data packets received by a server decreases, and the number of data packets sent by a server decreases.
[0057] 6. Abnormal events of call chain indicators: the call volume of a process decreases, the error rate of a process call increases, the time consumed by a process call increases, and the time consumed by a process call itself increases.
[0058] The indicator anomaly detection operator outputs several indicator anomaly events to the system fault reasoning operator, and the system fault reasoning operator outputs corresponding system fault events based on these. In this embodiment, the system fault events include:
[0059] 1. Server system failures: a server's disk IO is busy, a server's disk space is consumed, a server's memory is consumed, a server's network failure, a server's network traffic is reduced, and a server's CPU is consumed;
[0060] 2. Process system failure: a process call fails or a process call is inefficient.
[0061] (3) Use the event inference tree to perform anomaly detection on the indicator set, infer the indicator abnormal events, and further infer the system failure events;
[0062] Extract indicators from the time series library to perform indicator abnormal event reasoning. For a single operation and maintenance indicator m, the operation and maintenance indicator data at the current time t is m t , take the time window T from the current timestamp 1 The data within minutes is used as the indicator anomaly detection data, that is, the time range is [tT 1 ,t].
[0063] In this embodiment, T 1 = 30 minutes. Indicator sequence S m The formula is expressed as
[0064] S m ={m t-30 ,m t-29 ,...,m t}
[0065] The anomaly detection algorithm infers the indicator S according to the inference rule of the indicator abnormal event. m Whether there are any abnormal indicator events.
[0066] As S m is the CPU usage of server A. In this embodiment, the inference rule of the event “a certain server CPU usage is high” can be defined as S m An anomaly is detected at time t, and m t >50%.
[0067] According to the inference rules of the inferred indicator abnormal events and system fault events, the fault events are inferred. For example, process a has three indicator abnormal events: "the number of calls to a certain process has decreased", "the time consumption of a certain process has increased", and "the time consumption of a certain process calling itself has increased". The system fault event "a certain process calling is inefficient" is defined in the event inference tree. The system fault event inference operator of the system fault event is "two or more of the three indicator abnormal events of 'a certain process calling volume has decreased', 'the time consumption of a certain process calling has increased', and 'the time consumption of a certain process calling itself has increased'". Through the event inference tree, it is inferred that process a has a system fault event of "a certain process calling is inefficient", and the system fault event of "a certain process calling is inefficient" in the event inference tree is set to True, and the result is output.
[0068] (4) Locate the root cause based on the system hardware and software topology and system failure events.
[0069] The faulty business process at the bottom of the call chain is located based on the process calling process, message middleware, database relationship and process failure events.
[0070] Based on the relationship between the process, message middleware, and database deployment host, the host where the bottom-level fault process is located is inferred. Check whether there is a host fault event on the host where the faulty process is located. If so, the fault alarm issued includes the process fault event and node fault event of the current host. Otherwise, only the process fault event is alarmed.
[0071] For example, in this embodiment, a service process call topology is as follows: "Process A → Process B → Process C → Process D". At this time, process A, process B, and process C all have "process call failure" faults. According to the call topology relationship, process C is the root cause process of the fault. Process C is deployed on server A. At this time, check whether server A has a server-related system fault, such as "CPU consumption of server A". Finally, the root cause location output result is "CPU consumption fault occurs on server A, and process C call fails."
[0072] An embodiment of the present invention also provides a system, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, a method and system for locating the root cause of a fault in a scheduling automation system based on event reasoning as described in any one of the above items are implemented.
Claims
1. An explainable method for locating the root cause of a dispatching automation system fault, characterized in that: Includes steps: (1) Collect indicator data from the system, store the collected indicator data in a time series database, and construct an indicator set; build the system hardware and software topology based on the system architecture information and the call chain log generated by the system; (2) defining an event reasoning tree, which includes indicator values, indicator abnormal events, system failure events, indicator abnormality detection operators, and system failure reasoning operators; (3) Use the event inference tree to perform anomaly detection on the indicator set, infer the indicator abnormal events, and further infer the system failure events; (4) Locate the root cause of the fault based on the system hardware and software topology and system fault events.
2. The method for locating the root cause of a fault according to claim 1, characterized in that: In step (1), an indicator collection period Δt is set, and the indicator data includes system indicators and business call chain indicators; Collect system indicator data from the system once every Δt; the system indicator data includes system load, CPU, disk, memory, and network; Every Δt, a business call chain index is extracted and counted from the call chain log generated by the process within the past Δt. The business call chain index includes the average duration of process call, the number of calls, and the call error rate.
3. The method for locating the root cause of a fault according to claim 1, characterized in that: The step (1) of constructing the system hardware and software topology is to extract the hardware and software topology information from the system architecture information and the call chain log generated by the system, and the hardware and software topology information includes concepts and relationships, wherein the concepts include host, process, message middleware, and database; the relationships include process calling process, process calling message middleware, process calling database, host deploying process, host deploying message middleware, and host deploying database.
4. The fault root cause location method according to claim 1, characterized in that: The step (2) of defining an event inference tree comprises: Indicator value: the indicator set defined in the actual system and its value size; Indicator anomaly detection operator: includes the calculation node of indicator anomaly detection rules, with the input being an indicator value and the output being an indicator anomaly event corresponding to the indicator; Indicator abnormality event: Boolean value, indicating whether a certain indicator has a certain abnormality. The value is True or False. True indicates that the indicator has an abnormality, and False indicates that the abnormality does not exist. System fault event inference operator: includes the computing nodes of the system fault inference rules, with inputs of multiple indicator abnormal events associated with the fault and outputs a system fault event; System fault event: Boolean value, indicating whether a system fault exists. The value is True or False. True indicates that the fault exists, and False indicates that the fault does not exist.
5. The method for locating the root cause of a fault according to claim 1, characterized in that: The inference of abnormal indicator events in step (3) comprises the following steps: (3.1) For a single indicator m in the time series database, set the current time to time t, and the operation and maintenance indicator data at this time is m t , take the data from the current time to the previous T1 minutes to construct the indicator sequence, that is, the time range is [t-T1,t]; the indicator sequence S m It is expressed as follows: (3.2) According to the indicator anomaly detection operators corresponding to different indicators, it is inferred whether there is an indicator abnormal event for the indicator, and if so, the indicator abnormal event is output.
6. The method for locating the root cause of a fault according to claim 5, characterized in that: The indicator anomaly detection operator detects the indicator value in the event reasoning tree according to a preset algorithm. If the indicator value has this kind of indicator abnormal event, the corresponding indicator abnormal event value in the event reasoning tree is set to True, otherwise it is set to False.
7. The method for locating the root cause of a fault according to claim 1, characterized in that: In step (3), the system fault event is further inferred, and the system fault event inference operator calculates multiple related indicator abnormal events through Boolean logic operation expressions to obtain the value of the system fault event: <h2 style=";text-align:left;direction:ltr">B<h2 style=";text-align:left;direction:ltr"> e <h2 style=";text-align:left;direction:ltr"> =F(a1,a2,…,a<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ) Where e represents a fault event, B e A Boolean value indicating whether a fault event e exists; F is a Boolean function, which is a Boolean logic operation expression used to infer whether a fault event e exists; a1, a2, …, a n A Boolean value representing the metric anomaly event associated with the failure event e.
8. The method for locating the root cause of a fault according to claim 7, characterized in that: The value of the system fault event indicates whether the system has this kind of fault, True indicates that this kind of fault exists, and False indicates that this kind of fault does not exist; the Boolean logic operation expression is set according to expert experience and preset in the system fault event reasoning operator.
9. The method for locating the root cause of a fault according to claim 1, characterized in that: The root cause location in step (4) includes the following steps: (4.1) Infer and locate the bottom-level fault business process based on the process calling process, message middleware, database relationship and process failure events; (4.2) Infer and locate the host where the bottom-level fault process is located based on the relationship between the process, message middleware, and database deployment host; (4.3) Check whether there is a host failure event on the host where the faulty process is located. If so, the fault alarm issued includes the process failure event and node failure event of the current host. Otherwise, only the process failure event is alarmed.
10. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement all steps of the fault root cause locating method according to any one of claims 1 to 9.