Fault node positioning method and device, electronic equipment and nonvolatile storage medium
By obtaining network topology information and monitoring indicator data, generating causal graphs, and using the GES algorithm to locate the root cause of failures, the problems of difficulty in positioning and operation and maintenance in the existing technology are solved, and accurate and efficient positioning of fault nodes is achieved.
Patent Information
- Application Number
- CN202411162978.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the management method based on manual drawing of system resource topology is not suitable for increasingly complex cloud-native asset management, which leads to difficulty in positioning fault nodes and operation and maintenance. In the face of massive operation and maintenance monitoring data, the system and indicators are The complex relationships lead to alarm storms and it is difficult to accurately locate the root cause.
By obtaining network topology information and monitoring indicator data, generating system topology diagrams and causal diagrams, using GES algorithm to build causal relationships, determining the transfer probability matrix, simulating the fault conduction process, and automatically performing alarm root cause positioning to achieve accurate and efficient positioning of the fault nodes.
It improves the efficiency and accuracy of fault positioning, reduces operation and maintenance difficulties, reduces manual operations, and achieves accurate and efficient fault node positioning.
Smart Images

Figure CN119030860A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of IT and software development, and in particular to a method, device, electronic device and non-volatile storage medium for locating a faulty node. Background Art
[0002] In the existing technology, the management method based on manually drawing the system resource topology is no longer suitable for the increasingly complex cloud-native asset management. In addition, faced with massive operation and maintenance monitoring data, the relationship between systems and indicators is becoming more and more complex. A node failure can easily trigger an alarm storm, affecting a wider range, making it time-consuming and laborious to locate the problem. An automatic topology that can reduce a large number of manual operations and a tool that can accurately locate alarms and root causes are needed to reduce the difficulty of operation and maintenance.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present application provide a method, device, electronic device and non-volatile storage medium for locating a faulty node, so as to at least solve the technical problems of difficulty in locating faulty nodes and difficulty in operation and maintenance caused by the fact that existing manual positioning methods are not suitable for increasingly complex networks.
[0005] According to one aspect of an embodiment of the present application, a method for locating a faulty node includes: obtaining network topology information, generating a system topology map based on the network topology information, wherein the network topology information includes information about each node in the network and relationship information between nodes, and the nodes include device nodes and cloud application nodes; obtaining monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data; generating a cause-and-effect graph based on the system topology map and the monitoring indicator data, wherein the cause-and-effect graph is used to represent the influence relationship between each node in the network; in the case where there are abnormal nodes in the network, determining a root cause graph based on the cause-and-effect graph, and locating the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between nodes, and the faulty node is a node that causes the abnormal node alarm.
[0006] Optionally, acquiring network topology information and generating a system topology map based on the network topology information includes: calling configuration information of each node stored in a configuration management database, determining the network topology information, and generating a system topology map based on the network topology information.
[0007] Optionally, obtaining monitoring indicator data includes: issuing a collection instruction for monitoring indicator data; obtaining collection data returned by the terminal device corresponding to the collection instruction; processing the collection data to obtain monitoring indicator data, wherein the processing includes data cleaning and format conversion of the collection data.
[0008] Optionally, generating a causal graph based on the system topology map and monitoring indicator data includes: training a causal graph model based on the system topology map and monitoring indicator data, wherein the causal graph model is constructed based on causal relationships, and causal relationships are used to indicate that there is an influence relationship between nodes, and changes in the state of the cause node will affect the state of the effect node; and obtaining the causal graph output by the trained causal graph model.
[0009] Optionally, in the event that abnormal nodes exist in the network, a root cause graph is determined based on the causal graph, and locating the faulty node based on the root cause graph includes: determining a transition probability matrix based on the causal graph, wherein the transition probability matrix is used to represent the correlation between nodes; processing the causal graph based on the transition probability matrix to determine the root cause graph, wherein the processing includes marking the transition probability between nodes on the causal graph; simulating the fault conduction process based on the root cause graph, counting the number of times each node appears in the conduction trajectory, sorting the nodes from most to least according to the number of times they appear in the conduction trajectory, and determining a preset number of nodes at the front after sorting as faulty nodes.
[0010] Optionally, determining a transition probability matrix based on a causal graph includes: constructing an initialized transition probability matrix, the initialized transition probability matrix having a target number of row data and a target number of column data, wherein the target number is the same as the number of nodes in the network; determining the transition probabilities between nodes in the network based on the causal graph, including a forward transfer probability, a self-stay probability, and a backward transfer probability, wherein the forward transfer probability is used to represent the transition probability from a result node to a root node in the causal graph, the self-stay probability is used to represent the probability of a node transferring to itself in the causal graph, and the backward transfer probability is used to represent the transition probability from a cause node to a result node in the causal graph; filling the transfer probabilities into the initialized transfer probability matrix, performing standardization, and determining the standardized transfer probability matrix as the final transfer probability matrix, wherein the sum of each row of data in the standardized transfer probability matrix is 1, and the sum of each column of data is 1.
[0011] Optionally, after locating the faulty node, the method further includes: determining a causal link based on the faulty node, and dispatching an alarm work order to the nodes in the causal link, wherein the causal link includes the faulty node and the conduction path of the faulty node, and the information in the alarm work order includes the alarm title, alarm resource, and alarm indicator.
[0012] According to another aspect of an embodiment of the present application, a fault node locating device is also provided, including: a first acquisition module, used to acquire network topology information, and generate a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between nodes, and the nodes include device nodes and cloud application nodes; a second acquisition module, used to acquire monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data; a generation module, used to generate a cause-effect graph based on the system topology map and the monitoring indicator data, wherein the cause-effect graph is used to represent the influence relationship between each node in the network; a determination module, used to determine a root cause graph based on the cause-effect graph when there are abnormal nodes in the network, and locate the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between each node, and the faulty node is the node that causes the abnormal node alarm.
[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute a fault node locating method.
[0014] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the fault node locating method is executed when the program is run.
[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, a method for locating a faulty node is implemented.
[0016] In the embodiment of the present application, an automatic IT management system is used to collect Iaas (Infrastructure as a Service), Paas (Platform as a Service), and SaaS (Software as a Service) data, and a CMDB (Configuration Management Database) topology is constructed. The historical alarms are intelligently analyzed to form a cause-and-effect graph with the help of the GES algorithm, and a probability matrix is constructed through a random walk strategy to automatically locate the root cause of the alarm, thereby achieving the purpose of improving the efficiency and accuracy of the data path and reducing the difficulty of fault location, thereby achieving the technical effect of accurately and efficiently locating the faulty nodes, and further solving the technical problems of difficulty in locating the faulty nodes and difficulty in operation and maintenance caused by the existing manual positioning methods being not suitable for increasingly complex networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0018] Figure 1 It is a structural schematic diagram of a computer terminal (mobile device) provided according to an embodiment of the present application;
[0019] Figure 2 It is a flowchart of a method for locating a faulty node provided in an embodiment of the present application;
[0020] Figure 3 This is a schematic diagram of a process for automatically discovering network topology information according to an embodiment of the present application;
[0021] Figure 4 It is a specific flow chart of a pingpoint tracking technology provided according to an embodiment of the present application;
[0022] Figure 5 It is an automatically formed topological structure diagram provided according to an embodiment of the present application;
[0023] Figure 6 This is a schematic diagram of a monitoring indicator data acquisition process provided according to an embodiment of the present application;
[0024] Figure 7 is a schematic diagram of an alarm cause-effect diagram provided according to an embodiment of the present application;
[0025] Figure 8 is a schematic diagram of a root cause map provided according to an embodiment of the present application;
[0026] Fig. 9 It is a structural schematic diagram of a fault node locating device provided according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0029] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0030] CMDB: Configuration management database, which contains information about the entire life cycle of configuration items and the relationships between configuration items.
[0031] GES: An algorithm for structural equation modeling (SEM) and causal models, in particular for inferring causal graphs or directed acyclic graphs (DAGs).
[0032] SNMP (Simple Network Management Protocol) is an application layer protocol widely used in the field of network management and monitoring. It allows network administrators to monitor and manage network devices such as routers, switches, and servers from a central location. SNMP helps administrators discover and solve problems in a timely manner by collecting status information, configuration parameters, and performance indicators of network devices, and realizes multiple functions such as remote monitoring, fault diagnosis, configuration management, and performance analysis. In addition, SNMP also supports event notification mechanisms, which can send warning information to administrators when a device fails or performance degrades. The working principle of SNMP is based on the Management Information Base (MIB), which is a hierarchical tree structure used to organize and store information about network devices. By providing operations such as GET, SET, GETNEXT, and GETBULK, SNMP can easily access and manage information in the MIB. Although SNMP itself does not provide a strong security mechanism, it can be used in combination with other security protocols to improve the security of network management. In general, SNMP is a simple and efficient network management protocol that can help network administrators manage and maintain network devices more easily and improve the stability and performance of the network.
[0033] Pinpoint is an open source APM (Application Performance Management) tool for large distributed systems written in Java / PHP. Inspired by Dapper, Pinpoint provides a solution that helps analyze the overall structure of the system and how the components in them are connected to each other by tracking transactions between distributed applications. Pinpoint features: Understand the application topology in the network at a glance; monitor applications in the network in real time; get code-level visibility for each transaction; use bytecode enhancement technology to add new features without modifying the code; minimal impact on performance (resource usage increased by about 3%). The Pinpoint system architecture is as follows: Point is based on Google Dapper and uses Hbase as storage. It has three main components: pinpoint-collector (log collector module), pinpoint-web (console view module), and pinpoint-agent (log agent client module). Specifically, the log collector module is mainly used to collect and store data information from the agent end. The console view module mainly displays the collector's data to the user in a visual way. The log agent client module is mainly used to perform point embedding in the client segment to obtain monitoring information in Pinpoint. The core data structure consists of Span, Trace, and TraceID: Span: is the basic unit of tracing, containing a TraceId; Trace: is a collection of multiple Spans, consisting of associated RPCs (Spans). The same trace shares the same TransactionID. Trace organizes the inheritance tree structure through SpanId and ParentSpanId. TraceID: A key set consisting of TransactionId, SpanId (64-bit integer), and ParentSpanId (64-bit integer). TransactionId indicates the message ID, while SpanId and ParentSpanId represent the parent-child relationship of the RPC. TransactionId: AgentIDs (hostname, server IP is recommended), JVM startup time, and sequence number.
[0034] IAAS (Infrastructure as a Service): A cloud computing service model that provides users with virtualized computing resources, including servers, storage, networks, and other infrastructures, through the Internet. Users can flexibly select and configure these resources according to their needs without having to purchase and maintain actual hardware equipment. IAAS provides users with a highly scalable, pay-as-you-go infrastructure platform that allows enterprises to quickly deploy and manage applications, reducing initial investment costs and operational risks. Compared with traditional self-built data centers, IAAS has higher flexibility and scalability, which can help enterprises cope with changing market demands and technical challenges. The main advantages of IAAS applications include: low cost-effectiveness, high availability, rapid deployment, security, and easy management: users can achieve centralized management and monitoring of computing resources through the management console or API provided by the IAAS provider. In short, as a flexible and scalable cloud computing service, IAAS can help enterprises reduce costs, improve efficiency, and respond to market changes. By leveraging IAAS, enterprises can devote more energy to the innovation and development of core businesses while enjoying the many advantages brought by cloud computing.
[0035] In the related technology, the management method based on manually drawing the system resource topology is no longer applicable to the increasingly complex cloud-native asset management. In addition, in the face of massive operation and maintenance monitoring data, the relationship between systems and indicators is becoming more and more complex. If a node fails, it is very easy to trigger an alarm storm, affecting a wider range. Therefore, there is a problem of time-consuming and labor-intensive positioning of the faulty node. In order to solve this problem, the relevant solution is provided in the embodiment of the present application, which is described in detail below.
[0036] According to an embodiment of the present application, a method embodiment of a fault node locating method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for locating a faulty node. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0038] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the fault node location method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned fault node location method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0040] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).
[0042] In the above operating environment, the embodiment of the present application provides a method for locating a faulty node, such as Figure 2 As shown, the method comprises the following steps:
[0043] Step S202, obtaining network topology information, and generating a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between nodes, and the nodes include device nodes and cloud application nodes;
[0044] In some embodiments of the present application, an automatic discovery method is used to obtain CMDB topology information. Figure 3 The figure shows a process of automatically discovering network topology information, which is the basic premise for locating the root cause of monitoring alarms. The key measure of this process is to use different automatic discovery technologies to collect information according to the characteristics of different nodes. Figure 3 As shown, the devices that need to be automatically discovered include network devices (such as physical devices that actually exist in the network, including hosts, that is, computers or devices connected to the network. It can be used as a Web server, mail server, file server or carry various cloud service applications, such as cloud computing, cloud storage, etc.) and cloud application devices, including IaaS application devices, PaaS application devices and SaaS devices.
[0045] Specifically, discovering network devices includes automatically discovering network devices through SNMP (Simple Network Management Protocol) and discovering IaaS and PaaS devices through self-developed agents. Self-developed agents are secondary developed based on the open source software telegraf architecture. They naturally have the characteristics of lightweight native software, low resource consumption, and strong scalability. At the same time, new features such as remote control, automatic update, log collection, and process guarding are added to the characteristics of this system, making it more suitable for enterprise production, close to daily operation and maintenance work, and capable of long-term reliable operation and remote maintenance. Obtaining the link relationship between nodes includes using pingpoint tracking technology to discover, Figure 4 The specific process of a pingpoint tracking technology is shown as follows: Figure 4 As shown in the figure, ppCollector receives the call chain data of the application service with pinpoint agent, and concatenates the batch call chain data to generate a topology snapshot through the real-time stream data processing capability of spark. The original resource data of cmdb is queried through the IP and port information of the service topology snapshot data, and then combined with the relationship of the service topology to convert it into the relationship between resources. CMDB cleans and organizes the data reported by various automatic discoveries, and automatically forms a three-layer topology of infrastructure layer, component layer, and application layer. Figure 5 A schematic diagram of an automatically formed topology is shown, such as Figure 5 As shown in the figure, the business system is a three-layer topology, which is divided into four sub-applications, two databases, and three hosts. The topological relationship between applications, databases, and hosts is shown in the following table. User-service is deployed on host0, and user-service is linked to user-db. User-db is deployed on host1; cmdb-db is deployed on host2, and cmdb-service, cmdb-storage, and cmdb-data-access are deployed on host3. cmdb-service is linked to cmdb-db. cmdb-storage calls cmdb-service, cmdb-storage calls cmdb-service, and cmdb-data-access calls cmdb-service.
[0046]
[0047]
[0048] Step S204, obtaining monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data;
[0049] In some embodiments of the present application, monitoring indicator collection includes: collecting monitoring indicators through self-developed Agent in three ways: open source exporter, jmx, and custom scripts. Figure 6 A monitoring indicator data acquisition process is shown, such as Figure 6 As shown in the figure, indicator collection is achieved through the following steps: 1. The collection server configures the component collection plan and related collection scripts, and sends the Agent notification collection through control commands. 2. The collection client (self-developed Agent) receives the collection plan and interprets and executes the collection script according to the plan. 3. The collection client calls exporter or jmx or custom python script to output the collected data. 4. The collection client sends the output collection data to the kafka middleware on the collection server. 5. The indicator processing application processes and stores the data to complete the indicator collection.
[0050] In some embodiments of the present application, monitoring indicator collection also includes using the northbound interface to collect indicators. By subscribing to the kafka middleware topic related to the northbound interface, according to the component collection plan configured by the collection server, the message in the topic is consumed and transferred to the kafka message of the collection server of this system, and then the indicator monitoring is completed through the above 5 steps. The data processing includes cleaning the data, removing duplicate and invalid information, formatting the obtained valid information, reading the recent indicator alarm data of the business system (such as the past 1 month), and organizing the indicator alarm data in the following form: 0 means no alarm occurs, 1 means an alarm occurs, as shown in the following table:
[0051]
[0052] Step S206, generating a cause-effect diagram based on the system topology diagram and the monitoring indicator data, wherein the cause-effect diagram is used to represent the influence relationship between each node in the network;
[0053] In some embodiments of the present application, generating a causal graph includes using the system topology graph automatically constructed in step S202 and the indicator data monitoring and collection implemented in step S204, extracting alarm data within a time period, using the GES algorithm to perform causal graph training and analysis, and performing probability calculation on causal points to form an alarm causal graph with probability for alarm root cause location decision-making. Generating a causal graph is the core step for realizing alarm root cause location. The specific process is as follows:
[0054] Specifically, the GES algorithm function is used to input the data collected in step S202 and step S204 to perform causal diagram training and automatically draw the following Figure 7 The alarm cause-effect diagram shown in Figure 7As described above, there is an influence relationship between each node, that is, the performance change of the cause node will affect the performance of the result node. Specifically, the following table shows the calculated resources and indicators with causal relationship, that is, the "cause-indicator" (changed performance indicator) of the "cause-resource" (cause node) will affect the "result-indicator" (affected performance indicator) of the "result-resource" (result node). For example, the CPU usage rate of host0 will affect the average response time of user-service:
[0055]
[0056] Step S208, when there are abnormal nodes in the network, determine the root cause graph based on the cause-effect graph, and locate the faulty node based on the root cause graph, wherein the abnormal node is the node where an alarm occurs in the network, the root cause graph is used to indicate the probability of fault transmission between nodes, and the faulty node is the node that causes the abnormal node alarm.
[0057] In some embodiments of the present application, a state transition probability matrix is constructed based on the causal graph, and the causal graph page is rendered according to the probability matrix to obtain a causal graph with probability, i.e., the root cause graph. The propagation process of the alarm is simulated using a random walk method based on the root cause graph to determine the fault node.
[0058] Specifically, assume that the causal graph is G(V,E), where V is the set of causal nodes and E is the set of causal edges. First, the state transition probability matrix Q needs to be constructed: the calculation process of the state transition probability matrix Q needs to include three dimensions: forward step, backward step, and self-stay. First, calculate the forward step: calculate the transition probability from the effect node to the cause node, Q ij =R(v i ,v j ), R(v i ,v j ) indicates v i and v j The correlation coefficient between ji ∈E (indicates that the causal relationship from node i to node j exists in the overall causal edge set). The correlation coefficient calculation formula R can be set to different functions R as needed. If the causal edge of the causal graph has a weight, the weight value is directly returned. Otherwise, the Pearson correlation coefficient is used to calculate the correlation of the performance values of two causal nodes, where X i is the i-th performance value of node X, is the average value of the sample X performance value:
[0059]
[0060] Then calculate the backward step: calculate the transition probability from the cause node to the result node, Q ji =ρ*R(vj ,v i ), where ρ is the parameter that controls the impact of the backward step, and is a hyperparameter ρ∈[0,1], indicating that the value range of the hyperparameter is between 0 and 1, where e ji ∈E, Then calculate the self-stay: If the correlation between a node and its neighbor nodes (nodes with direct causal edges with the node) and the abnormal node (alarm node) is low, then this node represents a possible root cause. Calculate the transition probability of self-stay:
[0061]
[0062] Finally, perform matrix normalization: normalize each row of Q so that the probability of each row adds up to 1:
[0063]
[0064] represents the final standardized value, Q ij represents the probability value of the i-th row and j-th column, ∑ j Q ij Indicates the sum of the probability values of all columns in the i-th row) According to the above calculation steps, the constructed probability matrix Q is a 22*22 square matrix, which is calculated in sequence according to the previous calculation formula as follows:
[0065]
[0066] Q 1,1 is the probability of host1.host_cpu_used_percent and host1.host_cpu_used_percent, and the result is 0.7. 1,2 is the probability of host1.host_cpu_used_percent and host1.host_mem_used_percent, the calculated result is 0, and so on, the final probability matrix generated is:
[0067]
[0068] Finally, the causal graph page is rendered according to the probability matrix to obtain the root cause graph, such as Figure 8 .
[0069] Based on the root cause graph, perform a random walk: Perform a random walk on the causal graph G, starting from the abnormal node v alarm At the beginning, each step of the walk is determined according to the transition probability matrix Q, the number of times all walkers of all alarm nodes stay at the node is counted, and the top three or top five nodes with the highest number of stays are output as possible root cause results.
[0070] Obtain the root cause links formed by all possible root cause results, select the causal link with the highest probability as the root cause of the alarm, and dispatch alarm work orders, which greatly reduces and compresses the number of alarm work orders and improves the accuracy of alarm work orders. The alarm work order contains information such as alarm title, alarm resource, alarm indicator, alarm root cause, alarm level, etc. The possibility is represented by the result of multiplying the probability of each transfer path on each link.
[0071] Through the above steps, it is possible to automatically enter CMDB resources by means of automatic discovery and automatically generate topology maps, which greatly improves data entry efficiency and data accuracy. In addition, the present application uses the GES intelligent algorithm to automatically draw alarm cause-and-effect diagrams and probability calculations, and achieves accurate compression of massive alarm data and root cause location, greatly reducing the difficulty of accurate fault location, thereby achieving the technical effect of accurately and efficiently locating faulty nodes, and further solving the technical problems of difficulty in locating faulty nodes and difficulty in operation and maintenance caused by the fact that existing manual positioning methods are not suitable for increasingly complex networks.
[0072] The present invention provides a fault node location device. Fig. 9 is a schematic diagram of the structure of the device, such as Fig. 9 As shown, the device includes: a first acquisition module 90, which is used to acquire network topology information and generate a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between nodes, and the nodes include device nodes and cloud application nodes; a second acquisition module 92, which is used to acquire monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data; a generation module 94, which is used to generate a cause-effect graph based on the system topology map and the monitoring indicator data, wherein the cause-effect graph is used to represent the influence relationship between each node in the network; a determination module 96, which is used to determine a root cause graph based on the cause-effect graph when there are abnormal nodes in the network, and locate the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between each node, and the faulty node is a node that causes the abnormal node alarm.
[0073] In some embodiments of the present application, the first acquisition module 90 acquires network topology information, and generating a system topology map based on the network topology information includes: calling the configuration information of each node stored in the configuration management database, determining the network topology information, and generating a system topology map based on the network topology information.
[0074] In some embodiments of the present application, the second acquisition module 92 acquires monitoring indicator data including: issuing collection instructions for monitoring indicator data; acquiring collection data returned by the terminal device corresponding to the collection instructions; processing the collection data to obtain monitoring indicator data, wherein the processing includes data cleaning and format conversion of the collection data.
[0075] In some embodiments of the present application, the generation module 94 generates a causal graph based on the system topology diagram and monitoring indicator data, including: training a causal graph model based on the system topology diagram and monitoring indicator data, wherein the causal graph model is constructed based on causal relationships, and causal relationships are used to indicate that there is an influence relationship between nodes, and changes in the state of the cause node will affect the state of the effect node; and obtaining the causal graph output by the trained causal graph model.
[0076] In some embodiments of the present application, when there are abnormal nodes in the network, the determination module 96 determines the root cause graph based on the causal graph, and locating the faulty node based on the root cause graph includes: determining a transition probability matrix based on the causal graph, wherein the transition probability matrix is used to represent the correlation between each node; processing the causal graph based on the transition probability matrix to determine the root cause graph, wherein the processing includes marking the transition probability between each node on the causal graph; simulating the fault conduction process based on the root cause graph, counting the number of times each node appears in the conduction trajectory, sorting each node from most to least according to the number of times it appears in the conduction trajectory, and determining a preset number of nodes at the front after sorting as faulty nodes.
[0077] In some embodiments of the present application, the determination module 96 determines the transition probability matrix based on the causal graph, including: constructing an initialized transition probability matrix, the initialized transition probability matrix has a target number of row data and a target number of column data, wherein the target number is the same as the number of nodes in the network; determining the transition probability between each node in the network based on the causal graph, including a forward transfer probability, a self-stay probability, and a backward transfer probability, wherein the forward transfer probability is used to represent the transition probability from the effect node to the root node in the causal graph, the self-stay probability is used to represent the probability of the node transferring to itself in the causal graph, and the backward transfer probability is used to represent the transition probability from the cause node to the effect node in the causal graph; filling the transfer probability into the initialized transfer probability matrix, standardizing it, and determining the standardized transfer probability matrix as the final transfer probability matrix, wherein the sum of each row of data in the standardized transfer probability matrix is 1, and the sum of each column of data is 1.
[0078] In some embodiments of the present application, after locating the faulty node, the method further includes: determining a causal link based on the faulty node, and dispatching an alarm work order to the nodes in the causal link, wherein the causal link includes the faulty node and the conduction path of the faulty node, and the information in the alarm work order includes the alarm title, alarm resource, and alarm indicator.
[0079] It should be noted that each module in the above-mentioned fault node locating device can be a program module (for example, a set of program instructions that implement a certain specific function) or a hardware module. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0080] An embodiment of the present application provides a non-volatile storage medium, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following fault node locating method, including: obtaining network topology information, and generating a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between nodes, and the nodes include device nodes and cloud application nodes; obtaining monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data; generating a cause-effect graph based on the system topology map and the monitoring indicator data, wherein the cause-effect graph is used to represent the influence relationship between each node in the network; in the case where there are abnormal nodes in the network, determining a root cause graph based on the cause-effect graph, and locating the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between nodes, and the faulty node is a node that causes the abnormal node alarm.
[0081] An embodiment of the present application provides an electronic device, including: a memory and a processor, the processor is used to run a program stored in the memory, wherein the following fault node locating method is executed when the program is running, including: obtaining network topology information, generating a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between nodes, and the nodes include device nodes and cloud application nodes; obtaining monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data; generating a cause-effect graph based on the system topology map and the monitoring indicator data, wherein the cause-effect graph is used to represent the influence relationship between each node in the network; in the case where there are abnormal nodes in the network, determining a root cause graph based on the cause-effect graph, and locating the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between nodes, and the faulty node is a node that causes the abnormal node alarm.
[0082] The embodiment of the present application provides a computer program product, including a computer program, which implements the following fault node locating method when executed by a processor, including: obtaining network topology information, generating a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between the nodes, and the nodes include device nodes and cloud application nodes; obtaining monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each node, including node alarm data; generating a cause-effect graph based on the system topology map and the monitoring indicator data, wherein the cause-effect graph is used to represent the influence relationship between each node in the network; in the case where there are abnormal nodes in the network, determining a root cause graph based on the cause-effect graph, and locating the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between each node, and the faulty node is a node that causes the abnormal node alarm.
[0083] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0084] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0086] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0087] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0088] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A fault node location method, characterized in that: include: Acquire network topology information, and generate a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between the nodes, and the nodes include device nodes and cloud application nodes; Acquire monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each of the nodes, including node alarm data; Generate a cause-and-effect diagram based on the system topology diagram and the monitoring indicator data, wherein the cause-and-effect diagram is used to represent the influence relationship between each node in the network; In the case where an abnormal node exists in the network, a root cause graph is determined based on the cause-effect graph, and the faulty node is located based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between the nodes, and the faulty node is a node that causes the abnormal node to alarm.
2. The fault node location method according to claim 1, characterized in that: Acquiring network topology information, and generating a system topology map based on the network topology information includes: The configuration information of each node stored in the configuration management database is called to determine the network topology information, and the system topology map is generated according to the network topology information.
3. The fault node location method according to claim 1, characterized in that: Obtaining monitoring indicator data includes: Publishing a collection instruction for the monitoring indicator data; Acquire the collected data corresponding to the collection instruction returned by the terminal device; The collected data is processed to obtain the monitoring index data, wherein the processing includes at least one of the following: data cleaning and format conversion of the collected data.
4. The fault node location method according to claim 1, characterized in that: Generating a cause-effect diagram according to the system topology diagram and the monitoring indicator data includes: According to the system topology diagram and the monitoring indicator data, a causal graph model is constructed, wherein the causal graph model is constructed based on a causal relationship, and the causal relationship is used to indicate that there is an influence relationship between nodes, and the state change of the cause node will affect the state of the effect node; Output the causal graph based on the causal relationship recorded in the causal graph model.
5. The fault node location method according to claim 1, characterized in that: In the case where an abnormal node exists in the network, determining a root cause graph according to the cause-effect graph, and locating the faulty node according to the root cause graph includes: Determine a transition probability matrix according to the causal graph, wherein the transition probability matrix is used to represent the correlation between nodes; Processing the causal graph according to the transition probability matrix to determine a root cause graph, wherein the processing includes marking the transition probability between nodes on the causal graph; The fault conduction process is simulated according to the root cause map, the number of times each node appears in the conduction trajectory is counted, and each node is sorted in descending order according to the number of times it appears in the conduction trajectory, and a preset number of nodes in the front after sorting are determined as faulty nodes.
6. The fault node location method according to claim 5, characterized in that: Determining the transition probability matrix based on the causal graph includes: Constructing an initialized transition probability matrix, wherein the initialized transition probability matrix has a target number of row data and a target number of column data, wherein the target number is the same as the number of nodes in the network; Determine the transition probability between each node in the network according to the causal graph, the transition probability includes: forward transition probability, self-stay probability and backward transition probability, wherein the forward transition probability is used to represent the transition probability from the result node to the root node in the causal graph, the self-stay probability is used to represent the probability of the node in the causal graph transferring to itself, and the backward transition probability is used to represent the transition probability from the cause node to the result node in the causal graph; The transition probability is filled into the initialized transition probability matrix, and the initialized transition probability matrix filled with the transition probability is standardized to obtain a final transition probability matrix, wherein the sum of each row of data in the standardized transition probability matrix is 1, and the sum of each column of data is 1.
7. The fault node location method according to claim 1, characterized in that: After locating the faulty node, the method further includes: A causal link is determined based on the fault node, and an alarm work order is dispatched to the nodes in the causal link, wherein the causal link includes the fault node and a conduction path of the fault node, and the information in the alarm work order includes at least one of the following: alarm title, alarm resource, and alarm indicator.
8. A fault node locating device, characterized in that: include: A first acquisition module is used to acquire network topology information and generate a system topology map based on the network topology information, wherein the network topology information includes information of each node in the network and relationship information between the nodes, and the nodes include device nodes and cloud application nodes; A second acquisition module is used to acquire monitoring indicator data, wherein the monitoring indicator data is used to quantitatively evaluate the performance and status of each of the nodes, including node alarm data; A generating module, used to generate a cause-effect diagram according to the system topology diagram and the monitoring indicator data, wherein the cause-effect diagram is used to represent the influence relationship between each node in the network; A determination module is used to determine a root cause graph based on the cause-effect graph when an abnormal node exists in the network, and locate the faulty node based on the root cause graph, wherein the abnormal node is a node in the network where an alarm occurs, the root cause graph is used to indicate the probability of fault transmission between the nodes, and the faulty node is a node that causes the abnormal node to alarm.
9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the fault node location method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the fault node location method according to any one of claims 1 to 7 when running.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the fault node location method described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Internet of Things abnormal root cause positioning method and device based on causal learning
CN115150250A
Fault root cause positioning method and device, electronic equipment and readable storage medium
CN115514627A
Fault root cause positioning method, device and equipment and computer storage medium
CN116560882A
Fault root cause positioning method and device, equipment and storage medium
CN117201289A
Artificial intelligence with dynamic causal model for failure analysis in mobile communication network
WO2022149149A1
Cited By
Alarm information processing method and device and nonvolatile storage medium
CN119520230A
Service end-to-end monitoring method and system based on eBPF
CN119652743A
Cloud network fault detection method and device, equipment, storage medium and program product
CN119835147A
Vehicle fault diagnosis method and device, electronic equipment and storage medium
CN120560231A
Vehicle fault diagnosis method, device, electronic device and storage medium
CN120560231B
