Information processing device, information processing method, and program

Abstract

To appropriately identify a node of a cause of an alert.SOLUTION: A storage section 11 stores configuration information indicating a relation between nodes in an information processing system 20 and alert information indicating an alert generated in the information processing system 20. A processing section 12 calculates a basic score corresponding to the alert for each of a plurality of nodes on the basis of the alert information. The processing section 12 calculates a propagation score obtained by multiplying a basic score of an adjacent node adjacent to a node, which is identified on the basis of the configuration information for each of the plurality of nodes, by a weighting coefficient corresponding to a dependence between the node and the adjacent node for each pair of the node and the adjacent node. The processing section 12 calculates a failure score obtained by totaling the basic score of the node and the propagation score corresponding to the pair of the node and the adjacent node for each of the plurality of nodes. The processing section 12 identifies a first node of a cause of the alert among the plurality of nodes on the basis of the failure score of each of the plurality of nodes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Currently, information processing systems are in operation in which numerous communication devices, physical machines, virtual machines, applications, and the like work together. Because these devices work together, they can be expressed as a topology structure, and therefore elements of the information processing system, such as communication devices, physical machines, virtual machines, and applications, can be expressed as nodes. In other words, a node corresponds to any element in the information processing system. By monitoring the nodes, a monitoring system can detect an alert for the node if an abnormality occurs in the node. An alert is information that notifies of an abnormality. Therefore, in the operation and management of an information processing system, alerts output by nodes may be monitored.

[0003] For example, a message output device has been proposed that makes it easier for administrators to understand abnormalities by selectively outputting messages related to elements included in a system based on the relationships between the elements included in the system.

[0004] There has also been proposed a management computer that generates a countermeasure procedure plan for changing the status of components in a computer system when a problem occurs in the computer system. There has also been a proposal for an information processing device that outputs log messages for each of multiple tasks executed in a distributed stream data processing infrastructure. The proposed information processing device acquires a log score for each task based on an index related to the likelihood of a failure occurring. The information processing device outputs the log messages for the task at an output frequency based on the log score. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2019-40515 [Patent Document 2] International Publication No. 2017 / 26017 [Patent Document 3] Japanese Patent Publication No. 2020-197853 Summary of the Invention [Problem to be solved by the invention]

[0006] When multiple nodes operate in cooperation with one another, an abnormality occurring in one node may affect other nodes and cause abnormalities in those other nodes. Therefore, when an alert occurs in an information processing system, it is not easy to identify the node that is the root cause of the alert. In one aspect, the present invention aims to appropriately identify the node that is the cause of the alert. [Means for solving the problem]

[0007] In one aspect, an information processing device is provided. The information processing device includes a storage unit and a processing unit. The storage unit stores configuration information indicating relationships between multiple nodes included in an information processing system and alert information indicating alerts that have occurred in the information processing system. The processing unit calculates a basic score corresponding to the alert for each of the multiple nodes based on the alert information. The processing unit calculates a propagation score for each pair of a node and an adjacent node by multiplying the basic score of an adjacent node adjacent to the node, which is identified for each of the multiple nodes based on the configuration information, by a weighting coefficient corresponding to the dependency relationship between the node and the adjacent node. The processing unit calculates a fault score for each of the multiple nodes by adding the basic score of the node and the propagation score corresponding to the pair of the node and the adjacent node. The processing unit identifies a first node among the multiple nodes that is the cause of the alert based on the fault scores of each of the multiple nodes.

[0008] In one aspect, an information processing method is provided. In another aspect, a program is provided. [Effects of the Invention]

[0009] On the one hand, it allows proper identification of the node that caused the alert. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. [Figure 2] FIG. 10 illustrates an example of an information processing system according to a second embodiment. [Figure 3] FIG. 10 illustrates an example of hardware of a failure point estimation server. [Figure 4] FIG. 10 illustrates an example of functions of a failure location estimation server. [Figure 5] FIG. 1 illustrates an example of a monitored system. [Figure 6] FIG. 10 is a diagram illustrating an example of a method for calculating a disability score. [Figure 7] 10A and 10B are diagrams illustrating examples of a node table and an edge table. [Figure 8] FIG. 10 is a diagram illustrating an example of an alert table. [Figure 9] FIG. 10 is a diagram illustrating an example of a basic score coefficient table. [Figure 10] FIG. 10 is a diagram illustrating an example of a propagation score coefficient table. [Figure 11] FIG. 10 is a diagram illustrating an example of a score table. [Figure 12] FIG. 10 is a diagram illustrating a display example of a failure location estimation result screen. [Figure 13] 10 is a flowchart illustrating an example of processing by a failure point estimation server. [Figure 14] FIG. 10 is a diagram showing a modified example of the basic score coefficient table. [Figure 15] FIG. 10 is a diagram showing a first modified example of a propagation score coefficient table. [Figure 16] FIG. 10 is a diagram showing a second modified example of the propagation score coefficient table. [Figure 17] FIG. 10 is a diagram illustrating an example of another method for calculating a disability score. [Figure 18] FIG. 10 is a diagram illustrating another example of a monitored system. [Figure 19] FIG. 10 is a diagram illustrating an example of calculating a disability score. [Figure 20] FIG. 10 is a diagram illustrating an example of a comparative example of calculating a disability score. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present embodiment will be described below with reference to the drawings. [First embodiment] A first embodiment will be described.

[0012] FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. The information processing device 10 is connected to an information processing system 20. The information processing system 20 includes a plurality of nodes. Each of the plurality of nodes is any of a communication device, a physical machine, a virtual machine, an application, etc. The information processing device 10 assists in identifying the node that is the cause of an alert that has occurred in the information processing system 20. Note that the information processing device 10 may be included in the information processing system 20.

[0013] The information processing device 10 has a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile storage device such as a random access memory (RAM), or a non-volatile storage device such as a hard disk drive (HDD) or flash memory. The processing unit 12 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The processing unit 12 may be a processor that executes a program. The term "processor" may include a collection of multiple processors (multiprocessor).

[0014] The storage unit 11 stores configuration information. The configuration information indicates the relationships between multiple nodes included in the information processing system 20. The relationships between nodes include a node being executed using the resources of another node, or a node being connected to another node in order to operate. For example, the relationships between nodes are represented by a graph connecting each node with edges. The smaller the number of edges separating two nodes, the stronger the relationship between the two nodes. If the number of edges separating two nodes is equal to or less than a threshold a, the two nodes are said to be adjacent to each other. a is an integer greater than or equal to 1. For example, a=1.

[0015] The configuration information also includes information indicating the dependency relationships between nodes. Each of the multiple nodes included in the information processing system 20 may have the following dependency relationships with other nodes. For example, an application is executed by a virtual machine. In this case, the application depends on the virtual machine. The virtual machine is depended on the application. Also, for example, a virtual machine is executed by a physical machine. In this case, the virtual machine depends on the physical machine. The physical machine is depended on the virtual machine. Also, for example, a physical machine is connected to an L2 (Layer 2) switch to communicate with other physical machines. In this case, the physical machine depends on the L2 switch. The L2 switch is depended on the physical machine. Furthermore, for example, an L2 switch is connected to a router to communicate with an upper network. In this case, the L2 switch depends on the router. The router is depended on the L2 switch.

[0016] These dependencies are represented, for example, by directed edges connecting nodes in the above graph. That is, the starting node of a directed edge depends on the ending node. The ending node of a directed edge is dependent on the starting node. In one example, information processing system 20 has nodes 21 and 22. The node ID (IDentifier) ​​of node 21 is n1. The node ID of node 22 is n2. Node 21 depends on node 22. That is, node 22 is dependent on node 21. In this case, the configuration information includes, for example, information indicating nodes 21 and 22, and information on the directed edge starting from node 21 and ending at node 22. However, information processing system 20 may include three or more nodes.

[0017] The storage unit 11 stores alert information. The alert information indicates alerts that have occurred in the information processing system 20 during a predetermined period of time. The alerts can be output by each of multiple nodes. The predetermined period of time may be determined, for example, as a period of time during which alerts have continued to occur at time intervals shorter than a predetermined time interval. This is because a group of alerts that have occurred consecutively at time intervals shorter than a fixed time interval are likely to have a common cause. In this case, the fixed time interval is determined in advance depending on the information processing system 20.

[0018] The processing unit 12 identifies the node that is the root cause of a group of alerts generated in the information processing system 20 by scoring and evaluating the contribution of each node to the alert. An index indicating the contribution of each node to the alert is called a fault score. For example, the higher the fault score of a node, the greater the contribution of that node to the alert. Therefore, the higher the fault score of a node, the more likely it is that that node is the cause of the alert.

[0019] The processing unit 12 calculates a basic score for each of the multiple nodes according to the alert based on the alert information. The basic score is a score that is the basis for calculating the fault score. In one example, the processing unit 12 calculates a basic score for a node that has output an alert according to the number of alerts. For example, the processing unit 12 increases the basic score of a node as the number of alerts that the node has output increases. The processing unit 12 assigns a basic score of "0" to nodes that have not output an alert.

[0020] For example, when alerts occur in nodes 21 and 22 during a predetermined period, processing unit 12 calculates a basic score V1 for node 21 according to the number of alerts in node 21, and calculates a basic score V2 for node 22 according to the number of alerts in node 22. Table 11a is information that holds the basic scores V1 and V2 for nodes 21 and 22, respectively. Table 11a is stored in storage unit 11.

[0021] The processing unit 12 calculates a propagation score for each pair of a node and its adjacent node by multiplying the basic score of the adjacent node, which is identified based on the configuration information, by a weighting coefficient according to the dependency relationship between the node and the adjacent node. Here, the propagation score is a score for reflecting the influence of the basic score of the adjacent node on the failure score of the node of interest. The weighting coefficient is a coefficient that determines the degree to which the influence of the basic score of the adjacent node is reflected on the failure score of the node of interest.

[0022] The weighting coefficient is α when the adjacent node depends on the node in question. The weighting coefficient is β when the adjacent node is dependent on the node in question, i.e., when the node in question depends on the adjacent node. Both α and β are positive real numbers. Also, α>β. This is because, of two adjacent nodes, the node that is dependent is more likely to be the location of the failure. Also, there may be multiple adjacent nodes for a node of interest. In this case, the processing unit 12 calculates a propagation score to be applied to each adjacent node for the node of interest.

[0023] For example, let the aforementioned threshold a=1. In this case, the processing unit 12 identifies node 22 as an adjacent node adjacent to node 21 based on the configuration information. Node 22 is dependent on node 21. Therefore, the processing unit 12 calculates the propagation score of node 22 for node 21 as β*V2. Furthermore, the processing unit 12 identifies node 21 as an adjacent node adjacent to node 22 based on the configuration information. Node 21 is dependent on node 22. Therefore, the processing unit 12 calculates the propagation score of node 21 for node 22 as α*V1. Table 11b is information that holds the propagation score β*V2 of node 22 for node 21 and the propagation score α*V1 of node 21 for node 22. Table 11b is stored in the storage unit 11.

[0024] Then, the processing unit 12 calculates a fault score for each of the multiple nodes by adding the base score of the node and the propagation score corresponding to the pair of the node and its adjacent node. For example, the processing unit 12 calculates the fault score of node 21 as V1+β*V2 based on tables 11a and 11b. The processing unit 12 also calculates the fault score of node 22 as V2+α*V1. Table 11c is information that holds the fault score V1+β*V2 of node 21 and the fault score V2+α*V1 of node 22. Table 11c is stored in the storage unit 11. In this way, the propagation score allows the influence of an event that occurred in an adjacent node to be appropriately reflected in the fault score of the node of interest.

[0025] The processing unit 12 identifies a first node that is the cause of the alert from among the multiple nodes based on the failure scores of each of the multiple nodes. For example, in the above example, the processing unit 12 compares the failure scores of nodes 21 and 22, and identifies the node with the larger failure score as the first node that is the cause of the alert. The first node is the node that is estimated to be the root cause of the alert, and corresponds to the failure location.

[0026] In one example, the number of alerts generated in a node in question is used as the basic score for that node. For example, suppose the number of alerts generated in node 21 is 1 and the number of alerts generated in node 22 is 1. In this case, V1 = V2 = 1. Also, in one example, the propagation scores are α = 1 and β = 0.5. In this case, the processing unit 12 calculates the fault score for node 21 as V1 + β * V2 = 1 + 0.5 * 1 = 1.5. Also, the processing unit 12 calculates the fault score for node 22 as V2 + α * V1 = 1 + 1 * 1 = 2. Therefore, the processing unit 12 identifies node 22 as the cause of the alerts generated in nodes 21 and 22.

[0027] As described above, the information processing device 10 calculates a basic score corresponding to the alert for each of the multiple nodes based on the alert information. A propagation score is calculated for each pair of a node and its adjacent node by multiplying the basic score of the adjacent node adjacent to the node by a weighting coefficient corresponding to the dependency relationship between the node and its adjacent node. A fault score is calculated for each of the multiple nodes by adding the basic score of the node and the propagation score corresponding to the pair of the node and its adjacent node. Based on the fault scores of each of the multiple nodes, the node with the highest fault score among the multiple nodes is identified, and the root cause of the alert, i.e., the first node that is the cause of the alert, is identified.

[0028] This allows you to properly identify the node that caused the alert. Here, in the information processing system 20, many nodes work in cooperation with each other. Therefore, when a failure occurs in the information processing system, the failure propagates and many alerts are generated, making it difficult to identify the node that is the root cause of the failure.

[0029] For example, when two or more nodes each output an alert, it can be difficult to identify which node is the root cause. In calculating the above fault scores, it is possible to simply use a uniform weighting factor, such as α = β = 0.5, without considering the dependency between the node and its neighboring nodes. However, in this case, the fault score will be the same for both nodes 21 and 22, making it difficult to identify the first node that caused the alert. For example, when V1 = V2 = 1 and α = β = 0.5, the fault scores for nodes 21 and 22 will both be 1.5, preventing proper narrowing down of the first node based on the fault scores.

[0030] Therefore, the information processing device 10 calculates the fault score of each node using a weighting coefficient according to the dependency relationship between the node and adjacent nodes, thereby making it possible to appropriately identify the first node that is the cause of the alert based on the fault score of each node.

[0031] Furthermore, for example, if the node causing the alert is a node other than the node that output the alert, it may not be possible to identify the node causing the alert from the source of the alert alone. In response to this, the information processing device 10 can calculate a fault score for each node that output an alert within a predetermined period and each node that did not output an alert. This increases the likelihood that the information processing device 10 can identify the node that caused the underlying cause of the alert from nodes other than the source of the alert. For example, even if a node other than the source of the alert is operating abnormally, the alert may not be output due to a flaw in the alert output settings, such as an inappropriate setting for the threshold for the amount of resource usage used to output the alert. In such cases, an abnormality in a node other than the source of the alert may be affecting other nodes. The information processing device 10 can appropriately identify such nodes other than the source of the alert as the underlying cause of the alert, i.e., the location of the fault.

[0032] The failure location estimating server 100 outputs information indicating the identified first node, thereby assisting the user in identifying the node that caused the alert. As a result, the failure location estimating server 100 can assist the user in quickly responding to the failure. The following provides a more specific example and explains the functions of the information processing device 10 in more detail.

[0033] [Second embodiment] Next, a second embodiment will be described. FIG. 2 illustrates an example of an information processing system according to the second embodiment.

[0034] The information processing system of the second embodiment includes a failure location estimation server 100, a monitored system 200, a configuration management server 300, and an anomaly detection server 400. The failure location estimation server 100, the monitored system 200, the configuration management server 300, and the anomaly detection server 400 are connected to a network 50.

[0035] The failure location estimating server 100 estimates the node that caused the alert, i.e., the failure location, based on alerts that occurred in the monitored system 200 during a predetermined period, and presents the estimate to the user. Specifically, for alerts that occurred in the monitored system 200, the failure location estimating server 100 evaluates a failure score, which is an index representing the contribution of each node in the monitored system 200 to the alert. For example, the higher the failure score of a node, the greater the contribution of that node to the alert. In other words, the higher the failure score of a node, the higher the possibility that that node is the cause of the alert. The failure location estimating server 100 identifies the node that is the root cause of the alert based on the failure score of each node. The failure location estimating server 100 is an example of the information processing device 10 of the first embodiment.

[0036] The monitored system 200 is a system to be monitored by the failure point estimation server 100, the configuration management server 300, and the anomaly detection server 400. The monitored system 200 includes multiple nodes such as communication devices, physical machines, virtual machines, and applications. The applications may be executed as containers. The monitored system 200 is an example of the information processing system 20 of the first embodiment.

[0037] The configuration management server 300 collects information on a plurality of nodes in the monitored system 200 , generates configuration information indicating the relationships between the nodes based on the collected information, and provides the configuration information to the failure point estimation server 100 .

[0038] The anomaly detection server 400 collects alerts generated in the monitored system 200 and provides alert information indicating the collected alerts to the failure location estimation server 100. An alert is a message reporting an event that has become abnormal due to the influence of a failure. For example, each node identifies an event in which the CPU usage rate or memory usage rate exceeds a threshold, or an event detected by anomaly detection, as an abnormal event and generates an alert.

[0039] FIG. 3 is a diagram illustrating an example of hardware of a failure point estimation server. The failure location estimation server 100 has a CPU 101, a RAM 102, a HDD 103, a GPU (Graphics Processing Unit) 104, an input interface 105, a media reader 106, and a NIC (Network Interface Card) 107. The CPU 101 is an example of the processing unit 12 of the first embodiment. The RAM 102 or the HDD 103 is an example of the storage unit 11 of the first embodiment. The failure location estimation server 100 may be a computer having the CPU 101 and the RAM 102.

[0040] The CPU 101 is a processor that executes program instructions. The CPU 101 loads at least a portion of the programs and data stored in the HDD 103 into the RAM 102 and executes the programs. The CPU 101 may include multiple processor cores. The failure location estimation server 100 may also have multiple processors. The processing described below may be executed in parallel using multiple processors or processor cores. A set of multiple processors may also be called a "multiprocessor" or simply a "processor."

[0041] The RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the CPU 101 and data used for calculations by the CPU 101. The failure location estimating server 100 may be provided with a type of memory other than RAM, or may be provided with multiple memories.

[0042] The HDD 103 is a non-volatile storage device that stores software programs such as an OS (Operating System), middleware, and application software, as well as data. The failure location estimation server 100 may also be provided with other types of storage devices such as flash memory or SSD (Solid State Drive), or may be provided with multiple non-volatile storage devices.

[0043] The GPU 104 outputs an image to a display 51 connected to the failure location estimation server 100 in accordance with an instruction from the CPU 101. The display 51 may be any type of display, such as a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD: Liquid Crystal Display), a plasma display, or an organic EL (OEL: Organic Electro-Luminescence) display.

[0044] The input interface 105 acquires input signals from input devices 52 connected to the failure location estimating server 100 and outputs them to the CPU 101. The input devices 52 may be pointing devices such as a mouse, touch panel, touch pad or trackball, a keyboard, a remote controller, or a button switch. In addition, multiple types of input devices may be connected to the failure location estimating server 100.

[0045] The medium reader 106 is a reading device that reads programs and data recorded on the recording medium 53. For example, a magnetic disk, an optical disk, a magneto-optical disk (MO: Magneto-Optical disk), a semiconductor memory, etc. can be used as the recording medium 53. Magnetic disks include flexible disks (FD: Flexible Disks) and HDDs. Optical disks include compact discs (CDs) and digital versatile discs (DVDs).

[0046] The medium reader 106 copies, for example, programs and data read from the recording medium 53 to another recording medium such as the RAM 102 or the HDD 103. The read programs are executed by, for example, the CPU 101. The recording medium 53 may be a portable recording medium, which may be used to distribute programs and data. The recording medium 53 and the HDD 103 may also be referred to as computer-readable recording media.

[0047] The NIC 107 is an interface that is connected to the network 50 and communicates with other computers via the network 50. The NIC 107 is connected to a communication device such as a switch or a router by a cable. The NIC 107 may also be a wireless communication interface.

[0048] The physical machines included in the monitored system 200 , the configuration management server 300 , and the anomaly detection server 400 are also realized by the same hardware as the failure point estimation server 100 . FIG. 4 illustrates an example of functions of the failure location estimation server.

[0049] The failure location estimation server 100 includes a configuration information storage unit 110, an alert information storage unit 120, a coefficient information storage unit 130, a score information storage unit 140, a configuration information management unit 151, an alert information acquisition unit 152, a basic score calculation unit 153, a propagation score coefficient calculation unit 154, a propagation score calculation unit 155, a failure score calculation unit 156, and a failure location output unit 157. The configuration information storage unit 110, the alert information storage unit 120, the coefficient information storage unit 130, and the score information storage unit 140 use storage areas of the RAM 102 and the HDD 103. The configuration information management unit 151, the alert information acquisition unit 152, the basic score calculation unit 153, the propagation score coefficient calculation unit 154, the propagation score calculation unit 155, the failure score calculation unit 156, and the failure location output unit 157 are realized by the CPU 101 executing programs stored in the RAM 102.

[0050] The configuration information storage unit 110 stores configuration information including the nodes in the monitored system 200 and the relationships between the nodes. The alert information storage unit 120 stores alert information in the monitored system 200.

[0051] The coefficient information storage unit 130 stores coefficient information indicating the base score coefficient used in calculating the base score and the propagation score coefficient used in calculating the propagation score. The score information storage unit 140 stores score information indicating the base score, propagation score, and fault score calculated for each node.

[0052] The configuration information management unit 151 acquires configuration information from the configuration management server 300 and stores it in the configuration information storage unit 110. The configuration information includes information indicating the relationships between the nodes included in the monitored system 200. The configuration information also includes information indicating the amount of resources held by a node allocated to another node. For example, the configuration information includes information indicating the amount of resources of a virtual machine allocated to an application, the amount of resources of a physical machine allocated to a virtual machine, etc.

[0053] The alert information acquisition unit 152 acquires alert information from the anomaly detection server 400 and stores it in the alert information storage unit 120 . The basic score calculation unit 153 calculates a basic score for each node based on the configuration information stored in the configuration information storage unit 110, the alert information stored in the alert information storage unit 120, and the coefficient information stored in the coefficient information storage unit 130. The basic score calculation unit 153 stores the calculated basic score in the score information storage unit 140.

[0054] The propagation score coefficient calculation unit 154 calculates the propagation score coefficient of an adjacent node for a certain node based on the configuration information stored in the configuration information storage unit 110, and stores the calculated coefficient in the coefficient information storage unit 130. Here, the propagation score coefficient corresponds to the weighting coefficient in the first embodiment. The propagation score coefficient is used to calculate the propagation score of an adjacent node for a certain node. The propagation score coefficient is determined based on the dependency relationship between the node and the adjacent node.

[0055] The propagation score calculation unit 155 calculates the propagation score of an adjacent node for a certain node based on the coefficient information stored in the coefficient information storage unit 130 and the score information stored in the score information storage unit 140. The propagation score is calculated for each pair of a node of interest and the adjacent node. The propagation score calculation unit 155 stores the calculated propagation score in the score information storage unit 140.

[0056] The fault score calculation unit 156 calculates a fault score for each node based on the basic score of each node stored in the score information storage unit 140 and the propagation score of the adjacent node for that node, and stores the calculated fault score in the score information storage unit 140.

[0057] The failure location output unit 157 identifies the node corresponding to the failure location and outputs information about the node based on the failure score of each node stored in the score information storage unit 140. Specifically, the failure location output unit 157 identifies the node with the largest failure score as the node corresponding to the failure location.

[0058] For example, the failure location output unit 157 visualizes the node that caused the alert, i.e., the node corresponding to the failure location, based on the configuration information stored in the configuration information storage unit 110 and the information on the failure score of each node stored in the score information storage unit 140. Specifically, the failure location output unit 157 displays a failure location estimation result screen on the display 51. The failure location estimation result screen is a screen that shows the estimation result of the failure location. The failure location estimation result screen displays, for example, each node in the monitored system 200 and the calculation result of the failure score of each node, and also highlights the node identified as the failure location.

[0059] FIG. 5 is a diagram illustrating an example of a monitored system. The monitored system 200 includes, for example, a router 201, switches 211 and 212, physical machines 221 and 222, virtual machines 231 and 232, and applications 241 and 242.

[0060] The switches 211 and 212 are connected to the router 201. The switches 211 and 212 are, for example, L2 switches. Access from the switch 211 to the upper network goes through the router 201. Therefore, the switch 211 depends on the router 201. The router 201 depends on the switch 211. Similarly, the switch 212 depends on the router 201. The router 201 depends on the switch 212.

[0061] The physical machine 221 is connected to the switch 211. Communication between the physical machine 221 and other physical machines goes through the switch 211. Therefore, the physical machine 221 depends on the switch 211. The switch 211 is dependent on the physical machine 221. The physical machine 222 is connected to the switch 212. The physical machine 222 depends on the switch 212. The switch 212 is dependent on the physical machine 222.

[0062] The virtual machine 231 runs on the physical machine 221. Therefore, the virtual machine 231 depends on the physical machine 221. The physical machine 221 is depended on by the virtual machine 231. The virtual machine 232 runs on the physical machine 222. Therefore, the virtual machine 232 depends on the physical machine 222. The physical machine 222 is depended on by the virtual machine 232.

[0063] The application 241 is executed by the virtual machine 231. Therefore, the application 241 depends on the virtual machine 231. The virtual machine 231 is depended on by the application 241. The application 242 is executed by the virtual machine 232. Therefore, the application 242 depends on the virtual machine 232. The virtual machine 232 is depended on by the application 242.

[0064] The application 241 cooperates with the application 242. For example, the application 241 transmits a request to the application 242 and executes processing in accordance with a response from the application 242. Therefore, the application 241 depends on the application 242. The application 242 is dependent on the application 241.

[0065] The relationships between the above nodes are represented, for example, by a graph in which edges connect a dependent node and a dependent node. A node that depends on another node is considered to be higher, and a node that is dependent on another node is considered to be lower. In this case, in the example of Figure 5, applications 241 and 242 are nodes at the highest level. Router 201 is a node at the lowest level.

[0066] Here, the node name of router 201 is "Router". The node name of switch 211 is "SW1". The node name of switch 212 is "SW2". The node name of physical machine 221 is "PM1". The node name of physical machine 222 is "PM2". The node name of virtual machine 231 is "VM1". The node name of virtual machine 232 is "VM2". The node name of application 241 is "appl1". The node name of application 242 is "appl2".

[0067] When a failure occurs in a dependent node, the dependent node is often affected, but a failure in a dependent node may also cause an alert to be issued in the dependent node. For example, if the memory of the physical machine 221 becomes overloaded, the virtual machine 231 also becomes overloaded and an alert is issued. On the other hand, if the virtual machine 231 executes a large number of jobs without any restrictions, the service of the physical machine 221 may become slow and an alert may be issued.

[0068] FIG. 6 is a diagram illustrating an example of a method for calculating a disability score. In Figure 6, arrows connecting nodes indicate directed edges. Directed edges indicate dependencies between nodes. The starting node of a directed edge depends on the ending node. The ending node of a directed edge depends on the starting node. In this example, the threshold a for determining adjacent nodes is set to 1. In this case, if the number of edges separating two nodes is 1, the two nodes are determined to be adjacent to each other. More specifically, if the number of directed edges separating nodes N1 and N2 is 1, that is, if nodes N1 and N2 are directly connected by a directed edge, node N2 is an adjacent node of node N1, and node N1 is an adjacent node of node N2.

[0069] For example, the failure point estimating server 100 calculates the failure score of the virtual machine 231 in response to an alert that has occurred in the switch 211, the physical machine 221, and the applications 241 and 242 as follows: The adjacent nodes of the virtual machine 231 are the application 241 and the physical machine 221.

[0070] First, the basic score calculation unit 153 calculates the basic score of each node in the monitored system 200. The basic score of the application 241 is x. The basic score of the virtual machine 231 is y. The basic score of the physical machine 221 is z. In FIG. 6, the basic scores of nodes other than the virtual machine 231 of interest and the adjacent nodes of the virtual machine 231 are omitted from the illustration.

[0071] In one example, the basic score calculation unit 153 sets the number of alerts issued by the node in question as the basic score. For example, if the number of alerts issued by the node in question is 12, the basic score is 12. The basic score calculation unit 153 may calculate the basic score by weighting the alerts according to their importance (ERROR, Warning, Information, etc.). For example, the basic score calculation unit 153 may weight the importance of the alerts as follows: ERROR: 3, Warning: 2, and Information: 1. For example, if the number of alerts is ERROR: 12, Warning: 11, and Information: 13, respectively, the basic score calculation unit 153 may calculate the basic score of the node in question as 3*12 + 2*11 + 1*13 = 71.

[0072] Furthermore, the basic score calculation unit 153 may adopt a self score εy obtained by multiplying the basic score y calculated for the virtual machine 231 in response to the alert by a basic score coefficient ε according to the attributes of the virtual machine 231 as the basic score of the virtual machine 231. εy is a corrected basic score obtained by correcting the basic score y with the basic score coefficient ε. Note that the basic score coefficient can be set not only for virtual machines, but also for attributes of all nodes such as applications and physical machines, as will be described later. FIG. 6 shows an example in which the basic score of the virtual machine 231 that is the target for calculating the failure score is set to εy.

[0073] The propagation score coefficient of the application 241 for the virtual machine 231 is α. The propagation score coefficient of the physical machine 221 for the virtual machine 231 is β. Both α and β are positive real numbers. Furthermore, α>β. This is because, of two adjacent nodes, the node on which the other is depended is more likely to be the location of a failure.

[0074] The propagation score calculation unit 155 calculates the propagation score of the adjacent node for the node of interest by multiplying the propagation score coefficient of the adjacent node for the node of interest by the basic score of the adjacent node. Specifically, the propagation score calculation unit 155 calculates the propagation score αx of the application 241 for the virtual machine 231. The propagation score calculation unit 155 also calculates the propagation score βz of the physical machine 221 for the virtual machine 231.

[0075] Then, the fault score calculation unit 156 calculates the sum of the basic score of the node of interest and the propagation scores of the adjacent nodes as the fault score of the node of interest. For example, fault score = basic score coefficient (ε) * basic score (y) + propagation score coefficient (α) of adjacent upper layer * basic score (x) of adjacent upper layer + propagation score coefficient (β) of adjacent lower layer * basic score (z) of adjacent lower layer. That is, for example, the fault score calculation unit 156 calculates the fault score εy + αx + βz for the virtual machine 231.

[0076] FIG. 7 illustrates an example of a node table and an edge table. The node table 111 includes fields for node name and node ID. A node name is registered in the node name field. A node ID is registered in the node ID field. For example, the node table 111 has a record with a node name of "appl1" and a node ID of "n1". This record indicates that the node with the node name "appl1", i.e., the node ID of the application 241, is "n1". The node table 111 also has records for the node names and node IDs of nodes other than the application 241.

[0077] The edge table 112 includes items for an edge ID, a start point, and an end point. The edge ID item registers an edge ID. The start point item registers a node ID of a node at the start point of an edge. The end point item registers a node ID of a node at the end point of an edge. For example, the edge table 112 has a record with an edge ID "e1", a start point "n1", and an end point "n2". This record indicates that the node ID of the start point node of the edge identified by the edge ID "e1" is "n1", and the node ID of the end point node of the edge is "n2". In other words, this record indicates that the adjacent node of the application 241 is the virtual machine 231, and that the adjacent node of the virtual machine 231 is the application 241. This record also indicates a dependency relationship in which the application 241 depends on the virtual machine 231. This dependency relationship can also be said to be a dependency relationship in which the virtual machine 231 is dependent on the application 241. The edge table 112 also has records indicating edges with edge IDs other than the edge ID "e1."

[0078] The node table 111 and the edge table 112 are examples of configuration information stored in the configuration information storage unit 110. FIG. 8 is a diagram illustrating an example of an alert table.

[0079] The alert table 121 includes fields for time, node ID, importance, and alert content. The time field stores the time when the alert occurred. The node ID field stores the node ID of the node where the alert occurred. The importance field stores the importance of the alert. For example, alert importance can be "ERROR," "Warning," or "Information." The alert content field stores the message of the alert.

[0080] For example, the alert table 121 has a record of time "t1", node ID "n1", severity "Warning", and alert content "...". This record indicates that an alert with severity "Warning" occurred in the node with node ID "n1", i.e., application 241, at time t1. The alert table 121 also has records indicating other alerts. The alert table 121 is an example of alert information stored in the alert information storage unit 120.

[0081] FIG. 9 is a diagram illustrating an example of a basic score coefficient table. The basic score coefficient table 131 is stored in the coefficient information storage unit 130. The basic score coefficient table 131 includes a basic score coefficient field. A basic score coefficient is registered in the basic score coefficient field. In the example of FIG. 9, the basic score coefficient is 1.0 for all nodes. The basic score coefficient is used to calculate the self score of each node. For example, the self score of a node may be calculated by multiplying the number of alerts issued by the node, or the sum of the number of alerts and a weight according to the importance of the alert, by the basic score coefficient. The basic score coefficient may be a value other than 1.0. Furthermore, as will be described later, the basic score coefficient may be a value according to the attributes of the node, for example.

[0082] FIG. 10 is a diagram illustrating an example of a propagation score coefficient table. The propagation score coefficient table 132 is stored in the coefficient information storage unit 130. The propagation score coefficient table 132 includes fields for dependency, propagation direction, and propagation score coefficient. The field for dependency stores information indicating the dependency between nodes. The field for propagation direction stores the direction in which the propagation score of an adjacent node is applied, i.e., the propagation direction. The field for propagation score coefficient stores a propagation score coefficient according to the propagation direction.

[0083] For example, information indicating that "node1" depends on "node2" is registered in the dependency relationship field of the propagation score coefficient table 132. In the example of Fig. 10, the dependency relationship in which "node1" depends on "node2" is expressed as "node1->node2".

[0084] Furthermore, the propagation score coefficient table 132 has a record for the dependency relationship "node1->node2", with a propagation direction of "node1->node2" and a propagation score coefficient of "1.0". This record indicates that when "node1" depends on "node2", the propagation score coefficient of "node1" with respect to "node2" is set to "1.0".

[0085] Furthermore, the propagation score coefficient table 132 has a record with a propagation direction of "node1<-node2" and a propagation score coefficient of "0.7" for the dependency relationship "node1->node2." This record indicates that when "node1" depends on "node2," the propagation score coefficient of "node2" for "node1" is set to "0.7."

[0086] In the propagation score coefficient table 132, the propagation score coefficients are set so that the propagation score coefficient α of the depending node for the depended-on node is greater than the propagation score coefficient β of the depended-on node for the depending node.

[0087] The basic score coefficient table 131 and the propagation score coefficient table 132 are examples of coefficient information stored in the coefficient information storage unit 130. FIG. 11 is a diagram illustrating an example of a score table.

[0088] The score table 141 is stored in the score information storage unit 140. The score table 141 includes the following items: node name, node ID, number of alerts by weight, base score, own score, upper node ID, lower node ID, resource propagation score coefficient, propagation score of upper node, propagation score of lower node, and fault score.

[0089] The node name is registered in the node name field. The node ID is registered in the node ID field. The number of alerts by weight field is registered with the number of alerts by weight. For example, importance levels such as "ERROR", "Warning", and "Information" are associated with weights of "3", "2", and "1", respectively. The basic score field is registered with the basic score of the node in question. In the example of score table 141, the basic score is calculated by multiplying the number of alerts that occurred in the node in question by a weight according to the importance of the alert, and then adding up the results. The own score field is registered with the own score of the node in question. The node ID of the upper node is registered in the upper node ID field. The upper node is the adjacent node immediately above the node in question. The node ID of the lower node is registered in the lower node ID field. The lower node is the adjacent node immediately below the node in question.

[0090] The resource propagation score coefficient field registers a resource propagation score coefficient. The resource propagation score coefficient is a propagation score coefficient calculated by the propagation score coefficient calculation unit 154. When a node in question uses a resource of a lower node, the propagation score coefficient calculation unit 154 calculates the resource propagation score coefficient according to the proportion of the resource amount used by the node in the resource amount of the lower node.

[0091] For example, the resource propagation score coefficient is calculated using the formula: Resource propagation score coefficient = (resource allocation coefficient) * {(number of CPU cores used by the node) / (number of CPU cores owned by the lower node)}. In this way, if the node is using a large amount of CPU resources, the node is more likely to be affected by a failure in a lower node, and the resource propagation score coefficient will be large. Note that a preset fixed value is used as the resource allocation coefficient. In one example, the resource allocation coefficient is 1.0.

[0092] Note that the score table 141 shows an example in which resource propagation score coefficients are registered in the score table 141. However, the propagation score coefficient calculation unit 154 may store the resource propagation score coefficients of adjacent nodes (lower nodes) of the node of interest in the coefficient information storage unit 130.

[0093] The upper node propagation score field contains the upper node propagation score for the node in question. The lower node propagation score field contains the lower node propagation score for the node in question. The fault score field contains the fault score for the node in question.

[0094] For example, score table 141 has a record of node name "appl1", node ID "n1", weight 3 of the number of weight-specific alerts of "0", weight 2 of "1", weight 1 of "1", base score of "3", own score of "3", upper node ID of "-", lower node ID of "n2", resource propagation score coefficient of "0.5", upper node propagation score of "-", lower node propagation score of "0", and fault score of "3". This record indicates each score calculated for application 241. Because there is no node above application 241, the upper node ID and upper node propagation score are not set.

[0095] For example, the score table 141 has a record with the following: node name "VM1," node ID "n2," weights of "0" for the number of weighted alerts, base score "0," own score "0," upper node ID "n1," lower node ID "n3," resource propagation score coefficient "0.2," upper node propagation score "3," lower node propagation score "0.2," and fault score "3.2." This record indicates each score calculated for the virtual machine 231. Because the virtual machine 231 has not output an alert, the number of weighted alerts, base score, and own score are all "0." Meanwhile, the fault score of the virtual machine 231 is determined to be "3.2" based on the propagation scores of the application 241 and the physical machine 221, which are adjacent nodes of the virtual machine 231.

[0096] For example, the score table 141 has a record of the node name "PM1," the node ID "n3," the weights of the number of weighted alerts of 3 and 2 being "0," the weight of 1 being "1," the base score "1," the own score "1," the upper node ID "n2," the lower node ID "n4," the resource propagation score coefficient "-," the upper node propagation score "0," the lower node propagation score "0.7," and the fault score "1.7." This record indicates each score calculated for the physical machine 221. Because the physical machine 221 does not receive the allocation of resources of the lower node, no resource propagation score coefficient is set. If no resource propagation score coefficient is set, the propagation score coefficient for the propagation direction "node1<-node2" in the propagation score coefficient table 132 is used as the propagation score coefficient for the lower node.

[0097] The score table 141 also stores records indicating the base scores of nodes other than the application 241, the virtual machine 231, and the physical machine 221, the propagation scores of adjacent nodes for the node, and the fault scores of the node. The score table 141 is an example of score information stored in the score information storage unit 140.

[0098] FIG. 12 is a diagram showing an example of a failure location estimation result screen. The failure location estimation result screen 51a is displayed on the display 51. The failure location output unit 157 causes the display 51 to display the failure location estimation result screen 51a based on the node table 111, the edge table 112, the alert table 121 and the score table 141.

[0099] The failure location estimation result screen 51a includes an image showing the relationship between each node included in the monitored system 200. The failure location estimation result screen 51a includes a failure score calculated for each node. The failure location estimation result screen 51a also includes a mark indicating the node from which the alert originated, which was taken into consideration when calculating the failure score. In the example of the failure location estimation result screen 51a, the mark indicating the node from which the alert originated is an exclamation mark "!".

[0100] Furthermore, the failure location estimation result screen 51a highlights the node identified as the cause of the alert, i.e., the node identified as the failure location. In the example of the failure location estimation result screen 51a, the icon with the node name "VM1", i.e., the icon indicating the virtual machine 231, is highlighted as the node identified as the cause of the alert. As exemplified in the failure location estimation result screen 51a, the failure location estimation server 100 can also identify the node that caused the alert from among nodes that have not issued an alert.

[0101] Next, the processing procedure of the failure point estimation server 100 will be described. FIG. 13 is a flowchart showing an example of processing by the failure point estimation server. For example, the failure location estimation server 100 starts the following processing procedure periodically, when an alert is detected, or in response to a user request. Of the coefficient information, coefficients that are constant values, such as the basic score coefficient and the propagation score coefficient, have values ​​preset by the user and stored in the coefficient information storage unit 130. On the other hand, coefficients that can dynamically change depending on the system configuration, such as the resource propagation score coefficient, are calculated based on the latest configuration information in the following procedure.

[0102] (S10) The configuration information management unit 151 acquires the configuration information of the monitored system 200 from the configuration management server 300 and stores it in the configuration information storage unit 110. The configuration information includes a node table 111 and an edge table 112. The configuration information also includes information indicating the amount of resources allocated to a given node by other nodes.

[0103] (S11) The basic score calculation unit 153 identifies the node that issued the alert based on the alert table 121, calculates a basic score for each node, and registers the calculated score in the score table 141. For example, the basic score calculation unit 153 multiplies the number of alerts in each node by a weight according to the importance of the alert, and calculates the total value as the basic score for the node. The basic score calculation unit 153 also calculates the self-score of each node (=basic score coefficient*basic score) based on the basic score of each node and the basic score coefficient table 131, and registers the self-score in the score table 141. At this time, the basic score calculation unit 153 registers the node name, node ID, upper node ID, and lower node ID of each node in the score table 141 based on the node table 111 and the edge table 112.

[0104] (S12) The propagation score coefficient calculation unit 154 calculates a resource propagation score coefficient based on information included in the configuration information indicating the amount of resources allocated to the node by other nodes, and registers the calculated resource propagation score coefficient in the score table 141. Nodes that do not receive resource allocation from other nodes are excluded from the calculation of the resource propagation score coefficient. The propagation score coefficient calculation unit 154 may also store the calculated resource propagation score coefficient in the coefficient information storage unit 130.

[0105] (S13) The propagation score calculation unit 155 calculates the propagation score of the upper node for each node by multiplying the basic score of the upper node by the propagation score coefficient of the upper node, and registers the result in the score table 141. Also, the propagation score calculation unit 155 calculates the propagation score of the lower node for each node by multiplying the basic score of the lower node by the propagation score coefficient of the lower node, and registers the result in the score table 141. Here, the propagation score calculation unit 155 calculates the propagation score for a node that uses the resource of the lower node by using the resource propagation score coefficient. On the other hand, for a node that does not have a resource propagation score coefficient, the propagation score calculation unit 155 calculates the propagation score by using the propagation score coefficient registered in the propagation score coefficient table 132.

[0106] (S14) The fault score calculation unit 156 calculates the fault score of each node as fault score = (own score + propagation score of upper node + propagation score of lower node). The fault score calculation unit 156 registers the calculated fault score in the score table 141. Note that the own score can be said to be a basic score after correcting the original basic score using a basic score coefficient. Therefore, the above fault score formula may be expressed as fault score = (corrected basic score + propagation score of upper node + propagation score of lower node). Furthermore, when the basic score coefficient = 1.0, the own score = the original basic score.

[0107] (S15) The failure location output unit 157 visualizes the configuration of the monitored system 200 and the failure score of each node. Specifically, the failure location output unit 157 displays the failure location estimation result screen 51a on the display 51 based on the node table 111, edge table 112, and score table 141. Then, the processing of the failure location estimation server 100 ends.

[0108] When the above processing procedure is started upon detection of an alert, the failure point estimation server 100 updates the failure score of each node for a group of alerts that occurred at intervals shorter than the fixed time interval each time an alert is detected at intervals shorter than the fixed time interval. This is because a group of alerts that occur consecutively at intervals shorter than the fixed time interval are likely to have a common cause. The fixed time interval is set in advance to a value that corresponds to the monitored system 200, such as 1 minute, 3 minutes, 5 minutes, or 10 minutes.

[0109] Furthermore, when the above processing procedure is started at regular intervals, the failure location estimation server 100 calculates the failure score of each node based on the alert information accumulated in the alert information storage unit 120 from the previous timing to the current timing.

[0110] Furthermore, if it is clear that no changes have been made to the configuration information between the time when the configuration information was last acquired and the present time, the failure point estimating server 100 may skip step S10 and execute step S11.

[0111] Next, a modified example of the coefficient information stored in the coefficient information storage unit 130 will be described. FIG. 14 is a diagram showing a modified example of the basic score coefficient table. The basic score coefficient table 131a is stored in the coefficient information storage unit 130 in place of the basic score coefficient table 131. The basic score coefficient table 131a includes items for node attribute and basic score coefficient. The node attribute item registers the attribute of the node. The node attributes include a virtual machine (VM), a physical machine (PM), an application, a switch, and a router. The basic score coefficient item registers the basic score coefficient.

[0112] For example, the basic score coefficient table 131a has a record of a node attribute "VM" and a basic score coefficient of "0.9." This record indicates that the basic score coefficient of a node with the node attribute "VM" is "0.9." The basic score coefficient table 131a also includes records indicating basic score coefficients for other node attributes, such as the node attribute "PM."

[0113] The basic score calculation unit 153 may calculate, for each node, a self-score (= corrected basic score) to be used in calculating the fault score of the node, based on the basic score coefficient table 131a, using a basic score coefficient according to the node attributes of the node.

[0114] FIG. 15 is a diagram showing a first modified example of the propagation score coefficient table. The propagation score coefficient table 132a is stored in the coefficient information storage unit 130 in place of the propagation score coefficient table 132. The propagation score coefficient table 132a includes items for dependency relationship, propagation direction, and propagation score coefficient. The propagation score coefficient table 132a differs from the propagation score coefficient table 132 in that the information set in the items for dependency relationship and propagation direction includes node attributes. For example, the propagation score coefficient table 132a has a record indicating that, for a dependency relationship in which a virtual machine (VM) depends on a physical machine (PM), the propagation score coefficient of the VM with respect to the PM is 1.0 and the propagation score coefficient of the PM with respect to the VM is 0.5.

[0115] The propagation score coefficient table 132a also has records of propagation score coefficients for other dependency relationships, such as a propagation score coefficient for a dependency relationship in which a physical machine (PM) depends on a switch (SW).

[0116] The propagation score calculation unit 155 may calculate a propagation score for a pair of a node and an adjacent node based on the propagation score coefficient table 132a, using a propagation score coefficient according to the node attribute of the node and the node attribute of the adjacent node.

[0117] FIG. 16 is a diagram showing a second modified example of the propagation score coefficient table. The propagation score coefficient table 132b is stored in place of the propagation score coefficient tables 132 and 132a in the coefficient information storage unit 130. The propagation score coefficient table 132b includes items for dependency, propagation direction, and propagation score coefficient.

[0118] The propagation score coefficient table 132b differs from the propagation score coefficient table 132 in that the information set in the dependency relationship and propagation direction fields includes node attributes. The propagation score coefficient table 132b also differs from the propagation score coefficient table 132a in that the propagation score coefficient of a depended-on node for a dependent node is calculated as the resource propagation score coefficient. The resource propagation score coefficient is a smaller value than the propagation score coefficient of a dependent node for a dependent node.

[0119] For example, the propagation score coefficient table 132b has a record indicating that the propagation score coefficient of a VM to a PM is 1.0 and that the propagation score coefficient of a PM to a VM is a resource propagation score coefficient. The propagation score coefficient table 132b also has records of propagation score coefficients for other dependencies, such as a propagation score coefficient for a dependency relationship in which a physical machine (PM) depends on a switch (SW).

[0120] The propagation score calculation unit 155 may calculate the propagation score for each node based on the propagation score coefficient table 132b, using a propagation score coefficient according to the node attribute of the node and the node attribute of the adjacent node. In this case, the propagation score coefficient calculation unit 154 calculates a resource propagation score coefficient for each node according to the node attribute of the node and the node attribute of the adjacent node.

[0121] The resource propagation score coefficient is calculated individually for each pair of a node and an adjacent node. For example, if the virtual machine 231 is constructed on the physical machine 221, the resource allocation coefficient is set to a constant value such as 1.0. If the number of CPU cores used by the virtual machine 231 is 2 and the number of CPU cores owned by the physical machine 221 is 20, the resource propagation score coefficient of the physical machine 221 for the virtual machine 231 is calculated as follows: Resource propagation score coefficient = Resource allocation coefficient (1.0) * {Number of CPU cores used by the node (2)} / {Number of CPU cores owned by the lower node (20)} = 0.1. The propagation score coefficient calculation unit 154 calculates the resource propagation score coefficient so that the resource propagation score coefficient is smaller than the propagation score coefficient from the upper node to the lower node.

[0122] In the above example, the threshold value a for determining adjacent nodes is set to 1, but as mentioned above, a may be equal to or greater than 2. Therefore, next, an example of a method for calculating the fault score when a=2 will be described.

[0123] FIG. 17 is a diagram showing an example of another method for calculating the disability score. When a=2, a node that can be reached from a given node within two edges, i.e., within two hops, becomes an adjacent node. In this case, an adjacent node that can be reached via one edge is called a first adjacent node. Also, an adjacent node that can be reached via two edges is called a second adjacent node. For example, with respect to a physical machine 221 in the monitored system 200, the virtual machine 231 and the switch 211 are both first adjacent nodes. Also, with respect to the physical machine 221, the application 241 and the router 201 are both second adjacent nodes. The application 241 depends on the physical machine 221 via the virtual machine 231. Also, the router 201 depends on the physical machine 221 via the switch 211.

[0124] In this case, where v is the base score of the application 241, w is the base score of the virtual machine 231, x is the base score of the physical machine 221, y is the base score of the switch 211, and z is the base score of the router 201, the fault score of the physical machine 221 is calculated as follows: fault score of the physical machine 221 = εx + α1w + α2v + β1y + β2z.

[0125] Here, ε is the basic score coefficient of the physical machine 221. α1 is the propagation score coefficient of the virtual machine 231 for the physical machine 221. α2 is the propagation score coefficient of the application 241 for the physical machine 221. β1 is the propagation score coefficient of the switch 211 for the physical machine 221. β2 is the propagation score coefficient of the router 201 for the physical machine 221. α1, α2, β1, and β2 are all positive real numbers. Also, α1>α2. α1>β1. α2>β2. β1>β2. This is because it is estimated that a pair of nodes with a small number of edges separating them and a strong correlation will have a greater degree of mutual influence than a pair of nodes with a large number of edges separating them and a weak correlation.

[0126] In this way, the fault score calculation unit 156 may calculate the fault score of each node by setting a = 2. Similarly, the fault score calculation unit 156 may calculate the fault score of each node by setting a = 3 or more.

[0127] Next, an example of how the failure location estimation server 100 identifies a failure location in another monitored system will be described. FIG. 18 is a diagram illustrating another example of a monitored system.

[0128] The failure point estimating server 100 may monitor a monitored system 500. The monitored system 500 includes physical machines 521, 522, and 523, virtual machines 531, 532, 533, 534, and 535, and containers 541, 542, 543, 544, 545, 546, 547, and 548. The physical machines 521, 522, and 523 are connected to switches. The switches are omitted from FIG. 18 .

[0129] Virtual machines 531, 532, and 533 run on a physical machine 521. Virtual machine 534 runs on a physical machine 522. Virtual machine 535 runs on a physical machine 523. Container 541 runs on a virtual machine 531. Container 542 runs on a virtual machine 532. Containers 543, 544, and 545 run on a virtual machine 533. Container 546 runs on a virtual machine 534. Containers 547 and 548 run on a virtual machine 535.

[0130] For example, the failure location estimating server 100 acquires an alert L1 that has occurred in the physical machine 523 and an alert L2 that has occurred in the virtual machine 535. The failure location estimating server 100 has not acquired alerts from other nodes in the monitored system 500. In this case, the failure location estimating server 100 calculates the failure score as follows:

[0131] FIG. 19 is a diagram illustrating an example of calculation of the disability score. The basic score coefficients of both the physical machine 523 and the virtual machine 535 are 1. Furthermore, one alert has occurred in each of the physical machine 523 and the virtual machine 535, so the basic scores of both the physical machine 523 and the virtual machine 535 are also 1. Furthermore, the propagation score coefficient of the virtual machine 535 with respect to the physical machine 523 is 1. Furthermore, the propagation score coefficient of the physical machine 523 with respect to the virtual machine 535 is 0.5.

[0132] Therefore, the failure location estimating server 100 calculates the failure score of the physical machine 523 as 1*1+1*1=2.0. The failure location estimating server 100 also calculates the failure score of the virtual machine 535 as 1*1+0.5*1=1.5. Note that the failure scores of nodes other than the physical machine 523 and the virtual machine 535 will obviously be smaller than the failure scores of the physical machine 523 and the virtual machine 535, and therefore a description thereof will be omitted. Therefore, the failure location estimating server 100 identifies the node that was the root cause of the acquired alerts L1 and L2 as the physical machine 523. The failure location estimating server 100 then outputs information indicating that the node that caused the alerts L1 and L2 is the physical machine 523, and presents this information to the user.

[0133] Next, a comparative example will be described in which the propagation score coefficient is kept constant regardless of the node dependency relationship and the fault score is calculated. FIG. 20 is a diagram showing an example of an example of calculating a disability score.

[0134] For example, the propagation score coefficient of the virtual machine 535 with respect to the physical machine 523 is 0.5. Also, the propagation score coefficient of the physical machine 523 with respect to the virtual machine 535 is 0.5. In this case, the failure location estimating server 100 calculates the failure score of the physical machine 523 as 1*1+0.5*1=1.5. Also, the failure location estimating server 100 calculates the failure score of the virtual machine 535 as 1*1+0.5*1=1.5.

[0135] In this way, if the propagation score coefficient is set constant without considering the dependency between nodes, the fault score of the physical machine 523 and the fault score of the virtual machine 535 will both be the same value, and it will not be possible to identify which node is the cause of alerts L1 and L2 from the fault score.

[0136] On the other hand, the failure point estimating server 100 can appropriately identify the node that is the cause of the alert, as illustrated in FIG. 19, by using a propagation score coefficient according to the dependency relationship between nodes.

[0137] As described above, the failure point estimating server 100 executes, for example, the following process. The configuration information storage unit 110 stores configuration information indicating the relationships between the multiple nodes included in the monitored system 200. The configuration information is acquired, for example, from the configuration management server 300. The alert information storage unit 120 stores alert information indicating alerts that have occurred in the monitored system 200. The alert information is acquired, for example, from the anomaly detection server 400. The basic score calculation unit 153 calculates a basic score corresponding to the alert for each of the multiple nodes based on the alert information. The propagation score calculation unit 155 calculates a propagation score for each pair of a node and its adjacent node by multiplying the basic score of the adjacent node, identified based on the configuration information, by a weighting coefficient corresponding to the dependency relationship between the node and its adjacent node. The fault score calculation unit 156 calculates a fault score for each of the multiple nodes by adding the basic score of the node and the propagation score corresponding to the pair of the node and its adjacent node. The fault location output unit 157 identifies a first node among the multiple nodes that is the cause of the alert based on the fault scores of each of the multiple nodes.

[0138] This allows the failure location estimation server 100 to appropriately identify the node that caused the alert. For example, the failure location estimation server 100 can identify the node that caused the underlying cause of the alert not only from the node that output the alert, but also from nodes that did not output the alert. The propagation score coefficient is an example of a weighting coefficient. Furthermore, as described above, the multiple nodes may include communication devices such as switches and routers, physical machines, virtual machines, applications, and containers.

[0139] The alert information may also indicate multiple alerts that occurred in the monitored system 200 during a predetermined period. In this case, the basic score calculation unit 153 calculates a basic score for each of the multiple nodes corresponding to the multiple alerts based on the alert information. The failure location output unit 157 identifies a first node that is the cause of the multiple alerts among the multiple nodes based on the failure scores of each of the multiple nodes. If the failure scores are the same, there may be multiple first nodes. Furthermore, in addition to the first node with the highest failure score, the node with the next highest failure score may be identified as the second node, thereby identifying a node that may be the cause of the second alert. In this way, the possibility of the cause of the alert may be identified as a ranking. For example, the failure location output unit 157 may identify two or more nodes, including the first node, in order of likelihood that they are the cause of the alert based on the failure scores of each of the multiple nodes, and output a ranking indicating the likelihood that each of the two or more nodes is the cause of the alert. In this way, outputting the ranking of the possibility of the cause of the alert can also indicate to the user the order in which to respond to the failure.

[0140] Furthermore, as described above, the configuration information may be information indicating a graph in which two related nodes are connected by an edge. The propagation score calculation unit 155 may refer to the configuration information and identify an adjacent node adjacent to the node of interest based on the number of edges separating the two nodes. Furthermore, the edges in the graph may be directed edges that indicate dependency relationships between nodes. Based on the directed edges between a node and an adjacent node, the propagation score calculation unit 155 may identify a dependency relationship in which the node depends on the adjacent node, or a dependency relationship in which the adjacent node depends on the node.

[0141] When calculating the propagation score, if an adjacent node depends on the node of interest, the propagation score calculation unit 155 uses a first weighting factor as a weighting factor. On the other hand, if the node of interest depends on an adjacent node, the propagation score calculation unit 155 uses a second weighting factor that is smaller than the first weighting factor as a weighting factor.

[0142] This allows the failure point estimating server 100 to appropriately reflect the influence of adjacent nodes on a node of interest in the failure score of that node, and to appropriately identify the node that is the cause of the alert.

[0143] For example, when a node of interest depends on an adjacent node, the propagation score coefficient calculation unit 154 may increase the second weighting coefficient as the proportion of the amount of resources allocated to the node in the amount of resources owned by the adjacent node increases.

[0144] This allows the failure location estimation server 100 to appropriately reflect the influence of the depended-on node in the failure score of the depending node, and appropriately identify the node that caused the alert. The resource propagation score coefficient is an example of a second weighting coefficient that is determined according to the amount of resource.

[0145] Furthermore, the propagation score calculation unit 155 may change the weighting coefficient used to calculate the propagation score depending on the attributes of the node and the attributes of the adjacent nodes. This enables the failure location estimating server 100 to appropriately reflect the influence of adjacent nodes on a node of interest in the failure score of that node, and appropriately identify the node that is the cause of the alert.

[0146] Furthermore, the basic score calculation unit 153 may calculate the basic score of a node based on the number of alerts output by that node. This enables the failure location estimating server 100 to properly calculate the basic score that forms the basis of the failure score of each node, and to properly identify the node that caused the alert.

[0147] For example, the basic score calculation unit 153 may calculate the basic score of the node by multiplying the number of alerts of each importance level output by the node by a weight according to the importance level and adding up the results. This enables the failure location estimation server 100 to properly calculate the basic score that forms the basis of the failure score of each node, and to properly identify the node that caused the alert.

[0148] Furthermore, the propagation score calculation unit 155 may identify a first adjacent node and a second adjacent node adjacent to the node via the first adjacent node as adjacent nodes for the node. The propagation score calculation unit 155 may use a weighting coefficient for the first adjacent node for the node that is greater than the weighting coefficient for the second adjacent node for the node.

[0149] This allows the failure point estimating server 100 to appropriately reflect the influence of adjacent nodes on a node of interest in the failure score of that node, and to appropriately identify the node that is the cause of the alert.

[0150] Furthermore, when calculating the fault score of a node of interest, the fault score calculation unit 156 may correct the base score of the node by using a first base score coefficient that is constant for multiple nodes or a second base score coefficient that corresponds to the attribute of the node. In this case, the fault score calculation unit 156 calculates the fault score of the node by summing the corrected base score and the propagation score corresponding to the pair of the node and its adjacent node.

[0151] This allows the failure location estimation server 100 to appropriately evaluate the contribution of the node to the alert according to the attributes of the node in question, and to appropriately identify the node that caused the alert. Note that the basic score coefficients registered in the basic score coefficient table 131 are an example of first basic score coefficients. The basic score coefficients registered in the basic score coefficient table 131a are an example of second basic score coefficients.

[0152] Furthermore, the failure location output unit 157 may cause the display device to display a screen that highlights the first node among the multiple nodes. This allows the failure location estimating server 100 to assist the user in identifying the node that caused the alert. As a result, the failure location estimating server 100 can assist the user in quickly dealing with the failure. The display 51 is an example of a display device. However, the display device may be connected to another information processing device that communicates with the failure location estimating server 100 via the network 50. For example, the failure location estimating server 100 may receive a request for failure location estimation from another information processing device and, in response to the request, return to the other information processing device a screen that highlights the first node, i.e., information on a screen that shows the results of the failure location estimation. The failure location estimation result screen 51a is an example of a screen that highlights the first node.

[0153] The information processing of the first embodiment can be realized by causing the processing unit 12 to execute a program. The information processing of the second embodiment can be realized by causing the CPU 101 to execute a program. The program can be recorded on a computer-readable recording medium 53.

[0154] For example, the program can be distributed by distributing recording medium 53 on which the program is recorded. Alternatively, the program may be stored in another computer and distributed via a network. For example, a computer may store (install) a program recorded on recording medium 53 or a program received from another computer in a storage device such as RAM 102 or HDD 103, and then read and execute the program from the storage device. [Explanation of symbols]

[0155] 10. Information processing equipment 11 Storage section Tables 11a, 11b, and 11c 12 Processing section 20 Information Processing Systems 21,22 nodes

Claims

1. a storage unit that stores configuration information indicating relationships between nodes in a plurality of nodes included in an information processing system and alert information indicating an alert that has occurred in the information processing system; calculating a basic score corresponding to the alert for each of the plurality of nodes based on the alert information; calculating a propagation score for each pair of the node and the adjacent node by multiplying the basic score of the adjacent node adjacent to the node, which is specified for each of the plurality of nodes based on the configuration information, by a weighting coefficient according to the dependency relationship between the node and the adjacent node; calculating a fault score for each of the plurality of nodes by adding the base score of the node and the propagation score corresponding to the pair of the node and the adjacent node; a processing unit that identifies a first node that is a cause of the alert among the plurality of nodes based on the failure score of each of the plurality of nodes; An information processing device having the above.

2. When calculating the propagation score, the processing unit If the neighboring node depends on the node, use a first weighting factor as the weighting factor; If the node depends on the adjacent node, a second weighting factor smaller than the first weighting factor is used as the weighting factor.

2. The information processing device according to claim 1.

3. When the node depends on the adjacent node, the processing unit increases the second weighting coefficient as the ratio of the amount of resources allocated to the node to the amount of resources owned by the adjacent node increases.

3. The information processing device according to claim 2.

4. the processing unit changes the weighting coefficient used to calculate the propagation score according to an attribute of the node and an attribute of the adjacent node.

2. The information processing device according to claim 1.

5. the processing unit calculates the basic score of the node based on the number of the alerts output by the node.

2. The information processing device according to claim 1.

6. the processing unit calculates, as the basic score of the node, a value obtained by multiplying the number of the alerts output by the node for each level of importance of the alerts by a weight corresponding to the level of importance and adding up the products; 6. The information processing device according to claim 5.

7. the processing unit identifies a first adjacent node and a second adjacent node adjacent to the node via the first adjacent node as the adjacent nodes for the node, and uses a value as the weighting coefficient of the first adjacent node for the node that is greater than the weighting coefficient of the second adjacent node for the node; 2. The information processing device according to claim 1.

8. When calculating the fault score, the processing unit corrects the base score of the node using a first base score coefficient that is constant for the multiple nodes or a second base score coefficient that corresponds to an attribute of the node, and calculates the fault score of the node by summing the corrected base score and the propagation score.

2. The information processing device according to claim 1.

9. the processing unit identifies two or more nodes including the first node in order of likelihood that they are the cause of the alert based on the failure scores of each of the plurality of nodes, and outputs a ranking indicating the likelihood that each of the two or more nodes is the cause of the alert.

2. The information processing device according to claim 1.

10. The computer acquiring configuration information indicating relationships between nodes in a plurality of nodes included in an information processing system and alert information indicating an alert that has occurred in the information processing system; calculating a basic score corresponding to the alert for each of the plurality of nodes based on the alert information; calculating a propagation score for each pair of the node and the adjacent node by multiplying the basic score of the adjacent node adjacent to the node, which is specified for each of the plurality of nodes based on the configuration information, by a weighting coefficient according to the dependency relationship between the node and the adjacent node; calculating a fault score for each of the plurality of nodes by adding the base score of the node and the propagation score corresponding to the pair of the node and the adjacent node; identifying a first node among the plurality of nodes that is a cause of the alert based on the failure score of each of the plurality of nodes; Information processing methods.

11. acquiring configuration information indicating relationships between nodes in a plurality of nodes included in an information processing system and alert information indicating an alert that has occurred in the information processing system; calculating a basic score corresponding to the alert for each of the plurality of nodes based on the alert information; calculating a propagation score for each pair of the node and the adjacent node by multiplying the basic score of the adjacent node adjacent to the node, which is specified for each of the plurality of nodes based on the configuration information, by a weighting coefficient according to the dependency relationship between the node and the adjacent node; calculating a fault score for each of the plurality of nodes by adding the base score of the node and the propagation score corresponding to the pair of the node and the adjacent node; identifying a first node among the plurality of nodes that is a cause of the alert based on the failure score of each of the plurality of nodes; A program that causes a computer to perform a process.

Images