Automatic root cause analysis based on matching sets

Abstract

A set of observed symptoms of a physical system is received. The set of observed symptoms is matched to a fault scenario database to generate a plurality of matching fault scenarios, wherein the fault scenario database includes combinations of symptoms and corresponding root cause identifiers. The plurality of matching fault scenarios are refined. A determination is made as to whether a first fault scenario in the plurality of matching fault scenarios is to be merged with a second fault scenario. In response to a determination that a first fault scenario in the plurality of matching fault scenarios is to be merged with a second fault scenario, the second fault scenario is included in the plurality of matching fault scenarios, the first fault scenario is deprioritized, or both. A plurality of matching fault scenarios are output as potential root causes.

Description

[0001] Cross-references to other applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 645,656, filed on March 20, 2018, and entitled “MATCHSET-BASED AUTOMATIC ROOT CAUSE ANALYSIS,” which is incorporated herein by reference for all purposes. Background Art

[0003] Determining the root cause of problems in complex systems is challenging for several reasons.

[0004] First, a fault may propagate to cause a high density of symptoms, each of which manifests itself as a failure or misbehavior of some component. For example, a low-level computer network link failure may cause signal loss at each end of the link, interruption of data flow at the transport layer, and then cause timeouts and errors at the application level.

[0005] Second, there is uncertainty associated with the available information. For example, a temperature sensor may indicate a warm reading for a refrigerated area that is assumed to be cooled, indicating that the refrigeration system has failed. However, it is also possible that the sensor has failed and is providing incorrect readings. In addition, the sensor or reporting device may fail or be unreachable, so the associated telemetry is not available.

[0006] Finally, the available information may not be sufficient to disambiguate between two or more potential root cause faults, even if only one fault actually exists. That is, it is not only necessary to filter out extraneous faults that are caused by the root cause, but also to do so when there is uncertainty about how correct, current, or complete the telemetry is on the system being diagnosed.

[0007] Given these challenges, it may be impossible to determine with absolute certainty what the actual root cause is. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0009] Figure 1 is a functional diagram illustrating a programmed computer / server system for match set based automated root cause analysis according to some embodiments.

[0010] Figure 2 is a block diagram illustrating an embodiment of an example network.

[0011] Figure 3is a block diagram illustrating an embodiment of a power example.

[0012] Figure 4 is a flow chart illustrating an embodiment of a process for automatic root cause analysis based on matching sets.

[0013] Figure 5A is a flow chart illustrating an embodiment of a process for determining whether a first failure scenario among a plurality of matching failure scenarios is to be merged with a second failure scenario.

[0014] Figure 5B is a flow chart illustrating another embodiment of a process for determining whether a first failure scenario among a plurality of matching failure scenarios is to be merged with a second failure scenario.

[0015] Fig. 6A , Figure 6B and Figure 6C is a flow chart illustrating an embodiment of a process for refining multiple matching failure scenarios based on relationships. DETAILED DESCRIPTION

[0016] The present invention can be implemented in numerous ways, including as a process; an apparatus; a system; a material composition; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on a memory coupled to the processor and / or provided by the memory. In this specification, these implementations or any other form that the present invention can take can be referred to as techniques. In general, within the scope of the present invention, the order of steps of the disclosed processes can be changed. Unless otherwise stated, components such as processors or memories described as being configured to perform tasks can be implemented as general components that are temporarily configured to perform tasks at a given time, or as specific components that are manufactured to perform the tasks. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0017] A detailed description of one or more embodiments of the present invention is provided below, as well as drawings illustrating the principles of the present invention. The present invention is described in conjunction with such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical materials known in the technical field related to the present invention are not described in detail so that the present invention will not be unnecessarily obscure.

[0018] Identifying the most likely root cause or causes of a problem in a complex system is disclosed. As described above, it may not be possible to determine with absolute certainty what the actual root cause may be, and therefore one useful solution is to identify the most likely root cause.

[0019] Disclosed is a refinement of an automatic root cause analysis (ARCA) that provides a multiple matching capability. Multiple matches for a given set of symptoms are refined based on a relationship between at least one of: the symptoms that matched, the identity of the match, and the confidence level of the match by exploiting the fact that the multiple matching capability is redundant to some extent. These relationships may indicate a refinement via at least one of: merging base scenarios by derived scenarios associated with the base scenarios; determining two matches as alternative explanations and / or indicating the matches as part of the same root cause group; and determining two matches as separate possible faults and / or indicating the matches as part of different root cause groups. Thus, disclosed is a technique for allowing partial matching. Disclosed is processing of additional matches to reduce and organize the matches so that they are easier to interpret efficiently.

[0020] The disclosed technology generalizes to the broader technology of rule-based systems. Technically, a "rule" in computing is expressed as: condition → action. So, ARCA is just a condition as a set of symptoms, and the action is: "report the associated root cause". More specifically, a condition is a combination of "sub-conditions", that is, "have symptom A but not symptom B", etc.

[0021] For example, adding indexes is essential to improving the performance of database queries. If a query condition specifies sub-conditions that are handled by an index, the query processor can access the index and iterate only on those records indicated by the index as relevant, matching the remaining query conditions, rather than performing a full table scan. The latter incurs a cost of O(N), where N is the number of rows in the table. If the index contains K entries, where K is much smaller than N, a scan of the K entries is much faster than scanning all N entries because there are fewer records to check and because the sub-conditions handled by the index reduce the query conditions. In addition, if each record is B bytes in length on average, then if the data is in memory, a full scan typically incurs N*B bytes of I / O or memory access, and the index significantly reduces this I / O or memory requirement.

[0022] The index mechanism can then be constructed as an index table that is optimized for efficient full scans while performing multiple index matches as part of the scan. In particular, each column represents an index. The row contains the entry value of each index for the corresponding row in the data table. For example, the value of the subcondition X in row M is the value of the index corresponding to X in row M of the data table. This value is allowed to be empty (NULL), indicating "don't care" or "don't know". The matching mechanism takes as input an index table and an index row mask, where the mask specifies a value for each index, and returns a list of rows that match the mask. The mask can also specify a null value for the index, indicating "don't know". The collection of indexes is rotated into a table, rather than each being treated as a separate (associated) array of values.

[0023] The term subcondition is an explanation of the index value because it is more general than just indexing a specific value in a column. For example, a true value in an index may simply indicate that the corresponding entry in the corresponding row of a table exceeds a specified threshold. Moreover, a subcondition such as exceeding a given threshold may be sufficient for an application and change much less frequently than the actual value. Using this terminology, a row in an index table contains a subcondition value for each index subcondition of the table row that corresponds to that index table row (although the subcondition value may be empty).

[0024] In one embodiment, if there are N rows in the original table, then the index table has N rows. Otherwise, if there is no corresponding row in the index table for a row in the original table, the row can only be found in the query by performing a full table scan. In essence, a method is disclosed for scanning an N-entry index table rather than using multiple indexes to avoid a full table scan. The index table is optimized to make scanning more efficient than scanning the original base data table. In other words, the focus is on optimizing the index table scan rather than avoiding N-row data table scans by using conventional indexing schemes. The key focus is on applications where the sparsity of the data table makes the use of the index table more efficient than using conventional indexes.

[0025] Thus, with respect to ARCA, the index table corresponds to a set of potential failure scenarios, one potential failure scenario per row. Symptoms correspond to sub-condition entries in the table. For example, for a data table consisting of a row for each oven in a factory and an entry corresponding to the oven temperature, an index corresponding to "temperature exceeds threshold" will set the entry in the column in the index table corresponding to that sub-condition for each row in which the corresponding oven exceeds the temperature threshold to true. Thus, the entries in the set of potential failure scenarios for a given symptom correspond to the indexes in the index table, and therefore to the columns in the index table. The output causes in ARCA correspond to the matching rows in the index table embodiment. The actual failure scenarios correspond to the masks in the index table embodiment. And, null is a special value corresponding to "don't care" or "don't know" because null is a term conventionally used in database literature. With this translation of the terminology, the matching method is the same as in ARCA, except that in query usage, the matching logic between masks and rows is reversed.

[0026] In one embodiment, one or more indexes in the table are bitmap indexes that are extended with the ability to indicate a "don't care" value as well as true and false. This corresponds to a ternary or quadruple match. Therefore, the minimum representation for a ternary or quadruple representation and match is two bits. Without limitation, examples of root cause analysis are given herein, but any person of ordinary skill in the art will recognize that root cause analysis is a specific application of an index table. In particular, root cause analysis can be viewed as being implemented as a query to an index table to determine the row(s) that match the set of symptoms specified in the index row mask, and then mapping each row to a corresponding root cause.

[0027] Figure 1 is a functional diagram illustrating a programmed computer / server system for automatic root cause analysis based on matching sets according to some embodiments. As shown, Figure 1 A functional diagram of a general purpose computer system programmed to provide matching set based automatic root cause analysis according to some embodiments is provided. As will be apparent, other computer system architectures and configurations may be used for matching set based automatic root cause analysis.

[0028] The computer system 100, including various subsystems described below, includes at least one microprocessor subsystem, also referred to as a processor or central processing unit ("CPU") (102). For example, the processor (102) may be implemented by a single-chip processor or by multiple cores and / or processors. In some embodiments, the processor (102) is a general-purpose digital processor that controls the operation of the computer system 100. Using instructions retrieved from a memory (110), the processor (102) controls the receipt and manipulation of input data, as well as the output of data and the display of data on an output device, such as a display and a graphics processing unit (GPU) (118).

[0029] The processor (102) is bidirectionally coupled to a memory (110), which may include a first main storage device, which is typically a random access memory ("RAM"), and a second main storage area, which is typically a read-only memory ("ROM"). As is known in the art, the main storage device can be used as a general storage area and as a temporary storage, and can also be used to store input data and processed data. In addition to other data and instructions for processes operating on the processor (102), the main storage device can also store programming instructions and data in the form of data objects and text objects. As is also known in the art, the main storage device generally includes: basic operating instructions, program code, data and objects, such as programming instructions, used by the processor (102) to perform its functions. For example, depending on whether data access needs to be bidirectional or unidirectional, the main storage device (110) may include any suitable computer-readable storage medium described below. For example, the processor (102) can also directly and very quickly retrieve frequently needed data and store the data in a cache memory not shown. The processor ( 102 ) may also include co-processors (not shown) as supplemental processing components to assist the processor and / or the memory ( 110 ).

[0030] The removable mass storage device (112) provides additional data storage capacity for the computer system 100 and is coupled to the processor (102) bidirectionally (read / write) or unidirectionally (read-only). For example, the storage device (112) may also include computer readable media such as flash memory, portable mass storage devices, holographic storage devices, magnetic devices, magneto-optical devices, optical devices, and other storage devices. The fixed mass storage device (120) may also provide additional data storage capacity, for example. An example of a mass storage device (120) is an eMMC or microSD device. In one embodiment, the mass storage device (120) is a solid-state drive connected via the bus (114). The mass storage devices (112), (120) typically store additional programming instructions, data, etc. that are not typically actively used by the processor (102). It will be appreciated that if desired, the information retained in the mass storage devices (112), (120) can be incorporated as part of the main storage device (110) (e.g., RAM) in a standard manner to serve as virtual memory.

[0031] In addition to providing access to the storage subsystem to the processor (102), the bus (114) may also be used to provide access to other subsystems and devices. As shown, these may include a display monitor (118), a communication interface (116), a touch (or physical) keyboard (104), and one or more auxiliary input / output devices (106) (including an audio interface, a sound card, a microphone, an audio port, an audio recording device, an audio card, a speaker, a touch (or pointing) device, and / or other subsystems as required). In addition to a touch screen and / or a capacitive touch interface, the auxiliary device (106) may be a mouse, a stylus, a trackball, or a tablet device, and may be used to interact with a graphical user interface.

[0032] The communication interface (116) allows the processor (102) to be coupled to another computer, computer network, or telecommunication network using a network connection as shown. For example, through the communication interface (116), the processor (102) can receive information (e.g., data objects or program instructions) from another network or output information to another network during the implementation of the method / process steps. Information, often represented as a sequence of instructions to be executed on the processor, can be received from another network and output to another network. An interface card or similar device and appropriate software implemented by the processor (102) (e.g., executed by / implemented on the processor (102)) can be used to connect the computer system 100 to an external network and transfer data according to a standard protocol. For example, various process embodiments disclosed herein can be executed on the processor (102), or can be implemented in conjunction with a remote processor that shares a portion of the processing, through a network such as the Internet, an intranet network, or a local area network. Throughout this specification, "network" refers to any interconnection between computer components, including the Internet, Bluetooth, WiFi, 3G, 4G, 4GLTE, GSM, Ethernet, TCP / IP, intranet, local area network ("LAN"), home area network ("HAN"), serial connection, parallel connection, wide area network ("WAN"), Fibre Channel, PCI / PCI-X, AGP, VLbus, PCI Express, Expresscard, InfiniBand, ACCESS.bus, wireless LAN, HomePNA, fiber optic, G.hn, infrared network, satellite network, microwave network, cellular network, virtual private network ("VPN"), universal serial bus ("USB"), FireWire, Serial ATA, 1-Wire, UNI / O, or any form of connecting homogeneous, heterogeneous systems and / or groups of systems together. Additional mass storage devices not shown may also be connected to the processor (102) via the communication interface (116).

[0033] Auxiliary I / O device interfaces, not shown, may be used in conjunction with the computer system 100. Such auxiliary I / O device interfaces may include general and custom interfaces that allow the processor (102) to send data and more generally receive data from other devices, such as microphones, touch-sensitive displays, transducer card readers, tape readers, voice or handwriting recognizers, biometric readers, cameras, portable mass storage devices, and other computers.

[0034] In addition, various embodiments disclosed herein further relate to a computer storage product with a computer-readable medium, which includes program code for implementing various computer-implemented operations. The computer-readable medium is any data storage device that can store data that can be read by a computer system thereafter. Examples of computer-readable media include, but are not limited to, all of the media mentioned above: flash media, such as NAND flash, eMMC, SD, compact flash; magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD-ROM disks; magneto-optical media, such as optical disks; and specially configured hardware devices, such as application-specific integrated circuits ("ASICs"), programmable logic devices ("PLDs"), and ROM and RAM devices. Examples of program code include both machine code, such as generated by a compiler, and files containing higher-level code (e.g., scripts) that can be executed using an interpreter.

[0035] Figure 1 The computer / server system shown in is merely an example of a computer system suitable for use with the various embodiments disclosed herein. Other computer systems suitable for such use may include additional or fewer subsystems. In addition, the bus (114) is illustrative of any interconnection scheme that serves to link the subsystems. Other computer architectures having subsystems with different configurations may also be utilized.

[0036] Input probabilities versus output probabilities. The traditional approach used with Bayesian networks and machine learning is to associate probabilities with inputs and pass those probabilities through a computational network to compute the output with the highest probability. For example, a temperature sensor can be considered to have a certain probability of P Exceeds the threshold and fills the probability 1- P Below the threshold, where P Reflects the uncertainty of the sensor not reporting the current temperature.

[0037] There are several problems when associating probabilities with inputs.

[0038] First, these input probabilities are unknown, and perhaps unknowable in practice, given that they may depend on many factors, including the age of the component, the manner in which it is installed, and the brand / model of the component. This is particularly problematic in ARCA due to the relatively rare failure events. For example, it may not be feasible to have data on how often a given brand and model of temperature sensor fails, causing it to report an incorrect temperature exceeding the set threshold for a particular system, given that this event may only be detected by having one or more redundant temperature sensors with which to compare the normal or in-use temperature sensor. That is, each sensor would require a second monitoring device and a record of how often it differs, which is an expensive redundancy that is not often performed in practice.

[0039] Second, different inputs are often not completely independent, since there may often be dependencies or correlations between different inputs, given that they are part of the same system being diagnosed. This dependency can be expressed probabilistically as a conditional probability between two inputs. However, given that such a conditional probability involves samples across a pair of elements, it is even more difficult to know. Furthermore, the actual conditional probability may vary over time and / or space based on various factors, including the actual values ​​of different sensor values, the age of the system, its operating mode, and the like.

[0040] Finally, the output of such systems is typically provided as the root cause with the highest probability, as calculated from these input probabilities, and is therefore a single root cause, given that only one can have the highest probability. In fact, these probability calculations can generate a large number of potential diagnoses and rank them by probability. However, given the difficulties in using the input probabilities mentioned earlier, it is unclear how to sensibly filter these diagnoses based on the calculated probabilities. For example, if the operator only considers potential root causes with a probability greater than 0.7, it is reasonable to ask how the user can have confidence that the actual root cause has at least that probability. That is, how can the user infer that this particular threshold is the correct threshold to include the actual root cause, without arbitrary numbers or the need for repetition to gain a manual qualitative intuition for the system.

[0041] Manual methods of root cause analysis traditionally use qualitative assessments of symptoms and human "common sense" and so are not well suited to automated root cause analysis systems. Similarly, these manual "hands-on" methods lack a computational framework for handling uncertainty, further making them difficult to automate.

[0042] An example of traditional ARCA is DellEMC's SMARTS program, which can apply probabilities to inputs. It does not appear to generate multiple root cause matches when it could use a closest match based on Hamming distance. The use of Hamming distance will generally only give one Top matches. Given the use of Hamming distance as an arbitrary metric, it is not necessarily clear what semantics or value to attach relative to the second closest match, the third closest match, and so on.

[0043] An efficient automated means is disclosed that generates a set of likely root cause faults corresponding to symptoms of a system under diagnosis. Potential failure scenarios As mentioned in this article, potential failure scenarios can correspond to the expected symptom As referred to herein, a symptom is an observable value or range of values, or a value that can be calculated from observable values ​​that is related to or indicative of a fault. Unlike SMARTS, which can apply probabilities to inputs, the disclosed technique associates probabilities with outputs, such as those expressed as Confidence Level As mentioned herein, the confidence level for a potential failure scenario is the probability of a scenario failure in the system given a set of symptoms for the potential failure scenario. In some embodiments, historical data is analyzed to construct a symptom set-confidence level mapping table. The symptom set is looked up in the mapping table to obtain the corresponding confidence level. Other techniques for determining confidence levels may be used given the symptom set.

[0044] For example, a symptom may be a "loss of signal" reported by a computer network switch on a particular one of its interfaces. Actual failure scenario ), the actual fault scenario is matched against the potential fault scenario set to produce a matching set referred to herein as the actual fault scenario, wherein if the potential fault scenario matches the actual fault scenario, then the potential fault scenario is a member of the matching set.

[0045] Then, based on the matching potential failure scenarios property and other information to refine the matching set. As mentioned herein, attributes include any information and / or relationships related to the matched potential failure scenarios, such as relationships between matched symptoms, matched identities, and / or matched confidence levels. The root cause failures associated with the entries in the refined matching set can then be output, thereby constituting the results of the root cause analysis. The root cause analysis can generate a set of potential root causes to be more likely to include an actual root cause failure or root cause failures.

[0046] Network Examples – Multiple potential failure scenarios. Figure 2 is a block diagram illustrating an embodiment of an example of a network. In this example, a symptom in the current symptoms of the network is a "loss of signal" reported by a computer network switch on a specific one of its interfaces.

[0047] The switch SW1 (202) includes: a network interface I1-a (204) and a power sensor (206) to sense power loss to the switch SW1. The network interface I1-a (204) is coupled to the link a (222), the link leads to switch SW2 (242), wherein switch SW2 also includes: network interface I2-a (244) and power sensor (246). If the network interface (204, 244) is powered separately, then Figure 2 Not shown may also be a power sensor for the network interface (204, 244).

[0048] The actual failure scenario of "loss of signal" reported by SW1 (202) on interface I1-a (204) may be matched to failure scenario FS1 corresponding to a link failure between switch SW1 (202) and switch SW2 (242). a (222) has a link failure. However, the same symptom can also match failure scenario FS2, in which both interfaces (204, 244) at either end of the link fail simultaneously. It can also match failure scenario FS3, which corresponds to link a The link failure in (222) is considered, but auxiliary symptoms such as the symptoms corresponding to power loss at SW1 sensor (206) and SW2 sensor (246) are known to be false. Therefore, in this example, the matching set consists of FS1, FS2, and FS3. A tabular representation of this is:

[0049] Tags / Metadata symptom root cause FS1 Signal loss at (204) & no power loss at (206) & no power loss at (246). Link (222) failed FS2 Signal loss at (204) & no power loss at (206) & no power loss at (246). Both interfaces (202, 244) fail at the same time. FS3 Signal loss at (204). Link (222) failed

[0050] The base scenarios are merged by the derived scenarios associated with the base scenarios. In one embodiment, the attributes of the potential fault scenarios indicate when a potential fault scenario FSa is merged by another potential fault scenario FSb. That is, whenever FSb is matched, FSa will also be matched. As mentioned in this article, FSa is Basic scene, And FSb is Derived Scene In the case where both FSa and FSb are matched, the refinement of the matching set is to remove FSa from the matching set before converting the fault scenarios to their associated root causes.

[0051] To illustrate this situation, continue Figure 2For the network example, the match refinement step will realize that FS3 is merged with FS1 because FS3 only needs to match a subset of the symptoms that FS1 is requiring.

[0052]

[0053] Another simple example of a base scenario being subsumed by a derived scenario is the medical example:

[0054] • Potential Failure Scenario FSm shows the root cause of influenza with a confidence level of 80%, taking into account the symptoms of high body temperature and pain; and

[0055] • Potential failure scenario FSn shows the root cause of influenza with a confidence level of 90%, taking into account the symptoms of high body temperature, aches and headaches.

[0056]

[0057] Therefore, in the case of an actual fault scenario including symptoms of high body temperature, pain, and headache, FSm is identified as a base scenario merged by derived scenarios FSn, and thus the root cause of influenza with a 90% confidence level is output.

[0058] .

[0059] Output a combination of probabilities. In one embodiment, refinement may realize that two potential failure scenarios present in the matching set are actually two different sets of symptoms for the same root cause, and may in fact both be true, so the output contains the potential root cause, possibly with an associated probability that is a probability combination of the two potential failure scenarios. For example, FSn may be a potential failure scenario that shows a root cause of flu with a 90% confidence level, given the symptoms of high temperature, aches, and headaches, and FSp may be a potential failure scenario that shows a root cause of flu with a 5% confidence level, given the symptoms of runny nose and ear pain.

[0060] .

[0061] Patients with symptoms of high body temperature, pain, headache, runny nose, and ear pain may be identified as having a combination of associated probabilities that is a combination of a 90% confidence level and a 5% confidence level. In one embodiment, the confidence levels may be linearly summed.

[0062] Tags / Metadata symptom Root Cause Identifier Confidence Level F High temperature & pain & headache. influenza 90％ FSp Runny nose & ear pain. influenza 5％ &&&&

[0063] In one embodiment, the attributes of a potential failure scenario indicate when one potential failure scenario FSc is a consequence of another potential failure scenario FSd. Alternative PossibilitiesTherefore, when both FSc and FSd appear in the matching set, refinement will indicate these as the actual fault scenarios. Alternative Part of a subset of potential root causes, rather than indicating the two matches as two separate possible faults, and / or indicating the two matches as parts of different root cause groups. In an embodiment, the property indicating a potential root cause as an alternative may be calculated by comparing the symptoms of two potential root causes. It is an alternative to a subset of the symptoms of other potential root causes, and it is not the same underlying root cause of which it is an alternative.

[0064] For example, using Figure 2 For the network example, refinement would indicate FS1 and FS2 as alternatives to each other, considering that both scenarios correspond to a common set or subset of symptoms.

[0065] Tags / Metadata symptom Root Cause Identifier FS1 (derived scene) Signal loss at (204) & no power loss at (206) & no power loss at (246). Link (222) failed FS2 (right) Signal loss at (204) & no power loss at (206) & no power loss at (246). Two interfaces (202, 244) failed simultaneously.

[0066] Another simple example of an alternative explanation is the medical example:

[0067] • Potential failure scenario FSn shows the root cause of influenza with a 90% confidence level, considering the symptoms of high body temperature, aches and headaches; and

[0068] • Potential Failure Scenario FSq shows the root cause of hay fever with a confidence level of 3%, which takes into account the symptoms of high body temperature, pain and headache;

[0069] Tags / Metadata symptom Root Cause Identifier Confidence Level F High temperature & pain & headache. influenza 90％ FqV High temperature & pain & headache. Hay fever 3％

[0070] Therefore, in the case of a realistic failure scenario including symptoms of high body temperature, pain, and headache, FSq was identified as an alternative explanation to FSn.

[0071]

[0072] In one embodiment, another property of a potential failure scenario is the probability of the failure scenario relative to its associated alternative failure scenario. Figure 2 For example, the probability of FS1 can be 0.95, and as a FS1 Alternative The probability of FS2 can be assigned to be 0.05. Match set refinement can then rank the associated root causes according to the probability associated with each alternative. Figure 2 In the network example, the refined set of root causes could be:

[0073] [RC1: 0.95, RC2: 0.05]

[0074] Among them, RC1 corresponds to the root cause associated with failure scenario FS1, and RC2 corresponds to the root cause associated with failure scenario FS2. This refinement eliminates the third entry because FS3 is merged by FS1.

[0075] Tags / Metadata symptom Root Cause Identifier Confidence Level FS1 (derived scene) Signal loss at (204) & no power loss at (206) & no power loss at (246). Link (222) failed 95％ FS2 (alternative interpretation of FS1) Signal loss at (204) & no power loss at (206) & no power loss at (246). Two interfaces (202, 244) failed simultaneously. 5％

[0076] Associating probabilities with potential failure scenarios may be more feasible than an input probability approach because each failure scenario represents a situation in which the top failure requires remediation. Therefore, operational data can indicate how often a given root cause occurs compared to alternatives (i.e., those with the same symptoms). For example, going back to Figure 2 Network example, if the broken link a Since (222) is the actual root cause out of 95 of the 100 times the associated symptoms were observed, and since only 5 of those 100 times were both interfaces (204, 244) actually failing simultaneously, the recorded operational data provides a basis for weighting and ranking the two alternative root causes using these probabilities.

[0077] Thus, first viewing the output as a remedial action of detecting a broken link a (222) will immediately resolve the actual root cause failure in the majority of cases, and will require switching to an alternative fault remedial action in only 5% of cases. In some cases, such as the case where two interfaces (204, 244) fail simultaneously, the user can estimate the probability-based average time to repair the interfaces, as well as the frequency of individual interface failures and the number of interfaces, further qualifying with the likelihood that the two interfaces that fail within the same recovery window are actually located at either end of the link. Note that it is possible, although unlikely, that both links fail and both interfaces fail. That is, the alternative root causes may not be mutually exclusive. In this case, remedial actions need to be taken for both failures.

[0078] Matching. In one embodiment, the matching of actual failure scenarios with potential failure scenarios as implemented by the matching mechanism is exact in the sense that each matched potential failure scenario may be required such that the actual failure scenario satisfies the symptom requirements specified in the matched potential failure scenario for each symptom.

[0079] For example, if a potential failure scenario specifies symptom Si as oven temperature greater than 100 degrees Celsius, then the actual failure scenario should include this symptom being reported as greater than 100 degrees Celsius.

[0080] This matching is in contrast to the input probability approach used, for example, in SMARTS, where there is some probability that the symptom is true even if the sensor does not report it, given the uncertainty about the sensor as captured by the associated probability. It is also in contrast to various seemingly arbitrary "distance-based" approaches (such as the Hamming distance approach), where the ARCA system selects the "best match" based on some metric distance between the actual symptom and the symptom associated with the root cause (analogous to the potential failure scenario).

[0081] In one embodiment, generation of the matching set is performed via the matching mechanism specified in U.S. patent application Ser. No. 15 / 973,847, entitled “AUTOMATIC ROOT CAUSE ANALYSIS USING TERNARY FAULT SCENARIO REPRESENTATION,” which is incorporated herein by reference for all purposes.

[0082] An unrefined set of fault scenario matches may include multiple members, even in the case of partial matches to a single actual fault, because the set of potential fault scenarios should cover cases where some telemetry is missing or erroneous. Figure 2 FS3 in the network example of , so that there is a match even if the telemetry used to assist the symptom is incomplete or incorrect. That is, a link cannot be diagnosed simply because one switch (202) or the other switch (242) is unable to report to the interface for power (206, 246). a A link failure in (222) would be unacceptable.

[0083] In general, matching can be done efficiently and can match multiple independent root causes simultaneously, as described in the above application for ternary fault scenario representation. Matching has the disadvantage that it fails to match when any specified symptom in the potential fault scenario corresponding to the actual fault scenario does not match the symptom determined from telemetry. This may occur even when a human evaluation of the symptom may quickly conclude what the root cause is.

[0084] Figure 3 is a block diagram illustrating an embodiment of a power example. In this power example, switch SW1 (302) is fully coupled to 24 other switches SW2 (342), SW3 (362) through SW25 (392) via interfaces and links. As previously described in Figure 2As shown in FIG. 1 , each switch (eg, switch SW1 ( 302 )) includes: a power sensor ( 302 z ), and one or more interfaces I1 - a ( 302 a ), I1 - b ( 302 b ), …, I1 - x ( 302 x ), each corresponding to a link a (322a), b (322b), …, x (322x).

[0085] If power fails to a computer network switch SW1 (302) including a SW1 power sensor (302z), then it would be expected that each interface to which the switch is linked would detect a loss of signal. However, if the switch in question is linked to 24 individual interfaces I2-a (342a), I3-b (362b), ... I25-x (392x), but only 23 of these interfaces report a loss of signal, and the 24th interface I25-x (392x) is missing from telemetry, then even if it is reasonable to conclude from the symptoms that the switch has failed, and further, that in the case of the switch SW1 power sensor (302z) reporting a power loss, the failure occurred due to a lack of power, the match will still fail to identify a potential failure scenario that specifies that all 24 individual interfaces have a power loss symptom.

[0086] The disclosed technology partially exploits such matching capabilities to match multiple failure scenarios simultaneously in order to compensate for this disadvantage. In particular, in addition to having potential failure scenarios corresponding to all symptoms, there are also potential failure scenarios corresponding to partial matches specified for the same root cause. The expansion of the associated attributes with potential failure scenarios allows the matching set to be refined to reduce the number of potential root causes that are actually output.

[0087] In particular, when a match to a full potential failure scenario occurs, partially matched potential failure scenarios corresponding to the same root cause are eliminated and / or merged. Similarly, the probabilistic attributes associated with potential failure scenarios allow the output to effectively indicate a lower confidence in the root cause in the output, where it exists only due to what is effectively a partial match.

[0088] In one embodiment, another technique that allows for partial matching is approximate matching as specified in U.S. Provisional Patent Application No. 62 / 799,613, entitled "APPROXIMATE MATCHING," which is incorporated herein by reference for all purposes. For example, approximate matching is used in situations where not all features (e.g., subconditions) must be known. Thus, approximate matching can be used in conjunction with partial matching.

[0089] In one embodiment, approximate matching is provided by specifying a distance threshold parameter and outputting rows as matches if they are within the distance threshold according to some distance metric defined between the row and the mask. Processing additional matches for efficiency in interpretation to reduce and organize matches can be improved by approximate matching, which is performed in part by, for example, treating an approximate match at distance D as the underlying root cause relative to a match at distance D-1.

[0090] Partial Match Potential Failure Scenario (PMPFS). PMPFS is referred to herein as a potential failure scenario that is added to efficiently handle partial matches with the matching mechanism. There are various techniques to define PMPFS.

[0091] Omit one symptom of PMPFS. First, for each full potential failure scenario for a root cause, there may be a PMPFS for each symptom that omits one of the symptoms. For example, using Figure 3 For example, a PMPFS may exist for each adjacent interface, which omits that interface as a symptom, or alternatively marks the symptom as "don't care". For example, PMPFS may omit I25-x (392x) as "don't care", so that I2-a (342a), I3-b (362b), ..., I24-w ( Figure 3 In the event that a signal loss is reported by a switch SW1 (not shown), the system can conclude that switch SW1 (302) has failed.

[0092] It may be possible to go a step further and provide PMPFS for a subset of symptoms of the full potential failure scenarios. For example, create a PMPFS for both I24-w and I25-x (392x) as "don't care". However, this may lead to an impractical number of PMPFS in a real complex system. For example, in the switch example with 32 directly adjacent switches, there are essentially 2 to the power of 32 or roughly 4 billion possible subsets. Here, approximate matching can solve the problem of excessive number of PMPFS. In other words, partial matching can be thought of as adding additional less complete rows, while approximate matching is relaxing the matching criteria so that rows that do not exactly match the mask or the actual complete set of symptoms can be matched.

[0093] Exclude PMPFS from a range of values. An efficient way to support partial matching while avoiding an exponential explosion in the number of PMPFS is to allow potential failure scenarios to specify a given symptom as excluding a certain value or range of values. The values ​​that are often used will contradict the associated failure as the root cause. Figure 3In the power example, PMPFS can be specified to require the signal loss symptom to be true or unknown. Then, as long as no neighboring switch claims to receive a signal from the switch that should be losing power, matching occurs. That is, if the symptom is unknown for some neighboring switches, such as unknown I25-x (392x), matching still occurs.

[0094] In one embodiment, the representation of PMPFS allows specification of exclusion-based matching in range specifications, rather than just inclusion-based matching. For example, in the referenced disclosure, the binary representation of ternary values ​​may use an "unknown but true" value (i.e., 01) that is otherwise not used to indicate "unknown to be true." In general, there are conventional techniques for data representation that can be used to efficiently encode additional information corresponding to exclusions and inclusions.

[0095] Limit the scope of PMPFS. Another way to effectively support partial matching while avoiding an exponential explosion in the number of PMPFS is to limit the range of PMPFS and their symptoms and correspondingly reduce the probabilities associated with them. Figure 3 In the power example of , a PMPFS can be generated that matches for the current power fault sensor (302z) of switch SW1 (302) and specifies a "don't care" with respect to the telemetry effects of neighboring switches (342a, 362b, ..., 392x). This PMPFS matches if the power sensor (302z) reports a power fault, but there is conflicting information from one or more neighboring switches, such as "unknown" for I25-x (392x) that may be incorrect or stale.

[0096] On the other hand, if the above PMPFS for the same switch SW1 (302) matches the exclusion-based match, the lower probability match is filtered out by the refinement step. In general, the generation of a PMPFS can be limited in scope based on relationships with other elements, types of other elements, specific characteristics of these elements, and other attributes.

[0097] Define cluster symptoms. Another way to efficiently support partial matching while avoiding an exponential explosion in the number of PMPFS is to define aggregation symptoms that are set based on telemetry across multiple sensor inputs. Figure 3 In the power example, we can define an aggregate symptom corresponding to more than a certain threshold K, SW25 (392) that have a loss signal from a given switch SW1 (302). The PMPFS for switch power loss can then specify this aggregate symptom so that a switch is considered to have had a power failure if most of its direct neighbors have a loss signal from it. For clarity, the benefit of incorporating this information from its direct neighbors is that it helps distinguish this situation from a situation where the current sensor on the switch has failed, rather than the power itself.

[0098] Back propagation. Another way to efficiently support partial matching is to exclude from the PMPFS symptom requirements the requirements that have been determined by what is referred to herein as symptom backpropagation. Figure 2 In the network example, in the link a One likely explanation for not receiving a signal at the far end / SW2 end (242) of (222) is a disconnected network cable. An alternative explanation for not receiving a signal at the far end of the link is that interface I1-a (204) at the near end / SW1 end (202) has lost power. This is because a power loss at an interface at one end of the link (222) effectively propagates the signal loss symptom to the interface at the other end of the link.

[0099] Using back propagation, the full failure scenario for the symptoms of this scenario requires that the power loss symptom of each interface and / or switch (206, 246) is false. However, this back propagation also means that if the current power sensor (206) of the switch SW1 is faulty, the automatic root cause analysis (ARCA) may not match the full failure scenario and therefore the root cause is not determined unless there is a matching PMPFS. In this case, there may be a PMPFS that excludes these symptoms generated by this back propagation, which generally have an associated lower probability given the uncertainty introduced by ignoring the symptoms that would otherwise be required due to back propagation.

[0100] Use a combination of back-propagation. Each of the earlier techniques or methods can also be applied to backpropagation, including: 1) using subsets of backpropagated symptoms 2) using aggregations of backpropagated symptoms, and 3) using exclusions of symptom values, rather than inclusive ranges.

[0101] In general, PMPFS allow for an engineering tradeoff between accuracy of root cause analysis and computer memory / processing for a large number of PMPFS. That is, with a smaller number of PMPFS, computer memory requirements can be reduced, and / or computer processing speed can be increased. A more accurate analysis requires more computing resources than a less accurate analysis. However, beyond a certain point, there are diminishing returns for using more PMPFS because uncertainty in the correctness and availability of telemetry limits the certainty of any analysis.

[0102] Using the techniques disclosed above, the major fallacies in the traditional approaches for ARCA are recognized and addressed; the assumption of a single root cause and the assumption that determining the actual root cause is feasible to determine using the certainty from the sensor input. The sensor input may be incorrect. Generating a set of potential root causes based on matching potential failure scenarios, some of which may correspond to the same root cause failure, is disclosed, and then a refinement step is provided to produce a refined set of potential root causes.

[0103] Spell checker analogy. In this sense, these techniques are similar to spell checker applications. Such a computer program may not precisely determine the root cause, just as a spell checker may not precisely determine all misspelled words: did the author mean to say "Hello, how are you today" when "ate" is the correctly spelled word? However, it can greatly reduce the set of potential errors to consider, just as a spell checker can reduce the number of words in a document that a human needs to verify as having correct spelling.

[0104] The match set approach is important to the matching approach for root cause analysis because multiple matches are critical to avoid matching failures due to only one or a small amount of discrepancy. For example, if a potential failure scenario is specified for a broken link with symptoms indicating that power is intact to the switches / interfaces at both ends of the link, then if the power sensor for either of these interfaces fails, matching will fail to match to the broken link.

[0105] Figure 4 is a flow chart illustrating an embodiment of a process for automatic root cause analysis based on matching sets. In one embodiment, Figure 4 The process of Figure 1 The system is implemented in .

[0106] In step 402, a set of observed symptoms of a physical system is received. As referred to herein, a physical system is any system with non-transient components, such as a refrigeration system, a computer network, and / or a biological system that may be exposed to disease, such as a human. For example, a sensor may be associated with Figure 2 Each element of the network instance is associated with the observed symptom set, and the observed symptom set can include the status of each operating sensor.

[0107] In step 404, the observed set of symptoms is matched with a fault scenario database to generate a plurality of matching fault scenarios, wherein the fault scenario database includes combinations of symptoms and corresponding root cause identifiers. In one embodiment, the fault scenario database also includes one or more confidence levels.

[0108] For example, Figure 2 An example set of observed symptoms for a network might include: a The far end / SW2 end (242) of (222) does not receive the signal. This observed symptom set may correspond to a broken link a A first matching failure scenario of (222) and a second matching failure scenario of power loss at interface I1-a (204) at the near-end / SW1 end (202).

[0109] In one embodiment, matching includes adding a partial match potential failure scenario (PMPFS) to handle partial matches with a matching mechanism.

[0110] In one embodiment, adding the PMPFS includes at least one of the following:

[0111] ● Allow potential failure scenarios to specify a given symptom as excluding one or more values;

[0112] ● Allow potential failure scenarios to limit the scope of PMPFS and its symptoms, and correspondingly reduce the probability associated with PMPFS;

[0113] ● Allows potential failure scenarios to be defined across multiple inputs with aggregated symptoms; and

[0114] ● Exclude the back-propagation symptom set from PMPFS.

[0115] In one embodiment, adding PMPFS includes at least one of: using a subset of back-propagated symptoms; using an aggregation of back-propagated symptoms; and using an exclusion of back-propagated symptom values.

[0116] In step 406, multiple matching fault scenarios are refined. Figure 5A and Figure 5BDetails of how to refine matching fault scenarios are described. As will be discussed in more detail below, the refinement includes: determining whether a first fault scenario will be merged with a second fault scenario, and if so, including the second fault scenario in multiple matching fault scenarios; deprioritizing the first fault scenario; or doing both of the above. In the following example, the first fault scenario is one of the matching fault scenarios, and the second fault scenario can be another of the matching fault scenarios or a fault scenario associated with an expression having the first fault scenario as a parameter. As used herein, when a first fault scenario is merged with a second fault scenario, the first fault scenario can be changed, marked, deprioritized, removed from the matching fault scenarios, or otherwise affected by the second fault scenario. In some embodiments, the refinement further includes: associating matching fault scenarios according to relationships. In some embodiments, the relationship includes at least one of the following: a symptom relationship, a root cause identifier relationship, and a confidence level relationship.

[0117] Matching failure scenarios are then output as potential root causes associated with the observed set of symptoms.

[0118] Figure 5A is a flow chart illustrating an embodiment of a process for determining whether a first fault scenario among a plurality of matching fault scenarios is to be merged with a second fault scenario. Figure 4 Step 406 includes Figure 5A process 500.

[0119] In this example, the refinement of the matching fault scenarios includes: determining whether a first fault scenario in a plurality of matching fault scenarios will be merged with a second fault scenario, which in turn includes: comparing the plurality of matching fault scenarios in pairs in step 502. For example, a disconnected link scenario is compared with an interface power loss scenario. In step 504, in response to a determination that a first matching fault scenario in a plurality of matching fault scenarios is merged with a second fault scenario, the first matching fault scenario is deprioritized. In some embodiments, the deprioritization includes: removing the first matching fault scenario from a set of matching fault scenarios. In some embodiments, the deprioritization includes: associating the first matching fault scenario with an alternative explanation root cause of a lower probability. In optional step 506, the refinement includes: sorting the plurality of matching fault scenarios according to a confidence level associated with a given matching fault scenario. In one embodiment, the plurality of matching fault scenarios are output as potential root causes of the observed symptoms.

[0120] In one embodiment, in the case where it is determined that the first matching fault scenario is a base scenario and the second matching fault scenario is a scenario derived from the base scenario, the first matching fault scenario is determined to be merged by the second matching fault scenario.

[0121] In one embodiment, when the first matching failure scenario and the second matching failure scenario share the same root cause identifier and the first matching failure scenario confidence level is lower than the second matching failure scenario confidence level, it is determined that the first matching failure scenario is a base scenario and the second matching failure scenario is a scenario derived from the base scenario.

[0122] In one embodiment, refining the plurality of matching failure scenarios further includes indicating the first matching failure scenario and the second matching failure scenario as part of the root cause group if the first matching failure scenario is determined to be an alternative explanation for the second matching failure scenario.

[0123] In one embodiment, where the first matching failure scenario and the second matching failure scenario share the same or similar set of symptoms, it is determined that the first matching failure scenario is an alternative explanation for the second matching failure scenario.

[0124] In one embodiment, refining the plurality of matching failure scenarios further includes, if a first matching failure scenario is determined to be a possible failure separate from a second matching failure scenario, indicating that the first matching failure scenario and the second matching failure scenario are not part of the same root cause group.

[0125] In one embodiment, where the first matching fault scenario and the second matching fault scenario share neither the same set of symptoms nor the same root cause identifier, it is determined that the first matching fault scenario is a possible fault separate from the second matching fault scenario.

[0126] Figure 5B is a flow chart illustrating another embodiment of a process for determining whether a first fault scenario among a plurality of matching fault scenarios is to be merged with a second fault scenario. Figure 4 Step 406 includes process 550 .

[0127] In this example, it is assumed that before process 550, certain metadata is established to be associated with individual fault scenarios. The metadata specifies, among other things, a predefined expression that includes the corresponding fault scenario as a parameter. In various embodiments, the association can be performed by adding additional fields in the database to the metadata associated with the fault scenario, by setting up a pointer or reference to the corresponding expression from the individual fault scenario, or by using any other appropriate technology. The fault scenario can be regarded as a base scenario, and the expression can be regarded as defining the corresponding derived scenario. For example, for the fault scenario FS1 corresponding to the root cause RC1, its corresponding metadata specifies the expression FSk=(FS1 or FS2) and (FS3 or FS4), where FS2, FS3 and FS4 are other fault scenarios included in the database. When FS1 is found to be a matching fault scenario by matching the observed symptoms, FS2, FS3 and FS4 may or may not be in the matching fault scenario, depending on the specific set of observed symptoms. The expression can be specified by a user or system administrator and generated by a compiler based on a fault model that analyzes the mapping of all symptoms to fault scenarios and uses Boolean algebra and / or any other appropriate technology to determine the expression. Assume further that a first failure scenario is selected, such as a failure scenario from the matching failure scenarios.

[0128] In this example, in step 552, an expression having a first fault scenario as a parameter and corresponding to a second fault scenario is determined. The determination of the expression can be performed by checking the corresponding metadata storing the expression. For example, the metadata corresponding to the first fault scenario may include an expression, or a pointer or reference to an expression object. Taking the example discussed above as an example, for fault scenario FS1, the expression corresponds to another fault scenario FSk, which is specified as: FSk=(FS1 or FS2) and (FS3 or FS4).

[0129] In step 554, it is determined whether the expression meets a preset criterion. In various embodiments, the preset criterion can be that the expression is true, the expression is false, the expression has a certain value, the expression exceeds a certain threshold, etc. In some expressions, additional parameters in the expression are also determined, and the expression is evaluated to determine whether the expression meets the preset criterion. This determination can be based on, for example, a set of matching fault scenarios. In the FSk example above, the determination is based on whether FS3 or FS4 is true / present in the set of matching fault scenarios. The parameters are applied to the expression to obtain a result, which is compared to the preset criterion (in this case, whether FSk is true).

[0130] If the expression does not meet the preset criteria, nothing additional needs to be done, and the process 550 can be repeated for the next fault scenario in the set of matching fault scenarios or end. However, if the expression meets the preset criteria, in various embodiments, at 556, the first fault scenario is considered to be merged with the second fault scenario. Continuing the above example, considering that FS1 is true, if neither FS3 nor FS4 is true, the expression of FSk does not meet the preset criteria for the expression to be true, and the process repeats or ends. However, if FS3, FS4, or both are true, the expression of FSk is true and meets the preset criteria, and FS1 is considered to be merged with FSk. Therefore, FSk is included in the set of matching fault scenarios to be output, FS1 is deprioritized (removed from the matching fault scenarios or assigned a lower probability), or both of the above are performed. In some embodiments, FS2, FS3, or FS4 (whichever is true) is also deprioritized. At this point, the process 550 can be repeated for the next matching fault scenario, or if all matching fault scenarios have been processed, the process 550 ends.

[0131] Fig. 6A , Figure 6B and Figure 6C is a flow chart illustrating an embodiment of a process for refining multiple matching fault scenarios based on relationships. In one embodiment, Figure 4 Step 406 includes Fig. 6A , Figure 6B and Figure 6C If it is determined in step 602 that X and Y have the same root cause identifier, control is transferred to step 604 ; otherwise, control is transferred to step 624 .

[0132] If it is determined in step 604 that X has a symptom subset of Y, then control is transferred to step 606, where X is indicated as being merged by Y; symmetrically, if not, but if in step 605, Y has a symptom subset of X, then in step 607, Y is merged by X; otherwise, control is transferred to step 608, in which it is identified that the two potential failure scenarios present in the matching set are actually two different sets of symptoms for the same root cause, and in fact both may be true, so the output contains the potential root cause, possibly with an associated probability, which is a combination of the output probabilities of the two potential failure scenarios associated with X and Y.

[0133] If it is determined in step 624 that X has the same or similar set of symptoms as Y, control is transferred to step 626, in which X is indicated as an alternative explanation to Y; otherwise, control is transferred to step 628, in which X is indicated as a possible fault separate from y. In this step 624, X and Y do not have to have exactly the same set of symptoms, because at this point it is known that they both match actual fault scenarios. For example, it is possible that in the column where X is true, Y has "don't care". The actual test is either X matches Y through a ternary match, or Y matches X. Otherwise, they are separate faults, which means there are two separate faults to remedy.

[0134] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system for automatic root cause analysis based on matching sets, comprising: processor; as well as A memory coupled to the processor, wherein the memory is configured to provide instructions to the processor that, when executed, cause the processor to: receiving a set of observed symptoms of a physical system; matching the observed set of symptoms to a failure scenario database to generate a plurality of matching failure scenarios, wherein the failure scenario database includes combinations of symptoms and corresponding root cause identifiers; and Refining the multiple matching failure scenarios includes: determining whether a first fault scenario among the plurality of matching fault scenarios is to be merged with a second fault scenario; and In response to a determination that a first failure scenario in the plurality of matching failure scenarios is to be merged with a second failure scenario: including the second failure scenario in the plurality of matching failure scenarios, deprioritizing the first failure scenario, or both including the second failure scenario in the plurality of matching failure scenarios and deprioritizing the first failure scenario, wherein deprioritizing includes associating the first failure scenario with an alternative explanation root cause of lower probability; and Output multiple matching failure scenarios as potential root causes.

2. The system of claim 1, wherein: The second failure scenario is one of the plurality of matching failure scenarios; and Determining whether a first failure scenario in the plurality of matching failure scenarios is to be merged with a second failure scenario includes comparing the plurality of matching failure scenarios in pairs. 3 . The system of claim 1 , wherein de-prioritizing the first failure scenario comprises removing the first failure scenario from the plurality of matching failure scenarios.

4. The system according to claim 1, wherein: De-prioritizing the first failure scenario includes identifying the first failure scenario as an alternative explanation associated with one or more root causes that lead to the observed set of symptoms.

5. The system according to claim 1, wherein: The failure scenario database further includes one or more confidence levels.

6. The system according to claim 5, wherein: Refining the multiple matching fault scenarios further includes associating the matching fault scenarios according to relationships.

7. The system according to claim 6, wherein: The relationship includes at least one of: a symptom relationship, a root cause identifier relationship, and a confidence level relationship.

8. The system according to claim 1, wherein: Refining the plurality of matching failure scenarios further includes ranking the plurality of matching failure scenarios according to a confidence level associated with a given matching failure scenario.

9. The system according to claim 1, wherein: In response to a determination that the first failure scenario is a base scenario and the second failure scenario is a scenario derived from the base scenario, the first failure scenario is determined to be merged by the second failure scenario.

10. The system according to claim 9, wherein: A determination is made that the first failure scenario is the base scenario and the second failure scenario is a scenario derived from the base scenario, the determination being made in response to the first failure scenario and the second failure scenario sharing a same root cause identifier and a confidence level of the first failure scenario being lower than a confidence level of the second failure scenario.

11. The system according to claim 1, wherein: Responsive to a determination that the first failure scenario is an alternate explanation for the second failure scenario, refining the plurality of matching failure scenarios further includes indicating the first failure scenario and the second failure scenario as part of a root cause group.

12. The system according to claim 11, wherein: In response to the first failure scenario and the second failure scenario sharing the same set of symptoms, the first failure scenario is determined to be an alternative explanation for the second failure scenario.

13. The system of claim 1, wherein: In response to a determination that the first failure scenario is a separate possible failure from the second failure scenario, refining the plurality of matching failure scenarios further includes indicating that the first failure scenario and the second failure scenario are not part of a same root cause group.

14. The system according to claim 13, wherein: In response to the first failure scenario and the second failure scenario sharing neither the same set of symptoms nor the same root cause identifier, it is determined that the first failure scenario is a separate possible failure from the second failure scenario.

15. The system of claim 1, wherein: Determining whether a first fault scenario among the plurality of matching fault scenarios is to be merged with a second fault scenario comprises: determining an expression having the first failure scenario as a parameter and corresponding to the second failure scenario; Determining whether the expression satisfies a preset criterion; as well as In response to the expression satisfying a preset criterion, the first fault scenario is considered to be merged with the second fault scenario.

16. The system of claim 1, wherein: Matching the observed symptom set includes adding a partial match potential failure scenario PMPFS to utilize the matching mechanism to handle partial matches.

17. The system of claim 16, wherein: Additions to PMPFS include allowing potential failure scenarios to specify a given symptom as excluding one or more values.

18. The system of claim 16, wherein: Adding PMPFS includes allowing potential failure scenarios to limit the scope of the PMPFS and its symptoms, and correspondingly reduce the probability associated with the PMPFS.

19. The system of claim 16, wherein: Additions to PMPFS include allowing potential failure scenarios to be defined across multiple inputs with aggregated symptoms.

20. The system of claim 16, wherein: Added PMPFS including: Exclude backpropagation symptom set from PMPFS.

21. The system of claim 20, wherein: Adding PMPFS includes at least one of: using a subset of back-propagated symptoms; using aggregation of back-propagated symptoms; and using exclusion of back-propagated symptom values.

22. The system of claim 1, wherein: Matching the observed set of symptoms includes performing an approximate match.

23. A method for automatic root cause analysis based on matching sets, comprising: receiving a set of observed symptoms of a physical system; matching the observed set of symptoms to a failure scenario database to generate a plurality of matching failure scenarios, wherein the failure scenario database includes combinations of symptoms and corresponding root cause identifiers; and Refining the multiple matching failure scenarios includes: determining whether a first fault scenario among the plurality of matching fault scenarios is to be merged with a second fault scenario; and In response to a determination that a first failure scenario in the plurality of matching failure scenarios is to be merged with a second failure scenario: including the second failure scenario in the plurality of matching failure scenarios, deprioritizing the first failure scenario, or both including the second failure scenario in the plurality of matching failure scenarios and deprioritizing the first failure scenario, wherein deprioritizing includes associating the first failure scenario with an alternative explanation root cause of lower probability; and Output multiple matching failure scenarios as potential root causes.

24. The method of claim 23, wherein: The second failure scenario is one of the plurality of matching failure scenarios; and Determining whether a first failure scenario in the plurality of matching failure scenarios is to be merged with a second failure scenario includes comparing the plurality of matching failure scenarios in pairs.

25. The method according to claim 23, wherein: De-prioritizing the first failure scenario includes removing the first failure scenario from the plurality of matching failure scenarios.

26. The method of claim 23, wherein: De-prioritizing the first failure scenario includes identifying the first failure scenario as an alternative explanation associated with one or more root causes that lead to the observed set of symptoms.

27. The method according to claim 23, wherein: The failure scenario database also includes one or more confidence levels.

28. The method according to claim 27, wherein: The refining of the plurality of matching fault scenarios further includes associating the matching fault scenarios according to relationships.

29. The method according to claim 28, wherein: The relationship includes at least one of: a symptom relationship, a root cause identifier relationship, and a confidence level relationship.

30. The method of claim 23, wherein: The refining of the plurality of matching failure scenarios further includes ranking the plurality of matching failure scenarios according to a confidence level associated with a given matching failure scenario.

31. The method of claim 23, wherein: In response to a determination that the first failure scenario is a base scenario and the second failure scenario is a scenario derived from the base scenario, the first failure scenario is determined to be merged by the second failure scenario.

32. The method according to claim 31, wherein: A determination is made that the first failure scenario is the base scenario and the second failure scenario is a scenario derived from the base scenario, the determination being made in response to the first failure scenario and the second failure scenario sharing a same root cause identifier and a confidence level of the first failure scenario being lower than a confidence level of the second failure scenario.

33. The method of claim 23, wherein: In response to the first failure scenario being determined to be an alternative explanation for the second failure scenario, the refining of the plurality of matching failure scenarios further includes indicating the first failure scenario and the second failure scenario as parts of a root cause group.

34. The method of claim 33, wherein: In response to the first failure scenario and the second failure scenario sharing the same set of symptoms, the first failure scenario is determined to be an alternative explanation for the second failure scenario.

35. The method of claim 23, wherein: In response to a determination that the first failure scenario is a separate possible failure from the second failure scenario, the refining of the plurality of matching failure scenarios further includes indicating that the first failure scenario and the second failure scenario are not part of a same root cause group.

36. The method of claim 35, wherein: In response to the first failure scenario and the second failure scenario sharing neither the same set of symptoms nor the same root cause identifier, it is determined that the first failure scenario is a separate possible failure from the second failure scenario.

37. The method of claim 23, wherein: The refinement of the multiple matching failure scenarios further includes: determining an expression having the first failure scenario as a parameter and corresponding to the second failure scenario; determining whether the expression satisfies a preset criterion; and In response to the expression satisfying a preset criterion, the first fault scenario is considered to be merged with the second fault scenario.

38. The method of claim 23, wherein: Matches for observed symptom sets include: Adding PMPFS to utilize the matching mechanism to handle partial matches.

39. The method of claim 38, wherein: Additions to PMPFS include allowing potential failure scenarios to specify a given symptom as excluding one or more values.

40. The method of claim 38, wherein: The addition of PMPFS includes allowing potential failure scenarios to limit the scope of the PMPFS and its symptoms, and correspondingly reduce the probability associated with the PMPFS.

41. The method of claim 38, wherein: Additions to PMPFS include allowing potential failure scenarios to be defined across multiple inputs with aggregated symptoms.

42. The method of claim 38, wherein: Additions to PMPFS include: Excluding the backpropagation symptom set from PMPFS.

43. The method of claim 42, wherein: The addition of PMPFS includes at least one of: using a subset of back-propagated symptoms; using an aggregation of back-propagated symptoms; and using an exclusion of back-propagated symptom values.

44. A computer program product embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: receiving a set of observed symptoms of a physical system; Match the observed symptom set with the fault scenario database to generate multiple matching fault scenarios, where The failure scenario database includes combinations of symptoms and corresponding root cause identifiers; as well as Refining the multiple matching failure scenarios includes: determining whether a first fault scenario among the plurality of matching fault scenarios is to be merged with a second fault scenario; and In response to a determination that a first failure scenario in the plurality of matching failure scenarios is to be merged with a second failure scenario: including the second failure scenario in the plurality of matching failure scenarios, deprioritizing the first failure scenario, or both including the second failure scenario in the plurality of matching failure scenarios and deprioritizing the first failure scenario, wherein deprioritizing includes associating the first failure scenario with an alternative explanation root cause of lower probability; and The plurality of matching failure scenarios are output as potential root causes.

Images