Managed entity configurator exception detection

Abstract

Anomalies are detected for a plurality of managed entities. Each configuration tree of a plurality of configuration trees represents configuration parameters at a corresponding managed entity, including annotations that identify a configuration that configures each parameter. For a particular configuration, a plurality of sub-trees is generated from the plurality of configuration trees. A set of weighted edit distances is computed from the plurality of sub-trees, each sub-tree representing a degree of difference between different pairs of sub-trees. The set of weighted edit distances is utilized to populate a distance matrix, and the distance matrix is used to identify an anomalous sub-tree(s) within the plurality of sub-trees for the particular configuration. In embodiments, a configuration corresponding to the anomalous sub-tree is deemed to have been abnormally applied by the particular configuration. Data of at least one managed entity and the particular configuration that is identified as being associated with the anomalous sub-tree is stored.

Description

Technical Field

[0001] This disclosure relates to systems, methods, and apparatus for detecting anomalies related to managed entity configurators. Background Technology

[0002] Computer systems are coupled to each other and to other electronic devices to form wired or wireless computer networks, through which computer systems and other electronic devices can transmit electronic data. Therefore, the performance of many computing tasks is distributed across a large number of different computer systems and / or many different computing environments. As computer networks proliferate and become more complex, various mechanisms have been developed to manage entities within computer networks (e.g., switches, routers, computers, etc.). Therefore, it is common to use more than one configuration tool to configure a given managed entity. For example, during the deployment of a switch, the switch is initially provided by a first configurator operated by the network administrator. Subsequently, various settings of the switch can be dynamically modified by various other configurators—for example, Virtual Router Forwarding (VRF) settings on the switch can be managed using different sets of configurators for Simple Network Management Protocol (SNMP).

[0003] While allowing multiple configurators to configure managed entities can be convenient, it can also present challenges, as configurators may inadvertently overwrite each other, resulting in undesirable configuration states in those managed entities. Summary of the Invention

[0004] The embodiments described herein identify anomalies in the configuration of a set of software-configurable managed entities applied by multiple configurators. In these embodiments, anomaly detection includes annotating a set of configuration trees, each representing a subset of at least one managed entity's configuration. In these embodiments, annotating each configuration tree includes information that identifies one or more configurators that can be used to identify the parameter modified by that configuration tree. Therefore, for each managed entity configured by more than one configurator, its corresponding annotated configuration tree can be used to determine which configurator(s)(s) is applied to which configuration settings(s).

[0005] In an embodiment, for a selected configurator, the anomaly detection described herein includes a set of subtrees generated from an annotated set of configuration trees, where each subtree is generated from a corresponding configuration tree and includes parameters(s) modified by the selected configurator in that configuration tree. Thus, for the selected configurator, the set of subtrees represents the parameters configured by that configurator through this set of managed entities. This subtree generation process can be repeated for each configurator, thereby generating a different set of subtrees for each configurator.

[0006] In embodiments, the anomalies described herein include using the set of subtrees of the topic configurator to determine which applied configuration is anomalous in one or more managed entities. In embodiments, this includes calculating a weighted edit distance for each subtree set in the topic configurator's subtree set, filling a distance matrix with those edit distances, and then using the distance matrix to determine whether any subtree in a set is anomalous in edit distance relative to subtrees in other sets. For each subtree identified as anomalous, a data structure indicating that the topic configurator may have applied an anomalous configuration to the managed entity associated with that subtree is generated.

[0007] Therefore, the embodiments described herein address technical challenges encountered when managing advanced networks, such as those found in data centers, including the ability to detect when configurators have overwritten each other. The technical effect of the embodiments described herein is that they identify configurators with anomalous configurations on managed entities within the network. These configurators can be used to alert network administrators to the anomaly and / or to take remedial action in light of the anomaly. Anomaly detection enables network administrators to remedy the causes of anomalies to prevent future anomalies and to correct any undesirable configuration states resulting from such anomalies. This improves network operation by identifying and eliminating errors, keeping network services available and functioning correctly, and enhancing the user experience.

[0008] In some aspects, the technology described herein relates to a method implemented in a computer system, including a processor, for detecting abnormal configurators for multiple managed entities. This method includes: identifying multiple configuration trees, each configuration tree using a hierarchical set of nodes to represent a set of managed entity configuration settings applied to corresponding managed entities across the multiple managed entities; each configuration tree identifying one or more parameters, the parameters being annotated with information about a configurator that identifies the configuration parameters, the configurator being selected from multiple configurators across the multiple managed entities; and generating multiple subtrees from the multiple configuration trees based on a specific configurator among the multiple configurators, each subtree corresponding to a different configuration tree among the multiple configuration trees and representing a specific configurator. The configurator applies one or more configuration settings; calculates a set of weighted edit distances from multiple subtrees, each weighted edit distance representing the degree of difference between different pairs of subtrees in the multiple subtrees, each weighted edit distance being weighted based on a decreasing function of node depth within at least one subtree; populates a distance matrix using the set of weighted edit distances; identifies one or more anomalous subtrees within the multiple subtrees using the distance matrix; and, based on identifying one or more anomalous subtrees within the multiple subtrees, stores a data structure identifying at least one of the following: (a) at least one managed entity from one or more managed entities associated with the one or more anomalous subtrees, and (b) the specific configurator.

[0009] In some aspects, the techniques described herein relate to a computer system for detecting anomalous configurators among multiple managed entities, comprising: a processor; and a computer storage medium storing computer-executable instructions executable by the processor to cause the computer system to at least: identify multiple configuration trees, each configuration tree using a hierarchical set of nodes to represent a set of managed entity configuration settings that have been applied to a corresponding managed entity configuration setting among the multiple managed entities, each configuration tree including one or more parameters, the one or more parameters utilizing configurator annotations identifying the configuration parameters, the configurators being selected from multiple configurators among the multiple managed entities; and generating multiple subtrees from the multiple configuration trees based on a specific configurator among the multiple configurators, each subtree corresponding to... Different configuration trees in multiple configuration trees and representing one or more configuration settings applied by a specific configurator; calculating a set of weighted edit distances from multiple subtrees, each weighted edit distance representing the degree of difference between different pairs of subtrees in the multiple subtrees, each weighted edit distance being weighted based on a decreasing function of node depth within at least one subtree; using the set of weighted edit distances to populate a distance matrix; using the distance matrix to identify one or more anomalous subtrees within the multiple subtrees, and based on identifying one or more anomalous subtrees within the multiple subtrees, storing a data structure that identifies at least one of the following: (a) at least one managed entity among one or more managed entities associated with one or more anomalous subtrees, and (b) a specific configurator.

[0010] In some aspects, the technology described herein relates to a set of computer storage media storing computer-executable instructions, which are computer-executable to cause a computer system to at least: identify multiple configuration trees, each configuration tree using a hierarchical set of nodes to represent a set of managed entity configuration settings that have been applied to a corresponding managed entity in multiple managed entities, each configuration tree including one or more parameters, the one or more parameters utilizing information annotations of configurators identifying the configuration parameters, the configurators being selected from multiple configurators of multiple managed entities; and generating multiple subtrees from the multiple configuration trees based on a specific configurator among the multiple configurators, each subtree corresponding to... The configuration is applied to different configuration trees in multiple configuration trees and represents one or more configuration settings applied by a specific configurator; a set of weighted edit distances is calculated from multiple subtrees, each weighted edit distance representing the degree of difference between different pairs of subtrees in the multiple subtrees, each weighted edit distance being weighted based on a decreasing function of node depth within at least one subtree; the set of weighted edit distances is used to populate a distance matrix; using the distance matrix, one or more aberrant subtrees in the multiple subtrees are identified, and a data structure is stored that identifies at least (a) one or more managed entities associated with one or more aberrant subtrees and (b) a specific configurator.

[0011] This summary is provided to introduce some concepts in a simplified form, which are described in more detail in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the protected subject matter. Attached Figure Description

[0012] To describe how the above and other advantages and features of the invention can be obtained, a more specific description of the invention, which has been briefly described above, will be presented with reference to specific embodiments of the invention shown in the accompanying drawings. It should be understood that these drawings are merely exemplary embodiments of the invention and should not be construed as limiting its scope. The invention will be described and explained with additional specificity and detail using the drawings, in which:

[0013] Figure 1 An example computer architecture is shown to facilitate anomaly detection by managed entity configurators;

[0014] Figure 2 An example of an exception configurator identifier is shown;

[0015] Figure 3 An example of a configuration tree is shown;

[0016] Figure 4 An example of a configuration tree with annotations is shown;

[0017] Figures 5A-5D It shows from Figure 4 Examples of configuration tree-specific subtrees generated in the commented configuration tree;

[0018] Figure 6 An example of a set of subtrees in a specific configurator is shown;

[0019] Figure 7 A flowchart is shown for an example method of detecting anomalous configurators in a set of managed entities; and

[0020] Figure 8 An example of a distance matrix is ​​shown. Detailed Implementation

[0021] Figure 1An example computer architecture 100 facilitating anomaly detection for managed entity configurators is illustrated. As shown, computer architecture 100 includes a computer system 101 comprising a processor 102 (or more processors), memory 103, one or more computer storage media (storage media 104), and a network interface 105, all interconnected via a bus 106. As shown, network interface 105 communicates via networks(multiple) 107, which connects to managed entities 108 (i.e., managed entities 108a to 108n) and configurators 110 (i.e., configurators 110a to 110m).

[0022] In this embodiment, each managed entity in managed entity 108 is a computing device whose configuration is modified via software. In various examples, managed entity 108 may include a switch, router, computer, etc. In computer architecture 100, configurator 110 is used to apply configuration settings to managed entity 108, such as by modifying hardware, software, and / or firmware configuration parameters. Therefore, as shown, each managed entity in managed entity 108 includes a corresponding configuration (e.g., configuration 109a for managed entity 108a, configuration 109n for managed entity 108n).

[0023] In an embodiment, more than one configurator in configurator 110 can be enabled to apply configuration settings to a given managed entity of managed entity 108. For example, if managed entity 108a is a switch, one configurator (such as configurator 110a) can apply initial provisioning settings to the switch, and one or more configurators (such as configurator 110m) can later dynamically modify portions of the switch configuration (such as modifying VRF settings, modifying SNMP settings) during its operation. Therefore, configurator 109a of managed entity 108a can be modified by various configurators of configurator 110 at various events. As a result, unintentionally, one or more configurators 110 can modify the same configuration within 109a, leading to an undesirable configuration state in managed entity 108a. Although configurators 110 are considered separate from managed entity 108, in an embodiment, configurators can be integrated into the managed entity (such as a local console of the managed entity).

[0024] In various embodiments, computer system 101 is configured to detect when one or more configurators of configurator 110 have made an anomalous configuration change within managed entity 110, and generate data that can be used to alert the administrator of computer architecture 100 about the detected anomaly. Therefore, computer system 101 enables network administrators to rectify the cause of the anomaly and prevent future anomalies (e.g., by correcting the configuration of configurator 110), as well as correct any unwanted configuration states resulting from this anomaly (e.g., by correcting an anomalous application configuration in managed entity 108). This improves the functionality of computer architecture 100 by identifying and eliminating errors caused by configurator 110, keeping network devices (such as those provided by managed entity 108) available and functioning correctly, and improving the experience of network users (such as clients of managed entity 108).

[0025] Storage medium 104 is shown storing computer-executable instructions that at least implement exception configurator identifier 112, and Figure 2 Example 200 of component 200 with exception configurator identifier 112 is shown. Figure 2 Each component of the depicted exception configurator identifier 112 represents a different function that the exception configurator identifier 112 may implement according to the various embodiments described herein. However, it should be understood that the depicted components—including their identity and arrangement—are presented merely as aids in describing the various embodiments of the exception configurator identifier 112.

[0026] In an embodiment, the exception configurator identifier 112 operates based on a configuration tree 114 (such as configuration tree 114a to configuration tree 114n, shown in one embodiment as residing in storage medium 104) generated by configuration tree generator 111. Figure 1 In this embodiment, configuration tree generator 111 is shown as separate from exception configurator identifier 112, but in some embodiments, configuration tree generator 111 is a component of exception configurator identifier 112. In embodiments, configuration tree generator 111 may even be separate from computer system 101, such that configuration tree 114 is received from one or more other computer systems via networks(multiple) 107.

[0027] In an embodiment, configuration tree generator 111 generates a different configuration tree for each managed entity in managed entity 108. In this embodiment, configuration tree generator 111 generates each configuration tree based on telemetry received from the corresponding managed entity in managed entity 108 (such as telemetry 113, shown as optionally residing in storage medium), the corresponding managed entity representing at least a portion of the configuration of the managing entity. Thus, in the example, configuration tree 114a corresponds to managed entity 108a and represents at least a portion of configuration 109a, and configuration tree 114n corresponds to managed entity 108n and represents at least a portion of configuration 109n.

[0028] In some embodiments, telemetry 113 includes a hierarchical data model, and configuration tree generator 111 generates configuration trees from the telemetry for managed entities given the model. The data model may be based on an explicit data model language (such as Yang) or an implicit data model language, such as a command-line interface (CLI). For CLI, nodes represent CLI keywords and parameters. In embodiments, each configuration tree represents the configuration state of the corresponding managed entity at a given time, such that the configuration tree is a snapshot of the configuration state of that managed entity. In an example where the managed entity is a network device (such as a switch or router), the configuration tree may represent the output of a “show running-config” command issued by the network device.

[0029] Figure 3 Example 300 of a configuration tree is shown. In example 300, the configuration tree includes multiple nodes, including a root node 301; intermediate nodes 302-308, 310, 317, and 318; and leaf nodes 309, 311-316, 319, and 320. In embodiments, intermediate nodes represent features and sub-features, and leaf nodes represent parameters (such as those used for features / sub-features). For example, in example 300, if the configuration tree represents the configuration of a switch, intermediate node 302 could represent the feature “VRF,” where VRF stands for Virtual Routing Function; intermediate node 306 could represent the sub-feature “Instance”; and leaf nodes 311-314 could represent parameters 1-4 (i.e., instance numbers), respectively. Furthermore, in Example 300, if the configuration tree represents the configuration of the switch, intermediate node 303 can represent the feature "SNMP", where SNMP stands for Simple Network Management Protocol; intermediate nodes 307 and 308 can represent the sub-features "iface" and "enable", respectively; and leaf nodes 315 and 316 can represent the parameter Mgm1 and the trap "traps", respectively.

[0030] In an embodiment, using tree annotator 201, the exception configurator identifier 112 annotates each configuration tree in at least a subset of configuration trees 114 with information identifying which configurator in configurators 110 modified at least a subset of the parameters represented by that configuration tree. While tree annotator 201 can annotate existing configuration trees (e.g., those generated by configuration tree generator 111), in this embodiment, tree annotator 201 operates in conjunction with the generation of each configuration tree to annotate the configuration trees during its generation process. It is noteworthy that it is possible for a configuration tree to be missing any of its represented parameters (e.g., if the corresponding managed entity has an empty configuration). In this case, the exception configurator identifier 112 avoids considering that configuration tree.

[0031] In some embodiments, the annotations include the user configured for the application (e.g., the user of the configurator), the location configured for the application (e.g., the address of a remote internet device, a local console connection, etc.), etc. In various embodiments, information for these annotations is derived from telemetry 113, logs obtained from configurator 110, etc.

[0032] Figure 4 Example 400 illustrates a configuration tree with annotations. Specifically, Example 400 shows the configuration tree of Example 300, but with annotations added to each leaf node of the configuration tree (e.g., annotation 401a for leaf node 311, annotation 401b for leaf node 312, annotation 401c for leaf node 313, annotation 401d for leaf node 314, annotation 401e for leaf node 315, annotation 401f for leaf node 316, annotation 401g for leaf node 309, annotation 401h for leaf node 319, and annotation 401i for leaf node 320). Using various shading patterns, Example 400 also demonstrates that a unique configurator can be determined for each annotated node based on these annotations. For example, the first configurator is associated with leaf node 311, the second configurator with leaf nodes 312-314, the third configurator with leaf nodes 315 and 316, and the fourth configurator with leaf nodes 309, 319, and 320.

[0033] In an embodiment, using subtree generator 202, exception configurator identifier 112 uses these annotations to generate configurator-specific subtrees from configuration tree 114. Each configurator-specific subtree includes leaf nodes(s) configured by a particular configurator plus all their ancestors. For example, Figures 5A-5D It shows the result of Figure 4Examples 500a, 500b, 500c, and 500d show the configurator-specific subtrees generated from the annotated configuration tree. Example 500a shows a first configurator-specific subtree including leaf node 311 (and its ancestors), example 500b shows a second configurator-specific subtree including leaf nodes 312-314 (and their ancestors), example 500c shows a third configurator-specific subtree including leaf nodes 315 and 316 (and their ancestors), and example 500d shows a fourth configurator-specific subtree including leaf nodes 309, 319, and 320 (and their ancestors).

[0034] In an embodiment, subtree generator 202 generates a different set of subtrees for each configurator. For example, in Figure 1 In this embodiment, subtree generator 202 generates subtree 115 (shown in one embodiment as residing in storage medium), including a subtree set 115a corresponding to configurator 110a and a subtree set 115n corresponding to configurator 110m. Each subtree set includes a set of subsets derived from multiple configurations of configuration tree 114. For example, subtree set 115a includes a first subtree of configuration tree 114a, which includes leaf nodes (and their ancestors) configured by configurator 110a, and a first subtree of configuration tree 114n, which includes leaf nodes (and their ancestors) configured by configurator 110a. On the other hand, subtree set 115n includes a second subtree of configuration tree 114a, which includes leaf nodes (and their ancestors) configured by configurator 110m, and a second subtree of configuration tree 114n, which includes leaf nodes (and their ancestors) configured by configurator 110m.

[0035] Figure 6 Example 600 of a subtree for a specific configurator is shown. In example 600, there is a subtree 601 in configuration tree 114 corresponding to a first configuration tree, which includes leaf nodes (and their ancestors) of the first configuration tree configured by the selected configurator; a subtree 602 in configuration tree 114 corresponding to a second configuration tree, which includes leaf nodes (and their ancestors) of the second configuration tree configured by the given configurator; a subtree 603 in configuration tree 114 corresponding to a third configuration tree, which includes leaf nodes (and their ancestors) of the third configuration tree configured by the selected configurator; a subtree 604 in configuration tree 114 corresponding to a fourth configuration tree, which includes leaf nodes (and their ancestors) of the fourth configuration tree configured by the selected configurator; a subtree 605 in configuration tree 114 corresponding to a fifth configuration tree, which includes leaf nodes (and their ancestors) of the fifth configuration tree configured by the selected configurator; and a subtree 606 in configuration tree 114 corresponding to a sixth configuration tree, which includes leaf nodes (and their ancestors) of the sixth configuration tree configured by the selected configurator.

[0036] In embodiments, when configurator 110 is used appropriately, the set of subtrees generated for a given configurator is expected to be similar. That is, all configurations configured by a single configurator are generally similar for each configuration tree. In embodiments, based on actions, the subtrees of a configurator are expected to be similar in the absence of exceptions. For example, refer to... Figure 6 Subtrees 601-605 have similar structures, while subtree 606 contains an entire branch that is not present in subtrees 601-605. This means that a configurator might have abnormally affected settings for managed entities from which subtree 606 is derived, and this could potentially conflict with other configurators within that managed entity. It is worth noting that due to the number of leaf nodes, conventional techniques (such as textual comparisons of configuration parameter lists) might incorrectly identify subtree 603 as abnormal, as it contains a different number of configuration parameters than subtrees 601, 602, 604, and 605. However, this embodiment utilizes weighted edit distance calculations to ensure that subtree 603 is not incorrectly identified as abnormal.

[0037] In this embodiment, edit distance is used to represent the similarity between two subtrees. Therefore, in this embodiment, edit distance calculator 203 and exception configurator identifier 112 are used to calculate a set of edit distances for one or more sets of subtrees generated by subtree generator 112. For example, Figure 1 A set of edit distances 116 for a configurator is shown (e.g., edit distances for configurator 110a calculated from subtree set 115a), where ellipses indicate that the edit distance calculator 203 can calculate multiple sets of edit distances for configurators. Various techniques exist for calculating edit distances between trees and can be used according to the embodiments herein.

[0038] In an embodiment, for a given set of subtrees, the edit distance calculator 203 calculates at least one corresponding edit distance for each unique pairing (i.e., a non-repeating combination of two subtrees) in the set of subtrees. In the text of Example 600, for example, the edit distance calculator 203 calculates a first edit distance for the pairing of subtrees 601 and 602, a second edit distance for the pairing of subtrees 601 and 603, a third edit distance for the pairing of subtrees 601 and 604, a fourth edit distance for the pairing of subtrees 601 and 606, a fifth edit distance for the pairing of subtrees 602 and 603, a sixth edit distance for the pairing of subtrees 602 and 604, and so on, for a total of fifteen edit distances.

[0039] In embodiments, the calculation of the edit distance uses a decreasing function of node depth in the tree—that is, a function that assigns higher weights to nodes closer to the root. In various embodiments, (a) this weighting is incentivized by configuring the semantics and leaf parameters of the feature / sub-feature of the tree, where intermediate nodes define configurable features / sub-features (e.g., where non-leaf nodes closer to the root refer to broader topics, and non-leaf nodes closer to the leaves refer to narrower topics), and (b) this weighting is incentivized because configurator 110 typically configures the same features across managed entities 108. Therefore, the deviation between subtrees of a given configurator at a higher level can indicate that the configurator has modified features it would not normally modify. In various embodiments, the weighted edit distance is calculated based at least on the number of a node's direct children (i.e., the weight given to a node increases with the number of its descendants) and / or the depth of a node's descendants (i.e., the weight given to a node increases with the depth of its descendants).

[0040] In embodiments, edit distances can be represented as distance matrices. As those skilled in the art will understand, a distance matrix can be conceptualized as a two-dimensional array containing the distances obtained in pairs between set elements. Therefore, in various embodiments, using distance matrix generator 204, exception configurator identifiers populate corresponding distance matrices for one or more configurators in configurators 110, where each distance matrix includes weighted edit distances between unique subtree pairs in the subtree set for the configurator. For example, Figure 1 Distance matrix 117 is shown for a configurator (e.g., configurator 110a) based on edit distance 116, where ellipses indicate that distance matrix generator 204 can generate distance matrices for multiple configurators. In this embodiment, instead of using one-to-one comparisons between subtrees, using distance matrices can identify not only the differences between two subtrees, but also subtree clusters, peripheral subtrees, and so on.

[0041] Figure 8Example 800 illustrates a distance matrix filled with pairs of subtrees from Example 600. In Example 800, cells represent edit distances (e.g., 801–815, 801'–815'), and each cell at the intersection of a given pair represents the edit distance calculated for that pair. As shown in Example 800, the edit distance of a given subtree to itself is zero. In some embodiments, each pair has the same edit distance, regardless of the order of the pairs (e.g., the edit distance 801 of pairing subtrees 602 and 601 is the same as the edit distance 801' of pairing subtrees 601 and 602). However, in other embodiments, swapping the order of the pairs may result in different edit distances. It is understood that, given Example 800, while the distance matrix may be conceptualized as a two-dimensional array, it does not need to be stored in this way. For example, in an embodiment, only the identifier of each subtree and the values ​​of the edit distances 801–815 are stored, and the distance matrix is ​​derived from this.

[0042] In various embodiments, an anomaly detector 205 uses an anomaly configurator identifier to analyze a distance matrix to determine whether the distance matrix represents an anomalous subtree, and thus whether it represents an anomalous configuration of the corresponding configurator. In various embodiments, given a distance matrix, high pairwise values ​​represent significant differences between subtrees. A subtree is considered anomalous compared to other subtrees when it has a set of high pairwise values ​​relative to the set of pairwise values ​​of other subtrees (e.g., a set of one or more individual cell values ​​that intersect with the subtree, including sets of cell rows corresponding to the subtree, sets of cell columns corresponding to the subtree, and sets of cell rows and columns corresponding to the subtree). Once an anomalous subtree is identified, the anomaly detector 205 can determine which configurator applied the anomalous configuration (i.e., based on the set of subtrees to which the subtree belongs) and which managed entity (i.e., based on the configuration tree from which its subtrees originated).

[0043] In embodiments, anomaly detector 205 may use one or more of a variety of techniques to identify anomalous subtrees. In one technique, anomaly detector 205 applies threshold-based analysis to identify anomalies based on edit distances exceeding a threshold. In some embodiments, the threshold is predetermined, while in other embodiments, the threshold is dynamically determined based on values ​​in a matrix. In some embodiments, the threshold is based on comparing a set of (multiple) first characteristics of a set of pairwise values ​​corresponding to a subtree with (multiple) second characteristics of a subset of the full distance matrix and / or the distance matrix (e.g., the set difference between the distance matrix and the set of pairwise values ​​corresponding to the subtree). Examples of (multiple) first characteristics include the average, mean, and / or standard deviation of the set of pairwise values. Examples of (multiple) second characteristics may similarly include any combination of the average, mean, and / or standard deviation of the values ​​of the full distance matrix (and / or a subset of the distance matrix, if applicable). As a concrete example, the threshold can be set to the value of mean(matrix) + (2*standard_deviation(matrix)) reflecting the second characteristic, and the subtree values ​​compared to this threshold can be set to the value of mean(set of pairwise values) reflecting the first characteristic.

[0044] In another technique, the anomaly detector 205 utilizes machine learning and classification of distance matrices (e.g., label-based distance matrices as training data). Various classification algorithms can be trained and applied, such as linear / logistic regression, support vector machines, random forests, decision trees, k-nearest neighbors, stochastic gradient descent, etc. In yet another technique, the anomaly detector 205 utilizes hierarchical clustering to identify clusters of similar trees, thereby identifying anomalies.

[0045] In embodiments, using an anomaly data generator 206, when an anomaly detector 205 has detected an anomaly subtree, an anomaly configurator identifier is generated and stored (e.g., to memory 103 and / or storage medium 104) in a data structure. In various embodiments, this data structure includes at least the identity of one or more configurators to which the anomaly configuration has been applied, and which (or which) managed entities the anomaly configuration has been applied to. In embodiments, the anomaly data generator 206 also generates alerts that include at least a portion of this data structure. The form of the alert can vary, for example, opening a service ticket, generating an email, generating a push notification, etc. In various embodiments, these alerts are set for the administrator of computer architecture 100, enabling the administrator to remedy the cause of the anomaly and prevent future anomalies (e.g., by correcting the configuration of configurator 110), and to correct any undesirable configuration states caused by the anomaly (e.g., by correcting the configuration of the anomaly applied within managed entity 108). This improves the functionality of computer architecture 100 by identifying and eliminating faults caused by configurator 110, enabling network services (e.g., provided by managed entity 108) to remain available and function properly, and enhancing the experience of network users (e.g., clients of managed entity 108).

[0046] In some embodiments, the abnormal configurator identifier 112 initiates automatic remediation actions for the abnormality. In some embodiments, the automatic remediation action is performed on the configurator that made the abnormal configuration change. For example, the automatic remediation action could be disabling the configurator, isolating the configurator (e.g., using a firewall), or deactivating the configurator. In some embodiments, the automatic remediation action is performed on the managed entity to which the abnormal configuration was applied. Examples include migrating the service out of the managed entity, rolling back the managed entity's configuration to a previous state, isolating the managed entity, resetting the managed entity, or deactivating the managed entity.

[0047] Current combination Figure 7 To describe the component of exception configurator identifier 112, Figure 7 A flowchart of an example method 700 for detecting an anomaly configurator for multiple managed entities is shown. In various embodiments, instructions for implementing method 700 are encoded as computer-executable instructions (e.g., anomaly configurator identifier 112) stored on a computer storage medium (e.g., storage medium 104), which are executable by a processor (e.g., processor 102) to cause a computer system (e.g., computer system 101) to perform method 700.

[0048] The following discussion now involves many methods and method actions. Although method actions may be discussed in a specific order, or may occur in a flowchart as if they happen in a specific order, a specific order is not required unless specifically stated otherwise, or because an action depends on another action that is completed before the action is performed.

[0049] refer to Figure 7 In some embodiments, method 700 includes an action 701 of generating a set of configuration trees. In some embodiments, action 701 includes generating multiple configuration trees based on telemetry obtained from multiple managed entities. In an example, configuration tree generator 111 generates configuration tree 114 based on telemetry 113 received from managed entity 108. In some embodiments, the telemetry 113 is received from each managed entity 108 based on a request sent from computer system 101 to the managed entity. In other embodiments, the telemetry 113 is received from each managed entity 108 without a request from computer system 101 based on those managed entities that pushed the telemetry 113 to computer system 101. Action 701 is shown in dashed lines to indicate that it can be an optional step of method 700. For example, in an embodiment, computer system 101 receives configuration tree 114 from other computer systems(s) rather than generating it itself. The technical effect of action 701 includes obtaining a representation of the configuration of managed entity 108.

[0050] In some embodiments, 700 further includes an action 702 of annotating a set of configuration trees with configurator information. In some embodiments, action 702 includes annotating multiple configuration trees, each using a hierarchical set of nodes to represent a set of configuration settings applied to a corresponding managed entity across multiple managed entities. Annotating each configuration tree includes annotating one or more parameters represented by the configuration tree with information identifying the configurator that configured the parameter, selected from multiple configurators across multiple managed entities. In an example, tree annotator 201 annotates at least a subset of configuration tree 114 with annotations indicating or usable to identify which configurator modified at least a subset of the parameters represented by the configuration tree. For example, as shown in Example 400, the configuration tree contains leaf nodes annotated to represent three different configurators (i.e., a first configurator associated with leaf node 311, a second configurator associated with leaf nodes 312-314, and a third configurator associated with leaf nodes 309, 319, and 320). The technical effect of action 702 includes generating information sufficient to identify which configurators modified parameters at various managed entities.

[0051] In various embodiments, tree annotator 201 annotates leaf nodes, as shown in Example 400. Thus, in some embodiments of action 702, annotating one or more parameters represented by each configuration tree with information identifying the configurator that configures the parameters includes annotating one or more leaf nodes in each configuration tree.

[0052] As discussed, in various embodiments, the annotation includes the user applying the configuration (e.g., the user of the configurator), the location where the configuration is applied, etc. Therefore, in some embodiments of action 702, the information identifying at least one configurator includes at least one of the location where the specific parameter is applied or the user applying the specific parameter. In various embodiments, the location is specified as a remote network device address (e.g., IP address, MAC address), a local console connection, etc. In various embodiments, this can be observed based on telemetry obtained from the managed entity and / or the configurator, logs stored at the managed entity and / or the configurator, hooks or detours associated with the API used by the configurator, etc. Therefore, in some embodiments of action 702, the information identifying at least one configurator includes the location where the specific parameter is applied, and wherein the location includes at least one of a specified network location or a specified console connection.

[0053] Action 702 is shown in dashed lines, indicating that it may be an optional step of method 700. In these embodiments, method 700 includes identifying multiple configuration trees, each configuration tree using a hierarchical set of nodes to represent a set of configuration settings applied to a corresponding managed entity among multiple managed entities, each configuration tree including one or more parameters annotated with information identifying the configurator that configured the parameter. In an example, an exception configurator identifier 112 identifies multiple configuration trees.

[0054] Method 700 includes action 703 of generating a set of subtrees for a selected configurator. In some embodiments, action 703 includes generating multiple subtrees from multiple configuration trees based on a particular configurator among multiple configurators, each subtree corresponding to a different configuration tree among the multiple configuration trees and representing one or more configuration settings applied by the particular configurator. In an example, for a given configurator (e.g., configurator 110a), subtree generator 202 generates a set of subtrees 115a from configuration tree 114. An example of subtree set 115a could be subtrees 601-606 as shown in example 600. The technical effect of action 703 includes generating data across multiple managed entities that indicates which settings a particular configurator modified on each managed entity.

[0055] Method 700 also includes the action 704 of calculating a set of weighted edit distances for pairs of subtrees in the set. In some embodiments, action 704 includes calculating a set of weighted edit distances from a plurality of subtrees, each weighted edit distance representing the degree of difference between different pairs of subtrees in the plurality of subtrees, each weighted edit distance being weighted based on a decreasing function of node depth within at least one subtree. In one example, edit distance calculator 203 calculates edit distance 116 based on pairs of subtrees within subtree set 115a, corresponding to configurator 110a. In embodiments, the set of weighted edit distances includes the weighted edit distances of pairs of unique subtrees in the plurality of subtrees. The technical effect of action 704 includes numerically quantifying how similar the configurations applied to managed entities by a given configurator are to each other.

[0056] As described above, in embodiments, the weighted edit distance can be calculated based on the number of a node's direct child nodes, the number of a node's descendants, and / or the depth of a node's descendants. Therefore, in some embodiments of action 704, at least one weighted edit distance is also weighted based on at least one of the following: the number of direct child nodes of a first node in at least one subtree of a different subtree pair; the number of descendants of a second node in at least one subtree of a different subtree pair; or the depth of descendants of a third node in at least one subtree of a different subtree pair.

[0057] Method 700 also includes an action 705 of filling the distance matrix with the weighted set of edit distances. In some embodiments, method 705 includes filling the distance matrix with the weighted set of edit distances. In an example, distance matrix generator 204 fills distance matrix 117 with edit distances 116 derived from subtree set 115a of configurator 110a. The technical effect of action 705 includes combining the weighted edit distances calculated in action 704 into a form that can be used to identify subtree clusters in the set (e.g., as normal subtrees), peripheral subtrees in the set (e.g., as irregular subtrees), etc.

[0058] Method 700 also includes action 706 of using a distance matrix to identify anomalous subtrees(s). In some embodiments, action 706 includes using a distance matrix to identify one or more anomalous subtrees within a plurality of subtrees. In an example, anomaly detector 205 analyzes distance matrix 117 to identify one or more subtrees of subtree set 115a that are anomalous compared to their peers. From there, anomaly detector 205 can determine which configurator applied the anomalous configuration, and which managed entity did so. The technical effect of action 706 includes the automatic detection of anomalous configuration setting changes made by a configurator at a managed entity.

[0059] As described above, in the embodiments, the anomaly detector 205 uses one or more techniques from a variety of methods to analyze the distance matrix and identify anomalous subtrees, such as threshold-based analysis, machine learning and classification of the distance matrix, and / or hierarchical clustering. Therefore, in the embodiment of action 706, identifying one or more anomalous subtrees within a plurality of subtrees is based on determining that at least one weighted edit distance exceeds a threshold, inputting the distance matrix into a machine learning algorithm, or using hierarchical clustering to identify at least one of the non-clustered subtrees. In some embodiments, identifying one or more anomalous subtrees within a plurality of subtrees is based on inputting the distance matrix into a machine learning algorithm, and the machine learning algorithm is selected from a group consisting of linear regression, logistic regression, support vector machine, random forest, decision tree, k-nearest neighbor, or stochastic gradient descent.

[0060] Figure 7 This includes a dashed arrow connecting action 706 to action 703. This arrow indicates that actions 703 through 706 can be repeated, each time considering a different selected configurator. Therefore, method 700 can generate and analyze a subtree for each configurator of configurator 110.

[0061] Method 700 also includes action 707 of storing a data structure that identifies anomalies. In some embodiments, method 707 includes storing a data structure that identifies at least (a) at least one managed entity among one or more managed entities associated with the one or more abnormal subtrees, and (b) a particular configurator, based on identifying one or more abnormal subtrees within a plurality of subtrees. In one example, anomaly data generator 206 stores the data structure in memory 103 and / or storage medium 104, the data structure including the identity of configurator 110a and the identity of the managed entity to which configurator 110a applies abnormal configuration. The technical effect of action 707 includes the generation of data that can be used to identify and remedy anomalies.

[0062] In some embodiments, the anomaly data generator 206 also generates an alert that includes the identity of configurator 110a and the identity of the managed entity to which the anomaly configuration applied by configurator 110a is located. In various embodiments, the alert is a service ticket, email, push notification, etc. The technical effects include alerting the administrator to the existence of the anomaly configurator, enabling the administrator to correct the cause of the anomaly and prevent future anomalies (e.g., by correcting the configuration of configurator 110), and correcting any undesirable configuration states caused by the anomaly (e.g., by correcting the configuration of the anomaly application within managed entity 108).

[0063] In an embodiment, the abnormal configurator identifier 112 initiates an automatic remediation action for the abnormality. For example, in some embodiments, method 700 further includes initiating an automatic remediation action, including at least one of the following: disabling a specific configurator; isolating a specific configurator; deactivating a specific configurator; migrating a service from at least one managed entity associated with one or more abnormal subtrees; rolling back the configuration of at least one managed entity associated with one or more abnormal subtrees; isolating at least one managed entity associated with one or more abnormal subtrees; resetting at least one managed entity associated with one or more abnormal subtrees; or deactivating at least one managed entity associated with one or more abnormal subtrees. Depending on the actions taken, the technical effects may include preventing further damage from a configurator undergoing an abnormal configuration change, preventing the use of services from an incorrectly configured managed entity, and restoring the abnormal configuration change.

[0064] Although the subject matter has been described using language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above, or the order of such actions. Rather, the described features and actions are disclosed as exemplary forms of implementing the claims.

[0065] Embodiments of the present invention may include or utilize a dedicated or general-purpose computer system (e.g., computer system 101) comprising computer hardware, such as one or more processors (e.g., processor 102) and system memory (e.g., memory 103), as discussed in more detail below. Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or dedicated computer system. Computer-readable media storing computer-executable instructions and / or data structures are computer storage media (e.g., storage media 104). Computer-readable media carrying computer-executable instructions and / or data structures are transmission media. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0066] Computer storage media are physical storage media that store computer-executable instructions and / or data structures. Physical storage media include computer hardware such as RAM, ROM, EEPROM, solid-state drives (“SSDs”), flash memory, phase-change memory (“PCM”), optical disc storage, magnetic disk storage, or other magnetic storage devices, or any other hardware storage device that can be used to store program code in the form of computer-executable instructions or data structures, which can be accessed and executed by general-purpose or special-purpose computer systems to perform the functions disclosed in this invention.

[0067] Transmission media may include networks and / or data links, which may be used to carry program code in the form of computer-executable instructions or data structures, and which may be accessible by general-purpose or special-purpose computer systems. A “network” is defined as one or more data links capable of transmitting electronic data between computer systems and / or modules and / or other electronic devices. A computer system may consider a connection as a transmission medium when information is transmitted to or provided to it via a network or another communication connection (hardwired, wireless, or a combination of hardwired and wireless). Combinations of the above should also be included within the scope of computer-readable media.

[0068] Furthermore, upon arrival at various computer system components, code within computer-executable instructions or data structures can be automatically transferred from the transmission medium to the computer storage medium (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be cached in RAM within a network interface module (e.g., network interface 105) and then ultimately transferred to the computer system RAM and / or a less volatile computer storage medium at the computer system. Therefore, it should be understood that computer storage media can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0069] Computer-executable instructions include, for example, instructions and data that, when executed on one or more processors, cause a general-purpose computer system, a special-purpose computer system, or a special-purpose processing device to perform a particular function or group of functions. Computer-executable instructions can be, for example, binary code, such as intermediate format instructions in assembly language, or even source code.

[0070] Those skilled in the art will understand that this invention can be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, graphics tablets, pagers, routers, switches, etc. This invention can also be practiced in distributed system environments, where tasks are performed by local and remote computer systems via network links (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links). Thus, in a distributed system environment, the computer system can include multiple constituent computer systems. In a distributed system environment, program modules can reside in local and remote memory storage devices.

[0071] Those skilled in the art will also understand that the present invention can be implemented in a cloud computing environment. A cloud computing environment can be distributed, although this is not required. When distributed, a cloud computing environment can be distributed internationally within an organization and / or have components owned across multiple organizations. In this specification and the appended claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any of the many other advantages that can be obtained from such a model when properly deployed.

[0072] Cloud computing models can be composed of various characteristics, such as on-demand self-service, wide area network access, resource aggregation, rapid elasticity, and measurement services. Cloud computing models can also appear in various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Cloud computing models can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, and hybrid cloud.

[0073] Some embodiments, such as cloud computing environments, may include systems comprising one or more hosts, each capable of running one or more virtual machines. During operation, the virtual machines emulate an operable computing system, supporting an operating system and possibly one or more other applications. In some embodiments, each host includes a hypervisor that emulates virtual resources of the virtual machines using physical resources extracted from a view of the virtual machines. The hypervisor also provides appropriate isolation between virtual machines. Thus, from the perspective of any given virtual machine, the hypervisor provides the illusion of an interface between the virtual machine and physical resources, even if the virtual machine only interacts with the appearance of physical resources (e.g., virtual resources). Examples of physical resources include processing power, memory, disk space, network bandwidth, media drives, etc.

[0074] The invention may be embodied in other specific forms without departing from its essential characteristics. Such embodiments may include a data processing apparatus comprising means for performing one or more methods described herein; a computer program comprising instructions which, when executed by a computer, cause the computer to perform one or more methods described herein; and / or a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform one or more methods described herein. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All variations within the meaning and scope of equivalents of the claims are included within their scope. When elements are introduced in the appended claims, the articles “a,” “an,” “the,” and “the” are intended to indicate the presence of one or more elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements besides those listed may be present. Unless otherwise specified, the terms "set," "superset," and "subset" are intended to exclude the empty set, and thus a "set" is defined as a non-empty set, a "superset" as a non-empty superset, and a "subset" as a non-empty subset. Unless otherwise specified, the term "subset" excludes its entire superset (i.e., the superset includes at least one item not included in the subset). Unless otherwise specified, a "superset" may include at least one additional element, while a "subset" may exclude at least one element.

Claims

1. A method (700) for detecting anomalous configurators among a plurality of managed entities (108) implemented in a computer system (101), the computer system including a processor (102), the method comprising: Identify (112) multiple configuration trees (114), each configuration tree using a hierarchical set of nodes to represent a set (109a) of configuration settings that have been applied to the corresponding managed entity (108a) among the multiple managed entities, each configuration tree including one or more parameters, the one or more parameters being annotated with information identifying the configurator that configures the parameter, the configurator being selected from a plurality of configurators (110) among the multiple managed entities; Based on a specific configurator (110a) of the plurality of configurators, a plurality of subtrees (115a) are generated (202) from the plurality of configuration trees, each subtree corresponding to a different configuration tree in the plurality of configuration trees and representing one or more configuration settings applied by the specific configurator; (203) Calculate a set of weighted edit distances (116) from the plurality of subtrees, where each weighted edit distance represents the degree of difference between different pairs of subtrees in the plurality of subtrees, and each weighted edit distance is weighted based on a decreasing function of node depth within at least one subtree; Fill the (204) distance matrix (117) with the weighted edit distance set; Using the distance matrix, identify one or more abnormal subtrees within the plurality of subtrees (205); as well as Based on the identification of the one or more abnormal subtrees within the plurality of subtrees, storage (206) includes a data structure that identifies at least one of the following: (a) at least one managed entity among the one or more managed entities associated with the one or more abnormal subtrees, and (b) the specific configurator.

2. The method of claim 1, further comprising generating (111) the plurality of configuration trees from telemetry data obtained from the plurality of managed entities.

3. The method according to any one of claims 1 or 2, further comprising initiating an automatic remedial action, including at least one of the following: Disable the specific configurator; Isolate the specific configurator; Disable the specific configurator; Migrate the service from at least one managed entity associated with the one or more anomalous subtrees; Roll back the configuration of at least one managed entity associated with the one or more anomalous subtrees; Isolate at least one managed entity associated with the one or more anomalous subtrees; Reset at least one managed entity associated with the one or more abnormal subtrees; or Deactivate at least one managed entity associated with the one or more abnormal subtrees.

4. The method according to any one of claims 1 to 2, wherein the information identifying at least one configurator includes at least one of the following: The location where specific parameters are applied; or Users who apply the specific parameters.

5. The method of claim 4, wherein the information identifying at least one configurator includes the location where the specific parameter is applied, and wherein the location includes at least one of the following: The specified network location; or The specified console connection.

6. The method according to any one of claims 1 to 2, wherein at least one weighted edit distance is further weighted based on at least one of the following: The number of direct child nodes of the first node in at least one subtree of the different subtree pairs; The number of descendants of the second node in at least one subtree of the different subtree pairs; or The depth of the descendants of the third node in at least one subtree of the different subtree pair.

7. The method according to any one of claims 1 to 2, wherein identifying the one or more abnormal subtrees within the plurality of subtrees is based on at least one of the following: Determine at least one weighted edit distance that exceeds the threshold; Input the distance matrix into a machine learning algorithm; or Use hierarchical clusters to identify non-clustered subtrees.

8. The method of claim 7, wherein identifying the one or more anomalous subtrees within the plurality of subtrees is based on inputting the distance matrix into a machine learning algorithm, and wherein the machine learning algorithm is selected from a group consisting of: Linear regression algorithm; Logistic regression algorithm; Support Vector Machine algorithm; Random Forest Algorithm; Decision tree algorithm; k-nearest neighbor algorithm; or Stochastic gradient descent algorithm.

9. The method according to any one of claims 1 to 2, wherein the set of weighted edit distances comprises paired weighted edit distances for each unique subtree among the plurality of subtrees.

10. The method according to any one of claims 1 to 2, wherein the information annotation of the configurator that identifies the configuration of the parameters, representing one or more parameters by each configuration tree, includes annotations of one or more leaf nodes in each configuration tree.

11. The method according to any one of claims 1 to 2, further comprising annotating at least one of the plurality of configuration trees with information of a configurator that identifies configuration parameters.

12. The method according to any one of claims 1 to 2, further comprising generating an alarm, the alarm comprising at least (a) at least one managed entity of the one or more managed entities associated with the one or more abnormal subtrees, and (b) the specific configurator.

13. A computer system for detecting abnormal configurators of multiple managed entities, comprising: processor; as well as A computer storage medium storing computer-executable instructions, which are executable by the processor to cause the computer system to at least: Multiple configuration trees are identified, each configuration tree using a hierarchical set of nodes to represent a set of configuration settings that have been applied to the corresponding managed entity among the multiple managed entities. Each configuration tree includes one or more parameters, which are annotated with information identifying the configurator that configures the parameter, and the configurator is selected from multiple configurators of the multiple managed entities. Based on a specific configurator of the multiple configurators, multiple subtrees are generated from the multiple configuration trees, each subtree corresponding to a different configuration tree in the multiple configuration trees and representing one or more configuration settings applied by the specific configurator; A set of weighted edit distances is calculated from the plurality of subtrees, each weighted edit distance representing the degree of difference between different pairs of subtrees in the plurality of subtrees, and each weighted edit distance is weighted based on a decreasing function of node depth within at least one subtree; Fill the distance matrix using the weighted edit distance set; The distance matrix is ​​used to identify one or more abnormal subtrees within the plurality of subtrees; as well as Based on the identification of the one or more abnormal subtrees within the plurality of subtrees, a data structure is stored that identifies at least one of the following: (a) at least one managed entity among the one or more managed entities associated with the one or more abnormal subtrees, and (b) the specific configurator.

14. The computer system of claim 13, wherein the information identifying at least one configurator includes at least one of the following: The location where specific parameters are applied; or Users who apply the specific parameters.

15. The computer system according to any one of claims 13 or 14, wherein at least one weighted edit distance is further weighted based on at least one of the following: The number of direct child nodes of the first node in at least one subtree of the different subtree pairs; The number of descendants of the second node in at least one subtree of the different subtree pairs; or The depth of the descendants of the third node in at least one subtree of the different subtree pair.

Images