Root cause analysis

Abstract

Methods and apparatus for root cause analysis. The method includes obtaining measurement data comprising measurement data of features of a system. The method also includes generating a predicted value by applying a trained machine learning model to the measurement data and also generating a feature impact value by applying a generated machine learning model explainer to the measurement data. The method also includes updating an ontological representation of connections between features of the system and the predicted value using the generated feature impact value and outputting a proposed root cause that caused the predicted value based on the updated ontological representation.

Description

Technical Field

[0001] The embodiments described herein relate to methods and apparatus for root cause analysis, and more specifically, to methods and apparatus for applying machine learning techniques to root cause analysis. Background Technology

[0002] Determining how complex systems function and why is often a time-consuming and labor-intensive task, typically requiring extensive work from systems experts. An example of a complex system in this context is telecommunications or telecommunications networks; the performance of a telecommunications network (e.g., a mobile telecommunications network) can be assessed by evaluating the quality of experience for the network's end users (also known as network customers).

[0003] Customer experience management (CEM) can be challenging for mobile telecommunications network operators. Providing service guarantees (SA) to end users—that is, meeting various quality criteria (examples may include minimum data throughput, maximum dropped call rate, etc.)—typically requires end-to-end observability of the service. Network management across multiple domains may also be necessary.

[0004] Figure 1 This is a schematic overview of a typical end-to-end telecommunications network CEM architecture, where low-level network measurement data features (e.g., logs, tracks, probes, counters, measurement reports, etc.) are streamed to a central management entity (e.g., a core network node). These features can originate from many different network domains, such as User Equipment (UE), Radio Access Network (RAN), Core Network, Transport Network, Services, etc. To identify relevant data associated with a given mobile user, measurements collected from multiple domains need to be correlated, enabling operators to monitor key characteristics and performance of end-user sessions. The complexity of network management data increases because the diverse measurement data from multiple domains may also involve various access technologies, network layers, and protocols. Furthermore, some relevant data may be inaccessible to network management because specific network domains may be managed by third parties (e.g., over-the-top services, transport services, etc.).

[0005] Key performance indicators (KPIs) can be used to summarize network performance; using a telecommunications network as an example, examples of KPIs could include packet loss rate, latency, etc. KPIs can be directly monitored or estimated; returning to the telecommunications network example, they can be directly monitored or estimated using, for example... Figure 1 The low-level network measurement data shown is used to estimate telecommunications KPIs.

[0006] Further analytical methods are needed to perform more in-depth analysis of end-user sessions. By manually analyzing relevant measurements from multiple domains, access technologies, etc., network operators can gain a high level of insight and may be able to identify potential actions to be taken on the network in cases of poor and / or degraded network performance.

[0007] Root cause analysis (RCA) is the process of identifying the primary source of problems or performance degradation; typically, this process is performed by technically skilled operators using knowledge of network, KPIs, and low-level network measurement data. After finding the root cause of the problem / performance degradation, operators may potentially take further actions to fix the problem and / or modify the network to reduce the risk of similar problems recurring. To reduce the burden on operators from performing RCA, machine learning (ML) techniques can be used to assist operators by performing high-level analysis in network management; however, a considerable burden still remains on operators.

[0008] US 9,774,506B2 discloses how to explore causal relationships between events based on the temporal sequence of different events / event bursts. The system uses a bottom-up approach, where event bursts are detected and causal relationships between events and system operation reports are identified based on the detected event burst records representing the occurrence of burst behaviors within the events. Based on the found causal relationships, the causes of changes in system operation can be identified by determining parameters associated with events in event bursts related to changes in system operation. The impact of events and the correlation between events are not considered, and there is no room for feature aggregation. Summary of the Invention

[0009] The purpose of this disclosure is to provide methods, apparatus, and computer-readable media that at least partially address one or more of the challenges described above. Specifically, the purpose of this disclosure is to provide root cause analysis methods and apparatus capable of operating on complex datasets, requiring minimal human input, and taking into account causal relationships between data.

[0010] According to one aspect of an embodiment, a method for root cause analysis is provided. The method includes: acquiring measurement data, the measurement data including measurement data of features of a system; generating predicted values ​​by applying a trained ML model to the measurement data. The method further includes: generating feature influence values ​​by applying the generated ML model interpreter to the measurement data. The method further includes: updating an ontology representation of the connection between the system's features and the predicted values ​​using the generated feature influence values; and outputting a proposed root cause based on the updated ontology representation, wherein the proposed root cause leads to the predicted value. Combining the model interpreter and ontology representation using a trained ML model allows for the identification of root causes of predicted values ​​with minimal human input. These root causes can then be used to identify and address potential system problems and improve system performance.

[0011] In some aspects of the embodiments, the ontology representation can be a knowledge graph. A knowledge graph can have a static structure and can represent causal relationships between domains, predicted values, and measured data of features. Knowledge graphs are particularly well-suited for representing causal relationships in complex systems, and using a static structure for a knowledge graph ensures that the expert system knowledge used in preparing the knowledge graph can be preserved. Therefore, a knowledge graph can accurately represent the system.

[0012] In some aspects of the embodiments, the method may include training an ML model and / or generating an ML model interpreter, which can potentially be performed in parallel. This allows for the creation of the ML model and / or model interpreter to be as efficient as possible.

[0013] In some aspects of the embodiments, the system may be at least part of a telecommunications network, the predicted value may be a KPI value (e.g., VoLTE MOS), and the measurement data of the characteristics may be telecommunications network metrics. These aspects of the embodiments may be particularly well-suited for providing root cause information for complex systems such as telecommunications networks.

[0014] In some aspects of the embodiments, the method may further include: suggesting actions to address the identified root cause; and potentially performing the actions on the system. In this way, potential problems in the system can be quickly identified and resolved with minimal human input required.

[0015] According to other aspects of the embodiments, a root cause analyzer for root cause analysis is provided. The root cause analyzer includes processing circuitry and a memory containing instructions executable by the processing circuitry. The root cause analyzer is operable to perform a method comprising: acquiring measurement data, the measurement data including measurement data of features of a system; and generating a predicted value by applying a trained machine learning (ML) model to the measurement data. The root cause analyzer is further configured to generate feature influence values ​​by applying a generated ML model interpreter to the measurement data. The root cause analyzer is further configured to: update an ontology representation of the connection between the system's features and the predicted value using the generated feature influence values; and output a proposed root cause based on the updated ontology representation, wherein the proposed root cause leads to the predicted value. Some advantages provided by this root cause analyzer can be discussed above in the context of the method for root cause analysis. Attached Figure Description

[0016] This disclosure is described by way of example only, with reference to the following figures, in which:

[0017] Figure 1 This is a schematic overview of the CEM architecture of a telecommunications network;

[0018] Figure 2 This is a flowchart of various aspects of the methods according to the embodiments;

[0019] Figure 3A and Figure 3B This is a schematic diagram of the system according to various aspects of the embodiments;

[0020] Figure 4 This is a schematic overview of the training / generation of ML models and ML model interpreters according to various aspects of the embodiments;

[0021] Figure 5 It is an illustrative diagram showing how to obtain the predicted values;

[0022] Figure 6 This is an example representation of a knowledge graph according to one aspect of an embodiment;

[0023] Figure 7A and Figure 7B It is a schematic representation of root cause identification;

[0024] Figure 8A and Figure 8B This is an example representation of a knowledge graph based on various aspects of the embodiments;

[0025] Figure 9 Examples are obtained using methods according to various aspects of the embodiments; and

[0026] Figure 10 This is a schematic overview of the process by which one or more root causes of obtaining predicted values ​​can be obtained according to various aspects of the embodiments. Detailed Implementation

[0027] For purposes of explanation, details are set forth in the following description in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that these embodiments may be implemented without these specific details or with equivalent arrangements.

[0028] As mentioned above, ML technology can assist operators in performing RCA. However, ML systems typically operate as "black box" systems, where the manner in which an ML agent generates a specific output given a specific input is unknown. Consequently, a typical ML system can provide some assistance to operators performing RCA, but the scope of that assistance is limited. If an ML agent is to be used to predict KPI values, an ML model interpreter can be used to identify which features contribute to the predicted KPI values ​​and the magnitude of the contribution from each feature (information that may be useful for RCA).

[0029] ML model interpreters are used to identify why an ML model returns a given output when given a given input. Examples of ML model interpreters include the Eli5 package (for a more detailed discussion, see: https: / / eli5.readthedocs.io / en / latest / overview.html, since September 11, 2020) and the LIME (Local Interpretable Model-agnostic Explanations) method (for a more detailed discussion, see "Why Should I Trust You?" by Ribeiro, MT, Singh, S., and Guestrin, C.; ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016; available at: https: / / arxiv.org / abs / 1602.04938, since September 29, 2020). The additive ML model interpreter known as SHAP (SHapley Additive ex Planations) is discussed in more detail in “A Unified Approach to Interpreting Model Predictions” by Lundberg, SM, and Lee, SI (NIPS conference, 2017; available at: https: / / papers.nips.cc / paper / 7062-a-unified-approach-to-interpreting-model-predictions, effective September 11, 2020).

[0030] Using an ML model interpreter, the most relevant features that contribute to the predicted KPI values ​​can be identified. While the use of ML model interpreters allows for RCA in some simple systems, in complex systems such as telecommunications networks, the features are often highly interrelated. A particular problem or KPI degradation may be indicated by multiple features, making it difficult to find the root cause of the problem / degradation because causal relationships may not be identified. Detailed system knowledge may be required to determine causal relationships.

[0031] Embodiments of this disclosure provide methods and apparatus for at least partially automating RCA by using an ML model in conjunction with an ontology representation and an ML model interpreter. The ontology representation can be used to represent a system-related knowledge base, thereby indicating causal relationships between features and higher-level values ​​(derived from features). Typically, the ontology representation is initially compiled with the assistance of one or more system experts and is a representation of the knowledge base related to the given system by those experts. An example of an ontology representation that can be used is a knowledge graph.

[0032] A knowledge graph is essentially a graphical representation of multiple statements (that form a knowledge base). In a knowledge graph, it is typically represented as a set of entities and predicates in the form of a multidimensional graph. The relationships between entities (predicates) can be illustrated using links between them. In some knowledge graphs, the relative positioning of entities and links on the graph can be used to illustrate the relationships between different entities and links.

[0033] According to various aspects of the embodiments, ML models are used to extract low-level features (e.g., such as...) Figure 1 The network metric shown generates predicted values ​​(e.g., predicted KPI values) that indicate performance provided to end users. An ML model interpreter is used to generate influence values ​​for each feature of each individual sample. An ML model interpreter as described above can be used, preferably an additive ML model interpreter such as SHAP. The generated feature influence values ​​are then used to update the ontology representation (e.g., a knowledge graph) encoding the causal relationships between features and higher-level domains. As described above, the structure of the ontology representation is predefined by one or more human experts using an expert knowledge base; this structure can be fixed. Based on the updated ontology representation, the proposed root causes for the predicted values ​​can then be identified and output.

[0034] When the ontology representation is a knowledge graph, the generated feature influence values ​​can be assigned to low-level nodes of the knowledge graph (which may be referred to as leaf nodes, especially when the knowledge graph has a tree structure). The influence of a given higher-level domain of the knowledge graph can then be determined by iteratively summing the influence values ​​of the low-level nodes connected to that given higher-level domain; this process is simplified if an additive ML model interpreter is used. Once this process is implemented across higher-level domains, one or more root causes of the predicted values ​​can be identified based on the relative influence values ​​of these domains. The operation of some aspects of the embodiments is discussed in more detail below.

[0035] According to various aspects of the embodiments, RCA can be performed with reduced required operator input. These aspects enable operation using complex datasets, such as datasets derived from telecommunications networks, vehicle traffic management systems, web services, cloud data services, etc. The use of ontology representations allows for the investigation of causal relationships and also facilitates the incorporation of system expert knowledge.

[0036] exist Figure 2 The flowchart illustrates aspects of the method according to an embodiment. This method can be performed by any suitable apparatus. Figure 3A and Figure 3B Examples of suitable apparatuses for performing the method according to various aspects of the embodiments are schematically shown. Figure 3A and Figure 3B One or more devices shown may be incorporated into a system (e.g., the system is all or part of a telecommunications network), and one or more devices for performing the method may be incorporated into a base station, core network node, or other central management entity.

[0037] like Figure 2 As shown in step S202, the method includes acquiring measurement data, which includes measurement data of system characteristics. As mentioned above, characteristic measurement data can be acquired from multiple sources; using the example of a telecommunications network, the characteristic data can be telecommunications network metrics acquired from multiple domains. When using according to... Figure 3A In the case of the root cause analyzer 301 of the illustrated embodiment, feature measurement data can be obtained according to a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Alternatively, when using the root cause analyzer 301 according to the embodiment, feature measurement data can be obtained according to a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Figure 3B In the case of the root cause analyzer 351 of the illustrated embodiment, feature measurement data can be acquired by the acquirer 361. Once the measurement data is received, it can be stored in a database; Figure 3A The memory 315, or a memory incorporated into or accessible by the acquirer 361, can be used for this purpose. Measurement data can be processed by another component (e.g., a component forming part of or separate from the root cause analyzers 301, 351) and received in one or more batches. Alternatively, measurement data can be received directly after measurement without processing.

[0038] like Figure 2As shown in step S204, the received measurement data is then passed to a trained machine learning (ML) model to generate predicted values. Any suitable ML model can be used, such as neural networks, boosting decision trees, etc. The choice of ML model can be determined by the selection of the ML model interpreter (discussed in more detail below) and / or by its suitability for modeling the expected system. In some aspects of the embodiments, the ML model can be trained before or simultaneously with the acquisition of the measurement data. The ML model can be trained using training data that is expected to be similar to the acquired measurement data; optionally, the training data can be previously acquired feature measurement data. Any suitable training method can be used; suitable ML training methods are well known to those skilled in the art.

[0039] Using according to Figure 3A In the case of the root cause analyzer 301 of the illustrated embodiment, predicted values ​​can be generated based on a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Alternatively, when using... Figure 3B In the case of the root cause analyzer 351 in the illustrated embodiment, predictions can be generated by the data analyzer 363. The ML model can be part of the root cause analyzers 301 and 351, for example, it can be part of the data analyzer 363. Alternatively, the ML model can be accessible by the root cause analyzers 301 and 351, but not part of them.

[0040] The trained ML model generates predicted values, which can be numerical values ​​indicating system properties. Using an example implementation of a telecommunications network, the predicted values ​​can be KPIs, such as packet loss rate, latency, or Long Term Evolution (LTE) Voice Mean Opinion Score (VoLTE MOS). All the example KPIs listed above use numerical ranges; for example, packet loss rate uses a range bounded by 0 and 1, while VoLTE MOS uses a range bounded by 1 and 5. If the predicted value relates to a measurement that is not typically measured using numerical ranges, that measurement can be converted to use a numerical range.

[0041] In addition to applying the trained ML model to the measurement data to generate predicted values, the method also includes applying the generated ML model interpreter to the measurement data to generate feature influence values, as shown in step S206. Feature influence values ​​are estimates of the degree of influence of each feature of its input measurement data on the predicted values ​​generated by the ML model. Figure 3AIn the case of the root cause analyzer 301 of the illustrated embodiment, characteristic influence values ​​can be generated according to a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Alternatively, when using the root cause analyzer 301 according to the embodiment, characteristic influence values ​​can be generated according to a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Figure 3B In the case of the root cause analyzer 351 of the illustrated embodiment, the feature impact value can be generated by the feature impact value generator 365. The ML model analyzer can form part of the root cause analyzers 301 and 351, for example, it can form part of the feature impact value generator 365. Alternatively, the ML model analyzer can be accessible by the root cause analyzers 301 and 351, but not form part of the root cause analyzers 301 and 351.

[0042] Feature impact values ​​are numerical values; generally, a feature impact value can be positive if it has a beneficial effect on the predicted value, and negative if it has a detrimental effect. Taking a telecommunications network where the predicted value is the packet loss rate (KPI) as an example, the network metric of the serving cell's reference signal received power (RSRP) can be positive above a certain average performance value because a high serving cell RSRP will reduce packet loss, or negative below a certain average performance value because a low serving cell RSRP will increase packet loss. The components constituting a positive and negative metric depend on the specific system configuration and can be determined using system expert knowledge.

[0043] In some aspects of the embodiments, the ML model interpreter may be generated by the root cause analyzer prior to its use in generating feature influence values. The ML model interpreter can be generated using training data, which may be the same training data used to train the ML model (as described above). The training of the ML model can also be a factor in the generation of the ML model interpreter. Figure 4 The diagram provides a schematic overview of the training / generation of ML models and ML model interpreters, as well as their subsequent use. Figure 4 As shown, the same training data can be used for both ML model training and ML model interpreter generation, or different data can be used. This process produces a trained ML model and a generated ML model interpreter. ML model training and ML model interpreter generation can be performed in parallel (using different hardware) or sequentially using the same hardware. Typically, ML model training begins at least before ML model interpreter generation, as the ML model is a factor in ML model interpreter generation. The trained ML model and the generated ML model interpreter can then be applied to feature measurement data to obtain predictions and feature influence values, respectively.

[0044] Figure 5 This is an illustrative graph indicating how predicted values ​​(KPI values ​​in this example) can be generated from the corresponding influence values ​​of multiple features. Figure 5 In the example shown, the predicted KPI value is lower than the average (reference) KPI value. Features 1 and 3 have the largest negative impact values, and therefore contribute the most to the KPI downgrade. In contrast, features 2 and 5 have small positive impacts. Figure 5 The feature influence values ​​shown are generated by the ML model interpreter. Additive ML model interpreters (e.g., SHAP) are particularly suitable because multiple features can be grouped together, and the feature influence values ​​of composite features are determined by simply summing the feature influence values ​​of the individual features. In the case of using a non-additive ML model interpreter, the process for obtaining the feature influence values ​​of composite features can be complex.

[0045] The method also includes updating the ontology representation of the connection between features and predicted values ​​using the generated feature influence values, such as... Figure 2 As shown in S208. When using according to Figure 3A In the case of the root cause analyzer 301 of the illustrated embodiment, the ontology representation can be updated according to a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Alternatively, when using the root cause analyzer 301 according to the embodiment, the ontology representation can be updated according to a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Figure 3B In the case of the root cause analyzer 351 in the illustrated embodiment, the ontology representation can be updated by the ontology representation controller 367. The ontology representation can be part of the root cause analyzers 301 and 351, for example, it can be part of the ontology representation controller 367. Alternatively, the ontology representation can be accessible by the root cause analyzers 301 and 351, but not part of the root cause analyzers 301 and 351.

[0046] Some aspects of the embodiments also include generating an ontology representation, for example, using a knowledge base of expert knowledge related to a given system. In some aspects of the embodiments, the ontology representation may be a knowledge graph; knowledge graphs may be particularly well-suited for representing knowledge related to certain systems (e.g., telecommunications systems). When the ontology representation is a knowledge graph, the structure of the knowledge graph may be static, such that the nodes and edges (links between nodes) are fixed, but the corresponding weights of the nodes and edges can vary. Static structures may be particularly well-suited for representing systems where edges represent defined causal relationships between nodes. Alternatively, knowledge graphs with dynamic structures may be more suitable for some systems, where edges can be created and deleted based on data.

[0047] Figure 6 An example of a knowledge graph according to an embodiment is shown. Figure 6The example illustrates a knowledge graph structure related to end-user sessions in a telecommunications network. Causal relationships between KPIs and network parameters are established. In addition to KPIs and low-level features, there are mid-level domains representing different network domains and subdomains. Edges illustrate dependencies; for example, a KPI depends on a subdomain with arrows pointing to the KPI, and these subdomains may be the cause of the KPI issue. These subdomains can then further depend on other lower-level subdomains, and so on. The lowest level (leaves) of the graph contains the features for which measurement data is acquired (in this case, telecommunications network metrics). Figure 6 The knowledge graph illustrates a system in which each feature (for which measurement data is acquired) directly affects only a single domain; indicated by a single arrow originating from each lowest-level feature. In other aspects of the embodiments, a single feature may directly affect multiple domains, and this will be graphically indicated as multiple arrows originating from a single feature and terminating in different domains.

[0048] Once the ontology representation is updated with the feature values ​​generated by the ML model interpreter, it can be used to identify proposed root causes of predictions that are worse than expected. The proposed root causes can then be output, as shown in step S210. In cases where multiple root causes collectively lead to the prediction, these multiple root causes can be output. (This is based on...) Figure 3A In the case of the root cause analyzer 301 of the illustrated embodiment, the root cause can be output based on a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Alternatively, when using the root cause analyzer 301 according to the embodiment, the root cause can be output based on a computer program stored in memory 315, which is executed by processor 311 in conjunction with one or more interfaces 313. Figure 3B In the case of the root cause analyzer 351 of the embodiment shown, the root cause can be output by the output unit 369.

[0049] Using a telecommunications network as an example, when the expected value is 4, the ML model can predict a VoLTE MOS value (a KPI typically used in telecommunications networks) of 2. The ontology representation can indicate that the root cause of this worse-than-expected VoLTE MOS value is worse than the usual SINR value (e.g., it might be caused by atmospheric interference). In some aspects of the embodiments, the method may also include suggesting actions to address the proposed root cause(s) and outputting the suggested actions. Using the example of a worse-than-usual SINR value, a proposal to increase signal transmission power can be made to help improve SINR. This proposal can be obtained from a database of potential root causes associated with solutions, which can be accessed from or as part of a root cause analyzer. In some aspects of the embodiments, the method may also include implementing the suggested actions, i.e., performing the actions on the system. Referring to the example above, the root cause analyzer can trigger a transmission signal indicating an increase in transmission power. The root cause analyzer that performs the action can reduce the latency in the action being performed and is particularly suitable for situations where the root cause analyzer is part of the system (e.g., in the case where the system is a telecommunications network and the root cause analyzers 301 and 351 are integrated into the core network node).

[0050] The output of one or more root causes can be a single feature or multiple features, and / or a single domain or multiple domains. Figure 7A This is an illustrative representation of a situation where a single feature is the root cause of the problem (indicated by the assertion of a low KPI). Figure 7B This is a schematic representation of a situation where a single domain is the root cause of the problem. In Figure 7, the relative sizes of the boxes represent the influence values ​​(for simplicity, all features and domains in Figure 7 have negative influence values). Figure 7A In this model, it's clear that feature X has a larger influence value than any other lower-level feature. This large influence value propagates upwards along the knowledge graph levels; via causal links to the mid-level domains and finally to the top of the graph (where the target variable KPI is located). In this case, feature X will be output as the root cause. In contrast, in... Figure 7B In this context, domain Y has a larger influence value than other domains of the same level. The large influence value of domain Y is not due to any single low-level feature, but rather the result of contributions from multiple low-level features (such as...). Figure 7B (As shown). In this case, domain Y will be output as the root cause.

[0051] Figure 8A This illustrates one aspect of a knowledge graph that can be generated in an embodiment where the system is a telecommunications network and the predicted value is a VoLTE MOS value. (Reference) Figure 6 The diagram in the middle, Figure 8AThe low-level features in a knowledge graph are network metrics. Figure 8A In this context, low-level features are labeled f1 to f11; examples of network metrics that can be represented by each of f1 to f11 are shown in Table 1. Other network metrics may also be suitable. Examples of some network metrics that can be used in various aspects of the embodiments are discussed in 3GPP, TS 36.214v 9.1.0, “Evolved Universal Terrestrial Radio Access (E-UTRA); Physical layer-Measurements” (available at: https: / / www.etsi.org / deliver / etsi_ts / 136200_136299 / 136214 / 09.01.00_60 / ts_136214v090100p.pdf, effective from September 29, 2020).

[0052] Table 1

[0053] Low-level features Example network metrics f1 PRB (Physical Resource Block) utilization rate f2 SINR (Signal-to-Noise Ratio) f3 RSRP (Reference Signal Received Power) f4 RSRQ (Reference Signal Received Quality) f5 RSSI (Received Signal Strength Index) f6 Neighbor RSRP f7 Modify the number of bearer requests f8 Number of unsuccessful modification events f9 Number of unsuccessful UE context release events f10 Number of S1 interface failure events f11 RRC reconfigures the number of timeout events

[0054] The KPI is VoLTEMOS (Voice Average Opinion Score), which directly depends on the performance of the upstream RAN, downstream RAN, core network, and IP multimedia subsystem. Figure 8A Only low-level features and mid-level domains that are causally linked to the high-level domain "Upstream RAN" are shown in full. For clarity, low-level features and mid-level domains that are causally linked to the domains "Downstream RAN", "Core" (Network), and "IMS" (IP Multimedia Subsystem) are not shown.

[0055] exist Figure 8A In this model, each low-level feature is causally linked to a single mid-level domain (via an edge), each domain is causally linked to a single higher-level domain, and so on. Figure 8B The alternative knowledge graphs are shown in the figure. Figure 8BThe knowledge graph includes causal links from some features to multiple domains (e.g., see feature f6), from a domain at one level to multiple higher-level domains (e.g., see “Coverage”), and between domains at a single level (see “Channel Quality”). To illustrate the different effects of features / domains on the domains they are causally linked to, the edges forming the causal links can be weighted accordingly. As an example of this, if feature f6 has a smaller impact on the value of “Coverage” but a larger impact on the value of “Handling”, then the edge from f6 to “Coverage” can be assigned a smaller weight than the edge to “Handling”. The weights can be normalized to ensure consistency in the calculation, such that the weight of all edges from a single feature is equal to 1. As an example of this, if both edges starting from cell load are given a weight of 0.5, then the weight of the upstream RAN is calculated as 0.5W(cell load) + W(channel quality) + W(handling) + W(drop).

[0056] Figure 9 These are examples of results obtained using methods according to various aspects of the embodiments. Figure 9 In the example, the predicted value of the VoLTE MOS KPI for the telecommunications network is obtained; the predicted value is 1.38 lower than the expected value (e.g., Figure 9 The value of "Total Degradation" (shown as "-1.38") indicates a KPI degradation. In this example, for clarity, only the contributions from the upstream and downstream RAN domains to the KPI value are shown. The numbers in the boxes representing the domains indicate the corresponding domain impact value. Positive impact values ​​(e.g., 0.01 for switching in the downstream RAN domain) indicate better-than-average performance, while negative impact values ​​(e.g., -1.06 for channel quality in the upstream RAN) indicate worse-than-average performance. Fields without numbers (e.g., "Dropped" in the upstream RAN) provide average performance; the valid value for each of these fields is 0. Figure 9 The individual's characteristic values ​​are not shown, and none of the individual's characteristics have a particularly high or low influence value. Figure 9 The names of the features shown are specific to the system involved in the graph; the exact nature of each individual feature in the graph is irrelevant. An examination of higher-level domains below the KPI value clearly shows that the upstream RAN contributes significantly more to KPI degradation than the downstream RAN. Analysis of the knowledge graph indicates that the main contributor to KPI degradation is the SINR value of -0.78; therefore, root cause analysis will output this SINR as the root cause of the KPI degradation.

[0057] Figure 10 The diagram illustrates a schematic overview of the process for obtaining one or more root causes of predicted values ​​according to various aspects of an embodiment. Figure 10As shown, the (already acquired) feature measurement data is processed by a trained ML model and an interpreter generated from it. Additionally, an ontology representation is used to represent a knowledge base (potentially generated from expert knowledge from one or more experts). The ML model generates predictions based on the data, and the interpreter generates feature effects. The ontology representation is then updated using the feature effects, and one or more root causes of the predicted values ​​are derived using the updated ontology representation. These root causes are then output. Once expert knowledge is used in the generation of the ontology representation, the process can be performed with minimal human input. Therefore, this root cause analysis method is less labor-intensive than previous methods. Furthermore, the disclosed method and / or apparatus can be used to process highly complex datasets and can also incorporate causal relationships (beyond observed correlations) into the analysis. Therefore, the disclosed method and / or apparatus can provide more accurate analysis than existing methods. As a result of the accurate root cause analysis provided, problems (e.g., KPI degradation) can be rapidly prevented or reversed, and the overall operation of the system affected by the root cause analysis can be improved.

[0058] It should be understood that the examples disclosed herein can be virtualized so that the methods and processes described herein can run in a cloud environment.

[0059] The methods disclosed herein can be implemented in hardware or as software modules running on one or more processors. These methods can also be executed according to instructions of a computer program, and this disclosure also provides a computer-readable medium having a program stored thereon for performing any of the methods described herein. The computer program embodying this disclosure can be stored on a computer-readable medium, or it can be in the form of a signal, such as a downloadable data signal provided from an Internet website, or it can be any other form.

[0060] Generally, various exemplary embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device, although this disclosure is not limited thereto. While aspects of exemplary embodiments of this disclosure may be shown and described as block diagrams, flowcharts, or other graphical representations, it should be clearly understood that such blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or combinations thereof (as non-limiting examples).

[0061] Therefore, it should be understood that at least some aspects of the exemplary embodiments of this disclosure can be practiced in various components, such as integrated circuit chips and modules. Therefore, it should be understood that the exemplary embodiments of this disclosure can be implemented in devices embodied as integrated circuits, wherein the integrated circuits may include circuitry (and possibly firmware) embodying at least one or more of a data processor, digital signal processor, baseband circuitry, and radio frequency circuitry that can be configured to operate according to the exemplary embodiments of this disclosure.

[0062] It should be understood that at least some aspects of the exemplary embodiments of this disclosure may be embodied in computer-executable instructions, which are executed by one or more computers or other devices, for example, in one or more program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., which perform a specific task or implement a specific abstract data type when executed by a processor or other device in a computer. The computer-executable instructions may be stored on a computer-readable medium such as a hard disk, optical disk, removable storage medium, solid-state memory, RAM, etc. As those skilled in the art will understand, the functionality of a program module may be combined or distributed as needed in various embodiments. Furthermore, this functionality may be embodied, wholly or partially, in firmware or hardware equivalents, such as integrated circuits, field-programmable gate arrays (FPGAs), etc.

[0063] References to "an embodiment," "embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, it should be understood that implementing such a feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art, whether explicitly described or not.

[0064] It should be understood that while the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed terms.

[0065] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein specify the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. The terms “connected,” “linked,” “connected to,” and / or “connected” as used herein cover a direct and / or indirect connection between two elements.

[0066] This disclosure includes any novel feature or combination of features explicitly disclosed herein or in any general sense. Various modifications and adaptations to the foregoing exemplary embodiments of this disclosure will become apparent to those skilled in the art when read in conjunction with the accompanying drawings, in light of the foregoing description. However, any and all modifications will still fall within the non-limiting and exemplary scope of this disclosure. For the avoidance of doubt, the scope of this disclosure is defined by the claims.

Claims

1. A method for root cause analysis, the method comprising: Acquire measurement data, including measurement data of system characteristics; Predicted values ​​are generated by applying a trained machine learning (ML) model to the measurement data, wherein the predicted values ​​are key performance indicator (KPI) values. Additive feature influence values ​​are generated by applying the generated additive ML model interpreter to the measurement data, wherein the additive feature influence values ​​represent the additive contribution of each feature to the predicted value; The generated feature influence values ​​are used to update the ontology representation of the connection between the system's features and the predicted values; and The proposed root cause is output based on the updated ontology representation, wherein the proposed root cause leads to the predicted value.

2. The method according to claim 1, further comprising: The ontology representation is generated using expert knowledge relevant to the system.

3. The method according to claim 1, wherein, The ontology representation is a knowledge graph.

4. The method according to claim 3, wherein, The knowledge graph has a static structure.

5. The method according to claim 3, wherein, The knowledge graph represents the causal relationships between the domain, predicted values, and feature measurement data.

6. The method according to claim 5, wherein, A given feature for which measurement data is acquired directly affects a single domain, and / or another given feature for which measurement data is acquired directly affects multiple domains.

7. The method according to any one of claims 1 to 6, further comprising: Train the ML model and / or generate the ML model interpreter.

8. The method according to claim 7, wherein, ML model training and ML model interpreter generation are performed in parallel.

9. The method according to any one of claims 1 to 6, wherein, The ML model is a neural network or a boosting decision tree.

10. The method according to any one of claims 1 to 6, wherein, The proposed root causes are output as follows: multiple root causes that collectively lead to the predicted value are output.

11. The method according to any one of claims 1 to 6, wherein, The system is at least part of a telecommunications network.

12. The method according to claim 11, wherein, The KPI mentioned is the Long Term Evolution LTE Voice Mean Opinion Score (VoLTE MOS).

13. The method according to claim 11, wherein, The measurement data for the aforementioned feature is a telecommunications network metric.

14. The method according to any one of claims 1 to 6, further comprising: Suggested actions to address the root causes mentioned above.

15. The method of claim 14, further comprising: The action is performed on the system.

16. A root cause analyzer for root cause analysis, the root cause analyzer comprising processing circuitry and a memory containing instructions executable by the processing circuitry, thereby enabling the root cause analyzer to operate to: Acquire measurement data, including measurement data of system characteristics; The trained machine learning (ML) model is applied to the measurement data to generate predicted values, wherein, The predicted value is the key performance indicator (KPI) value. The generated additive ML model interpreter is applied to the measurement data and the trained ML model to generate additive feature influence values, wherein the additive feature influence values ​​represent the additive contribution of each feature to the predicted value; The generated feature influence values ​​are used to update the ontology representation of the connection between the features and the predicted values; and The proposed root cause is output based on the updated ontology representation, wherein the proposed root cause leads to the predicted value.

17. The root cause analyzer of claim 16 is further configured to generate the ontology representation using expert knowledge relevant to the system.

18. The root cause analyzer according to claim 16, wherein, The ontology representation is a knowledge graph.

19. The root cause analyzer according to claim 18, wherein, The knowledge graph has a static structure.

20. The root cause analyzer according to claim 18, wherein, The knowledge graph represents the causal relationships between the domain, predicted values, and feature measurement data.

21. The root cause analyzer according to claim 20, wherein, A given feature for which measurement data is acquired directly affects a single domain, and / or another given feature for which measurement data is acquired directly affects multiple domains.

22. The root cause analyzer according to any one of claims 16 to 21, further configured to: train the ML model and / or generate the ML model interpreter.

23. The root cause analyzer of claim 22 is further configured to: train the ML model and generate the ML model interpreter in parallel.

24. The root cause analyzer according to any one of claims 16 to 21, wherein, The ML model is a neural network or a boosting decision tree.

25. The root cause analyzer according to any one of claims 16 to 21, further configured to output a plurality of root causes that commonly lead to the predicted value.

26. The root cause analyzer according to any one of claims 16 to 21, wherein, The system is at least part of a telecommunications network.

27. The root cause analyzer according to claim 26, wherein, The KPI mentioned is the Long Term Evolution LTE Voice Mean Opinion Score (VoLTE MOS).

28. The root cause analyzer according to claim 26, wherein, The measurement data for the aforementioned feature is a telecommunications network metric.

29. The root cause analyzer according to claim 26, wherein, The root cause analyzer is part of the telecommunications network.

30. The root cause analyzer (301) according to claim 29, wherein, The root cause analyzer (301) is part of a network node in the telecommunications network.

31. The root cause analyzer according to any one of claims 16 to 21, further configured to: suggest actions for resolving the proposed root cause.

32. The root cause analyzer of claim 31 is further configured to: initiate the execution of the action on the system.

33. A root cause analyzer for root cause analysis, the root cause analyzer comprising: The acquirer is configured to acquire measurement data, which includes measurement data of system characteristics; A data analyzer is configured to apply a trained machine learning (ML) model to the measurement data to generate predicted values, wherein the predicted values ​​are key performance indicator (KPI) values. A feature influence value generator is configured to apply the generated additive ML model interpreter to the measurement data and generate additive feature influence values, wherein the additive feature influence values ​​represent the additive contribution of each feature to the predicted value; An ontology representation controller is configured to update the ontology representation of the connection between the features and the predicted values ​​using the generated feature influence values; and The output is configured to output the proposed root cause based on the updated ontology representation, wherein the proposed root cause leads to the predicted value.

34. A computer-readable medium comprising instructions that, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 15.

Images