System and method for data labeling using knowledge graph in the cloud

Abstract

A method, system and apparatus are disclosed. A method implemented in a management node configured with a knowledge graph including a plurality of links for a plurality of nodes in a communication network is provided. The method includes detecting anomalous metric data associated with at least one of the plurality of nodes of the knowledge graph based on a first historical dataset including metric data representative of a normal state of the plurality of nodes of the knowledge graph, determining at least one anomalous node of the plurality of nodes based on the anomalous metric data, and determining at least one label for the anomalous metric data based on at least one link from the knowledge graph associated with the at least one anomalous node.

Description

[0001] SYSTEM AND METHOD FOR DATA LABELING USING KNOWLEDGE GRAPH IN THE CLOUD

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to wireless communications, and in particular, to performance metric data labeling using knowledge graphs in a wireless communications network.

[0004] BACKGROUND

[0005] The Third Generation Partnership Project (3GPP) has developed and is developing standards for Fourth Generation (4G) (also referred to as Long Term Evolution (LTE)) and Fifth Generation (5G) (also referred to as New Radio (NR)) wireless communication systems. Such systems provide, among other features, broadband communication between network nodes, such as base stations, and mobile wireless devices (WD), as well as communication between network nodes and between WDs. The 3GPP is also developing standards for Sixth Generation (6G) wireless communication networks.

[0006] Knowledge bases have been in use for decades in telecom systems. Knowledge bases store facts about (a subset of) the world, and these facts can for example be used to reason and deduce new facts or highlight inconsistencies. Initially, knowledge bases were mainly parts of expert systems. This technology is gaining momentum again as part of machine reasoning solutions. The “knowledge” of a knowledge base is typically stored in a graph form (e.g., data structure), and as a result, the term “knowledge graph” may be used to refer to a knowledge base, or some aspect thereof. Expert and machine reasoning systems may use knowledge graphs to infer new facts and answer questions or, for example, pinpoint root causes, identify undesirable system states, etc.

[0007] As knowledge may be complex in nature, knowledge graphs may quickly become complex, e.g., with thousands of nodes and edges. Storing such graphs may be storage / memory consuming, and traversing them may consume significant time and computational resources. For real-time applications, decision making on large knowledge graphs by such traversals may not be practicable or feasible. In edge cloud contexts and / or loT environments, there may be significant memory and / or computational limitations, and thus an entire knowledge base may not be available for decision making. As a result, knowledge graph traversal may be more expensive, especially when it is required to traverse the knowledge graph repeatedly to answer different questions (e.g., “Is the system in a faulty state?”, “Is there a security threat?”, etc.).

[0008] With the rise of data-centric technologies in telecom systems, Machine Learning (ML) techniques are increasingly used in various domains including fault, performance and security management. Such ML techniques may use the data collected from the system(s), train detection and / or prediction models, and then use the models for online detection and / or prediction. ML techniques have become an important stepping-stone in building automated and intelligent cloud systems. For instance, they allow applications running in a cloud system to become more accurate when identifying its outcomes, without necessarily being explicitly programmed to do so. Different ML algorithms have been proposed in the literature, including, e.g., supervised machine learning, which uses labeled datasets to train algorithms to classify data or predict outcomes accurately. For example, some existing supervised machine learning systems have been characterized as answering questions with various levels of accuracy and efficiency (e.g., similar to the questions answered by the knowledge graphs).

[0009] Labeling is a noteworthy function for configuring supervised machine learning as the quality of labels may have a direct impact on the performance of supervised learning. Data labeling is usually a manual and costly process and requires prior knowledge about the monitored system. Such manual solutions may not be able to generically identify all labels in large cloud systems, and may also not be able to guarantee a full coverage of the labels, especially for large datasets. In addition, it may not be possible for such manual solutions to identify rare events or unknown events when they occur. Traditional label learning methods require the labeler to be an expert and have prior knowledge about the monitored system to guarantee the correctness of the output labels. This results in expensive labeling costs, and requires substantial time and effort to provide such manual configuration and knowledge. In addition, it may not be generalizable for large data sets, especially for large systems.

[0010] Some existing systems have built knowledge graphs from extracted facts from large- scale systems. One of the challenges in this process is inferring labels consistently. Some existing systems have considered an approach based on determining co-reference entities in the knowledge graph to produce consistent sets of labels and relations for each node, and use classification methods to label nodes by taking into account ontological information and neighboring labels.

[0011] In some existing systems, a knowledge embedding approach is utilized to reason new relational facts from a large-scale knowledge graph and text corpus. For example, a triplet classification task may be performed, which may label a triplet (node, relation, node) as true or false.

[0012] In some existing systems, outgoing and incoming edges on a given directed graph are labeled with integers based on the premise that edges should be assigned. For example, edgelabeling schemes may seek to balance and / or maximize the consecutiveness at every individual vertex so that local decisions greedily make progress towards the global optimal.

[0013] In some existing systems, learning is performed over a labeled graph to measure regularity and similarity of graph nodes. The graph may be partitioned into a few well- separated clusters within which labels are roughly constant. The labels may be assigned based on similarities between the grouped nodes in each cluster.

[0014] In some existing systems, parsing of the labeled knowledge graph is performed. This may include an identification of triples, including a source node, a destination node, and a link between them, each of which may be labeled. Natural language model priming may be used to provide input functionality to independently developed application programs.

[0015] In some existing systems, corresponding matches between a plurality of input instance and nodes in a knowledge graph are identified. For the identified matching objects, the system may identify a common class object represented by a corresponding node of the knowledge graph. A percentage of identified matching objects may be descendants of the common class object, and the percentage may be greater than a predefined threshold. A label of the common class object may be associated with the plurality of input instances.

[0016] Existing systems, however, lack a configuration or capability for labeling data describing a system state. For example, existing systems for inferring labels of nodes, links, subset of nodes, triplets linking nodes and links, etc. in a knowledge graph may use classification methods for labeling those data (nodes, links, subset of nodes, etc.), but such existing labeling approaches are not automated and rely on significant manual input from human operators. Further, such existing systems may be computationally expensive and cannot generalize for large systems, and may lead to imbalanced, inconsistent, and / or invalid labels, which could negatively impact performance (e.g., training results) for an ML model.

[0017] Thus, existing systems may suffer from one or more defects or inefficiencies.

[0018] SUMMARY

[0019] Embodiments disclosed herein may provide an automatic method for performing labeling for ML data in cloud systems using the knowledge inferred from a knowledge graph.

[0020] Embodiments of the present disclosure may provide a system or method that can be used to automatically label input data by running inferences using an existing knowledge graph. For example, input data in an input dataset may be paired with the inference result(s) from the knowledge base. Labeling may be an extremely time and resource consuming process when done manually by a human, as in some existing systems. Embodiments of the present disclosure may utilize knowledge which exists in a knowledge base for labeling input data, and by training machine learning models in conjunction with knowledge graphs, may improve efficiency over existing systems, e.g., by only utilizing relevant parts (subsets) of the captured knowledge base.

[0021] In some embodiments, the labelled dataset may be used to train a supervised machine learning model, for example, a feed forward neural network. By this approach, the relevant part(s) of the “knowledge” (i.e., answers to certain limited kind of input) from the knowledge graph can be transferred to a machine learning model. The machine learning model can be then executed more efficiently in real-time or constrained environments, compared to running inferences on the complete knowledge base. This process may be further enhanced with graphical processing units (GPUs) or other special purpose hardware accelerators.

[0022] In some embodiments of the present disclosure, a subset of knowledge from a knowledge base may be transferred into a trained machine learning model, such as a supervised machine learning model. The transfer may be achieved via presenting example inputs / questions to the knowledge base that are labelled based on the results. The labelled data then can be used to train a machine learning model via regular training methods.

[0023] In one or more embodiments of the present disclosure, one or more entities or nodes of the wireless communication system may: train an Anomaly Detection (AD) model using data collected from the managed system (e.g., monitoring metrics, log data, traces, etc.); perform analysis of the collected data and assign each data to the nodes in the knowledge graph; in the presence of data to be labeled, identify anomalous nodes using the trained AD model and the description of the nodes; analyze the set of candidate anomalous nodes and determine the set of paths linking them using the knowledge graph description; score the obtained paths using different criteria including number of anomalous nodes, the rate of anomaly for each node, etc.; select the path with the highest score as the candidate path that describes the input data to be labeled; generate a label to the selected path using the knowledge graph and the collected data; and / or checks consistency and / or validity of the generated label.

[0024] One or more embodiments of the present disclosure may provide one or more of the following advantages over existing systems: enable automatic labeling of data using a knowledge graph, which is an improvement over existing systems which substantial manual (human) input for labeling; enable extracting a limited (i.e., smaller subset of) knowledge from a knowledge graph that can be used efficiently in real-time, limited resource, and / or distributed environments; enable efficient upgrading of distributed deployments by re-creating and comparing (a subset of) labels after the knowledge base update, reducing computational and network resources required for upgrades / updates; enable training a variety of different unlabeled datasets, which may be updated as needed and / or periodically; may check consistency and / or validity of generated labels in a variety of different system contexts; and / or may store the generated labels in a repository (e.g., a cloud-based server, cloud-based storage, network node storage, management node storage, database, etc.) to be used in the future to label other datasets, etc.

[0025] According to a first aspect of the present disclosure, a management node configured to communicate with a plurality of network nodes in a communication system is provided. The management node may be configured with a knowledge graph including a plurality of links for a plurality of nodes in the communication network. The management node is configured to detect anomalous metric data associated with at least one of the plurality of nodes (e.g., network nodes, WDs, etc.) of the knowledge graph based on a first historical dataset comprising metric data representative of a normal state of the plurality of nodes of the knowledge graph. The management node is configured to determine at least one anomalous node (e.g., at least one network node and / or at least one WD, etc.) of the plurality of nodes based on the anomalous metric data. Management node is configured to determine at least one label for the anomalous metric data based on at least one link from the knowledge graph associated with the at least one anomalous node.

[0026] According to one or more embodiments of this aspect, the management node is further configured to train a machine learning (ML) model using the anomalous metric data and the at least one label. According to one or more embodiments of this aspect, the management node is further configured to detect additional anomalous metric data associated with at least one of the plurality of nodes of the knowledge graph, and determine at least one additional label for the additional anomalous metric data based on the trained ML model. According to one or more embodiments of this aspect, the determining of the at least one label for the anomalous metric data further includes determining a plurality of candidate paths for traversing the knowledge graph, each of the plurality of candidate paths traversing at least one anomalous node of the plurality of nodes, and determining the at least one label based on metrics associated with at least one of the plurality of candidate paths.

[0027] According to one or more embodiments of this aspect, the determining of the at least one label for the anomalous metric data further includes determining a rank order of the plurality of candidate paths, and determining at least one first label based on at least one corresponding highest-ranked candidate paths of the plurality of candidate paths. According to one or more embodiments of this aspect, the determining of the rank order of the plurality of candidate paths includes determining a respective score for each candidate path of the plurality of candidate paths, where the respective score is determined based on at least one of a number of anomalous nodes of the respective candidate path, a rate of anomaly of the respective candidate path, a severity of anomaly of the respective candidate path, and / or a number of anomalous metrics of the respective candidate path.

[0028] According to one or more embodiments of this aspect, the at least one highest-ranked candidate path traverses a first set of nodes, and the determining of the at least one first label based on the at least one highest-ranked candidate path includes determining a start node of the first set of nodes based on a hierarchical relationship among the first set of nodes, determining a first plurality of anomalous metrics associated with a first number of nodes of the first set of nodes, where the first plurality of anomalous metrics includes a first plurality of metric names and a corresponding first plurality of metric values, determining a first keyword based on the first plurality of metric names, and determining the at least one first label based on the first keyword. According to one or more embodiments of this aspect, the management node is further configured to validate the at least one first label based on the first number of nodes associated with the first plurality of anomalous metrics exceeding a preconfigured threshold number of nodes.

[0029] According to one or more embodiments of this aspect, the management node is further configured to, responsive to a failure to validate the at least one first label, determine a second candidate path traversing a second set of nodes, determine a second plurality of anomalous metrics associated with a second number of nodes of the second set of nodes, the second plurality of anomalous metrics including a second plurality of metric names and a corresponding second plurality of metric values, determine a second keyword based on the second plurality of metric names, determine at least one second label based on the second keyword, and validate the at least one second label based on the second number of nodes associated with the second plurality of anomalous metrics exceeding the preconfigured threshold number of nodes.

[0030] According to one or more embodiments of this aspect, the second candidate path is one of a second-highest-ranked candidate path of the plurality of candidate paths traversing the second set of nodes, and a sub-path of the at least one highest-ranked candidate path, where the sub-path traverses the second set of nodes, the second set of nodes is a subset of the first set of nodes, and the second set of nodes includes at least one dependent node of the start node. According to one or more embodiments of this aspect, the management node is further configured to receive, prior to the step of detecting, the first historical dataset including at least one of monitoring metrics, traces, and log files associated with the plurality of nodes of the knowledge graph.

[0031] According to one or more embodiments of this aspect, the management node is further configured to receive the knowledge graph from at least one network node in the communication network, and receive the anomalous metric data from at least one network node in the communication network. According to one or more embodiments of this aspect, the management node may be preconfigured with the knowledge graph.

[0032] According to another aspect of the present disclosure, a method implemented in a management node configured to communicate with a plurality of network nodes in a communication system is provided. The management node may be configured with a knowledge graph including a plurality of links for a plurality of nodes in the communication network. The method includes detecting anomalous metric data associated with at least one of the plurality of nodes (e.g., network nodes, WDs, etc.) of the knowledge graph based on a first historical dataset comprising metric data representative of a normal state of the plurality of nodes of the knowledge graph. The method includes determining at least one anomalous node (e.g., at least one network node and / or at least one WD, etc.) of the plurality of nodes based on the anomalous metric data. The method includes determining at least one label for the anomalous metric data based on at least one link from the knowledge graph associated with the at least one anomalous node.

[0033] According to one or more embodiments of this aspect, the method further includes training a machine learning (ML) model using the anomalous metric data and the at least one label. According to one or more embodiments of this aspect, the method further includes detecting additional anomalous metric data associated with at least one of the plurality of nodes of the knowledge graph, and determining at least one additional label for the additional anomalous metric data based on the trained ML model. According to one or more embodiments of this aspect, the determining of the at least one label for the anomalous metric data further includes determining a plurality of candidate paths for traversing the knowledge graph, each of the plurality of candidate paths traversing at least one anomalous node of the plurality of nodes, and determining the at least one label based on metrics associated with at least one of the plurality of candidate paths.

[0034] According to one or more embodiments of this aspect, the determining of the at least one label for the anomalous metric data further includes determining a rank order of the plurality of candidate paths, and determining at least one first label based on at least one corresponding highest-ranked candidate paths of the plurality of candidate paths. According to one or more embodiments of this aspect, the determining of the rank order of the plurality of candidate paths includes determining a respective score for each candidate path of the plurality of candidate paths, where the respective score is determined based on at least one of a number of anomalous nodes of the respective candidate path, a rate of anomaly of the respective candidate path, a severity of anomaly of the respective candidate path, and / or a number of anomalous metrics of the respective candidate path.

[0035] According to one or more embodiments of this aspect, the at least one highest-ranked candidate path traverses a first set of nodes, and the determining of the at least one first label based on the at least one highest-ranked candidate path includes determining a start node of the first set of nodes based on a hierarchical relationship among the first set of nodes, determining a first plurality of anomalous metrics associated with a first number of nodes of the first set of nodes, where the first plurality of anomalous metrics includes a first plurality of metric names and a corresponding first plurality of metric values, determining a first keyword based on the first plurality of metric names, and determining the at least one first label based on the first keyword. According to one or more embodiments of this aspect, the method further includes validating the at least one first label based on the first number of nodes associated with the first plurality of anomalous metrics exceeding a preconfigured threshold number of nodes.

[0036] According to one or more embodiments of this aspect, the method further includes, responsive to a failure to validate the at least one first label, determining a second candidate path traversing a second set of nodes, determining a second plurality of anomalous metrics associated with a second number of nodes of the second set of nodes, the second plurality of anomalous metrics including a second plurality of metric names and a corresponding second plurality of metric values, determining a second keyword based on the second plurality of metric names, determining at least one second label based on the second keyword, and validating the at least one second label based on the second number of nodes associated with the second plurality of anomalous metrics exceeding the preconfigured threshold number of nodes.

[0037] According to one or more embodiments of this aspect, the second candidate path is one of a second-highest-ranked candidate path of the plurality of candidate paths traversing the second set of nodes, and a sub-path of the at least one highest-ranked candidate path, where the sub-path traverses the second set of nodes, the second set of nodes is a subset of the first set of nodes, and the second set of nodes includes at least one dependent node of the start node. According to one or more embodiments of this aspect, the method further includes receiving, prior to the step of detecting, the first historical dataset including at least one of monitoring metrics, traces, and log files associated with the plurality of nodes of the knowledge graph.

[0038] According to one or more embodiments of this aspect, the method further includes receiving the knowledge graph from at least one network node in the communication network, and receiving the anomalous metric data from at least one network node in the communication network. According to one or more embodiments of this aspect, the method further includes preconfiguring the management node with the knowledge graph.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] A more complete understanding of the present embodiments, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:

[0041] FIG. 1 is a schematic diagram of an example network architecture illustrating a communication system according to principles disclosed herein;

[0042] FIG. 2 is a block diagram of a network node in communication with a wireless device over a wireless connection according to some embodiments of the present disclosure;

[0043] FIG. 3 is a flowchart of an example process in a management node for data labeling using a knowledge graph according to some embodiments of the present disclosure;

[0044] FIG. 4 is a flowchart of an example process for data labeling using a knowledge graph according to some embodiments of the present disclosure; FIG. 5 is a flowchart of another example process for data labeling using a knowledge graph according to some embodiments of the present disclosure;

[0045] FIG. 6 is a flowchart of another example process for data labeling using a knowledge graph according to some embodiments of the present disclosure;

[0046] FIG. 7 is a flowchart of an example process for data labeling using a knowledge graph according to some embodiments of the present disclosure;

[0047] FIG. 8 is an illustration of a knowledge graph for which data is labeled according to some embodiments of the present disclosure; and

[0048] FIG. 9 is an illustration of an example traversal and data labeling of a path of the knowledge graph of FIG. 8 according to some embodiments of the present disclosure.

[0049] DETAILED DESCRIPTION

[0050] Before describing in detail exemplary embodiments, it is noted that the embodiments reside primarily in combinations of apparatus components and processing steps related to data labeling of monitoring metrics and / or performance metrics based on knowledge graphs, e.g., for training machine learning models for error and anomaly detection in a wired and / or wireless communication network or other distributed computing system. Accordingly, components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0051] As used herein, relational terms, such as “first” and “second,” “top” and “bottom,” and the like, may be used solely to distinguish one entity or element from another entity or element without necessarily requiring or implying any physical or logical relationship or order between such entities or elements. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and / or “including” when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0052] In embodiments described herein, the joining term, “in communication with” and the like, may be used to indicate electrical or data communication, which may be accomplished by physical contact, induction, electromagnetic radiation, radio signaling, infrared signaling or optical signaling, for example. One having ordinary skill in the art will appreciate that multiple components may interoperate and modifications and variations are possible of achieving the electrical and data communication.

[0053] In some embodiments described herein, the term “coupled,” “connected,” and the like, may be used herein to indicate a connection, although not necessarily directly, and may include wired and / or wireless connections.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and / or “including” when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0055] The term “network node” used herein can be any kind of network node comprised in a distributed computing system such as a communication network, which may be a wired or wireless communication network. A network node may further comprise any of base station (BS), radio base station, base transceiver station (BTS), base station controller (BSC), radio network controller (RNC), g Node B (gNB), evolved Node B (eNB or eNodeB), Node B, multi-standard radio (MSR) radio node such as MSR BS, multi-cell / multicast coordination entity (MCE), relay node, donor node controlling relay, radio access point (AP), transmission points, transmission nodes, Remote Radio Unit (RRU) Remote Radio Head (RRH), a core network node (e.g., mobile management entity (MME), self-organizing network (SON) node, a coordinating node, positioning node, MDT node, etc.), an external node (e.g., 3rd party node, a node external to the current network), nodes in distributed antenna system (DAS), a spectrum access system (SAS) node, an element management system (EMS), physical or virtual databases, cloud-based entities / servers / devices, or any other nodes or entities in a distributed computing environment, communication system, etc. The network node may also comprise test equipment. The term “radio node” used herein may be used to also denote a wireless device (WD) such as a wireless device (WD) or a radio network node.

[0056] In some embodiments, the non-limiting term management node is used. The management node may be a network node, or may be any type of computer or computing entity (e.g., physical device / server / etc., virtual device / server / etc., cloud-based device / server / etc.), which performs one or more management functions in a distributed computing environment or communication network, such as configuring anomaly detection, configuring, training, and deploying configurations, such as machine learning models, for network nodes to utilize in performing anomaly detection, analyzing data (monitoring / performance metrics associated with / received from network nodes) and identifying anomalies, etc.

[0057] In some embodiments, the non-limiting terms wireless device (WD) or a user equipment (UE) are used interchangeably. The WD herein can be any type of wireless device capable of communicating with a network node or another WD over radio signals, such as wireless device (WD). The WD may also be a radio communication device, target device, device to device (D2D) WD, machine type WD or WD capable of machine to machine communication (M2M), low-cost and / or low-complexity WD, a sensor equipped with WD, Tablet, mobile terminals, smart phone, laptop embedded equipped (LEE), laptop mounted equipment (LME), USB dongles, Customer Premises Equipment (CPE), an Internet of Things (loT) device, or a Narrowband loT (NB-IOT) device etc.

[0058] Also, in some embodiments the generic term “radio network node” is used. It can be any kind of a radio network node which may comprise any of base station, radio base station, base transceiver station, base station controller, network controller, RNC, evolved Node B (eNB), Node B, gNB, Multi-cell / multicast Coordination Entity (MCE), relay node, access point, radio access point, Remote Radio Unit (RRU) Remote Radio Head (RRH).

[0059] Note that although terminology from one particular wireless system, such as, for example, 3GPP LTE and / or New Radio (NR), may be used in this disclosure, this should not be seen as limiting the scope of the disclosure to only the aforementioned system. Other wireless systems, including without limitation Wide Band Code Division Multiple Access (WCDMA), Worldwide Interoperability for Microwave Access (WiMax), Ultra Mobile Broadband (UMB) and Global System for Mobile Communications (GSM), may also benefit from exploiting the ideas covered within this disclosure.

[0060] Note further, that functions described herein as being performed by a wireless device or a network node may be distributed over a plurality of wireless devices and / or network nodes. In other words, it is contemplated that the functions of the network node and wireless device described herein are not limited to performance by a single physical device and, in fact, can be distributed among several physical devices.

[0061] Note further that the functions described herein may be performed by a variety of distributed computing environments, including but not limited to communications networks which include one or more cloud-based nodes, entities, servers, etc.

[0062] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0063] Referring to the drawing figures, in which like elements are referred to by like reference numerals, there is shown in FIG. 1 a schematic diagram of a communication system 10, according to an embodiment, such as for example a 3GPP-type cellular network that may support standards such as LTE and / or NR (5G), which comprises an access network 12, such as a radio access network, and a core network 14. The access network 12 comprises a plurality of network nodes 16a, 16b, 16c (referred to collectively as network nodes 16), such as NBs, eNBs, gNBs or other types of wireless access points, each defining a corresponding coverage area 18a, 18b, 18c (referred to collectively as coverage areas 18). Each network node 16a, 16b, 16c is connectable to the core network 14 over a wired or wireless connection 20.

[0064] The system may further include a management node 21, which may be a network node 16, such as a network node 16 which provides one or more management node 21 functionalities, a cloud based node and / or server, etc. Management node 21 may communicate with and / or receive information (e.g., performance metrics) associated with one or more of the plurality of network nodes 16, e.g., via one or more of the access network 12, the core network 14, an internet connection, etc., and may communicate with and / or receive information associated with the plurality of network nodes 16 directly and / or via one or more intermediate devices.

[0065] A first wireless device (WD) 22a located in coverage area 18a is configured to wirelessly connect to, or be paged by, the corresponding network node 16a. A second WD 22b in coverage area 18b is wirelessly connectable to the corresponding network node 16b. While a plurality of WDs 22a, 22b (collectively referred to as wireless devices 22) are illustrated in this example, the disclosed embodiments are equally applicable to a situation where a sole WD is in the coverage area or where a sole WD is connecting to the corresponding network node 16. Note that although only two WDs 22 and three network nodes 16 are shown for convenience, the communication system may include many more WDs 22 and network nodes 16.

[0066] Also, it is contemplated that a WD 22 can be in simultaneous communication and / or configured to separately communicate with more than one network node 16 and more than one type of network node 16. For example, a WD 22 can have dual connectivity with a network node 16 that supports LTE and the same or a different network node 16 that supports NR. As an example, WD 22 can be in communication with an eNB for LTE / E-UTRAN and a gNB for NR / NG-RAN.

[0067] A management node 21 is configured to include a data labeling unit 24 which is configured to perform one or more management node 21 functions disclosed herein, such as determining labels for an input dataset corresponding to performance metrics of network nodes 16 in a communication system 10 based on a knowledge graph of the communication system 10.

[0068] Example implementations, in accordance with an embodiment, of the management node 21 discussed in the preceding paragraphs will now be described with reference to FIG. 2.

[0069] The communication system 10 includes a management node 21 provided in a communication system 10 and including hardware 28 enabling it to communicate with the one or more network nodes 16 and / or WDs 22. The hardware 28 may include a communication interface 30 for receiving and / or sending information (e.g., datasets corresponding to performance metrics, knowledge graphs and related configurations, machine learning models and related parameters, etc.) to / from one or more network nodes 16 via one or more wired and / or wireless connections.

[0070] In the embodiment shown, the hardware 28 of the management node 21 further includes processing circuitry 36. The processing circuitry 36 may include a processor 38 and a memory 40. In particular, in addition to or instead of a processor, such as a central processing unit and / or a graphics processing unit (GPU), and memory, the processing circuitry 36 may comprise integrated circuitry for processing and / or control, e.g., one or more processors and / or processor cores and / or FPGAs (Field Programmable Gate Array) and / or ASICs (Application Specific Integrated Circuitry) adapted to execute instructions. The processor 38 may be configured to access (e.g., write to and / or read from) the memory 40, which may comprise any kind of volatile and / or nonvolatile memory, e.g., cache and / or buffer memory and / or RAM (Random Access Memory) and / or ROM (Read-Only Memory) and / or optical memory and / or EPROM (Erasable Programmable Read-Only Memory).

[0071] Thus, the management node 21 further has software 42 stored internally in, for example, memory 40, or stored in external memory (e.g., database, storage array, network storage device, etc.) accessible by the network node 16 via an external connection. The software 42 may be executable by the processing circuitry 36. The processing circuitry 36 may be configured to control any of the methods and / or processes described herein and / or to cause such methods, and / or processes to be performed, e.g., by network node 16. Processor 38 corresponds to one or more processors 38 for performing management node 21 functions described herein. The memory 40 is configured to store data, programmatic software code and / or other information described herein. In some embodiments, the software 42 may include instructions that, when executed by the processor 38 and / or processing circuitry 36, causes the processor 38 and / or processing circuitry 36 to perform the processes described herein with respect to management node 21. For example, processing circuitry 36 of the management node 21 may execute the instructions to perform the functions of the data labeling unit 24 disclosed herein, such as determining labels for an input dataset corresponding to performance metrics of network nodes 16 in a communication system 10 based on a knowledge graph of the communication system 10, e.g., stored in memory 40 and / or in a separate entity (e.g., cloud database, server, etc.) which may be accessible via communication interface 30.. The communication system 10 further includes network nodes 16 already referred to. The network nodes 16 may have similar hardware as the management node 21. In some embodiments, a network node 16 may be a management node 21. The network node 16 may include hardware 44. Hardware 44 of some network nodes 16 may include a communication interface 46, configured for providing a wired and / or wireless connection 47 with one or more other network nodes 16, management node 21, WD 22, etc. The network node 16 may include processing circuitry 48, which may be similar in one or more aspects to the processing circuitry 36 of the management node 21. In some embodiments, one or more network nodes 16 may have processing circuitry 48, including one or more processors 50 and / or memory 52, with substantially limited computational resources (e.g., processor 50 speed, processor 50 cores, memory 52 bandwidth, memory 52 size, etc.) as compared to processing circuitry 36 of the management node 21. In some embodiments, processing circuitry 48 of a first network node 16a may be configured for reporting and / or determining performance metrics associated with entities of communication system 10, such as network node 16a (or other network nodes 16b, 16c, etc.), WDs 22, core network 14 entities, cloud-based entities, other nodes, databases, servers, etc. In some embodiments, performance metrics associated with a network node 16 may be determined by the network node 16 itself and / or by one or more other entities in system 10 (e.g., management node 21, other network nodes 16, core network 14 entities, cloud-based entities / servers, etc.). Network node 16 may further include software 53, which may be similar in one or more aspects to software 42 of management node 21, described herein.

[0072] In some embodiments, the inner workings of the management node 21 and the network node 16 may be as shown in FIG. 2 and independently, the surrounding network topology may be that of FIG. 1.

[0073] The teachings of some of these embodiments may improve the data rate, latency, and / or power consumption and thereby provide benefits such as reduced user waiting time, relaxed restriction on file size, better responsiveness, extended battery lifetime, etc. In some embodiments, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve.

[0074] Although FIGS. 1 and 2 show various “units” such as data labeling unit 24 as being within a respective processor, it is contemplated that these units may be implemented such that a portion of the unit is stored in a corresponding memory within the processing circuitry. In other words, the units may be implemented in hardware or in a combination of hardware and software within the processing circuitry.

[0075] FIG. 3 is a flowchart of an example process in a management node 21 for data labeling for performance metrics based on a knowledge graph, as disclosed herein. One or more blocks described herein may be performed by one or more elements of management node 21 such as by one or more of processing circuitry 36 (including the data labeling unit 24), processor 38, and / or communication interface 30. The management node may be configured with a knowledge graph including a plurality of links for a plurality of nodes in the communication network. Management node 21 is configured to detect (Block S100) anomalous metric data associated with at least one of the plurality of nodes (e.g., network nodes 16, WDs 22, etc.) of the knowledge graph based on a first historical dataset comprising metric data representative of a normal state of the plurality of nodes of the knowledge graph. Management node 21 is configured to determine (Block S102) at least one anomalous node (e.g., at least one network node 16 and / or at least one WD 22, etc.) of the plurality of nodes based on the anomalous metric data. Management node 21 is configured to determine (Block SI 04) at least one label for the anomalous metric data based on at least one link from the knowledge graph associated with the at least one anomalous node.

[0076] In some embodiments, management node 21 is further configured to train a machine learning (ML) model using the anomalous metric data and the at least one label. In some embodiments, management node 21 is further configured to detect additional anomalous metric data associated with at least one of the plurality of nodes of the knowledge graph, and determine at least one additional label for the additional anomalous metric data based on the trained ML model. In some embodiments, the determining of the at least one label for the anomalous metric data further includes determining a plurality of candidate paths for traversing the knowledge graph, each of the plurality of candidate paths traversing at least one anomalous node of the plurality of nodes, and determining the at least one label based on metrics associated with at least one of the plurality of candidate paths.

[0077] In some embodiments, the determining of the at least one label for the anomalous metric data further includes determining a rank order of the plurality of candidate paths, and determining at least one first label based on at least one corresponding highest-ranked candidate paths of the plurality of candidate paths. In some embodiments, the determining of the rank order of the plurality of candidate paths includes determining a respective score for each candidate path of the plurality of candidate paths, where the respective score is determined based on at least one of a number of anomalous nodes of the respective candidate path, a rate of anomaly of the respective candidate path, a severity of anomaly of the respective candidate path, and / or a number of anomalous metrics of the respective candidate path.

[0078] In some embodiments, the at least one highest-ranked candidate path, i.e., the most anomalous path, traverses a first set of nodes, and the determining of the at least one first label based on the at least one highest-ranked candidate path includes determining a start node of the first set of nodes based on a hierarchical relationship among the first set of nodes, determining a first plurality of anomalous metrics associated with a first number of nodes of the first set of nodes, where the first plurality of anomalous metrics includes a first plurality of metric names and a corresponding first plurality of metric values, determining a first keyword based on the first plurality of metric names, and determining the at least one first label based on the first keyword. In some embodiments, management node 21 is further configured to validate the at least one first label based on the first number of nodes associated with the first plurality of anomalous metrics exceeding a preconfigured threshold number of nodes. For example, in some embodiments, to determine that nodes are affected by the deviations of those nodes, the threshold may be set to be higher than 50%, higher than 75%, higher than 90%, etc., as appropriate. This threshold may be configurable, e.g., by a network operator, end user, etc.

[0079] In some embodiments, management node 21 is further configured to, responsive to a failure, to validate the at least one first label, determine a second candidate path traversing a second set of nodes, determine a second plurality of anomalous metrics associated with a second number of nodes of the second set of nodes, the second plurality of anomalous metrics including a second plurality of metric names and a corresponding second plurality of metric values, determine a second keyword based on the second plurality of metric names, determine at least one second label based on the second keyword, and validate the at least one second label based on the second number of nodes associated with the second plurality of anomalous metrics exceeding the preconfigured threshold number of nodes. In some embodiments, the second candidate path is one of: a second-highest-ranked candidate path of the plurality of candidate paths, i.e., the most anomalous candidate path, traversing the second set of nodes, and a sub-path of the at least one highest-ranked candidate path, where the sub-path traverses the second set of nodes, the second set of nodes is a subset of the first set of nodes, and the second set of nodes includes at least one dependent node of the start node. In some embodiments, management node 21 is further configured to receive, prior to the step of detecting, the first historical dataset including at least one of monitoring metrics, traces, and log files associated with the plurality of nodes of the knowledge graph.

[0080] In some embodiments, management node 21 is further configured to receive the knowledge graph from at least one network node 16 in the communication network, and receive the anomalous metric data from at least one network node 16 in the communication network. In some embodiments, the management node 21 may be preconfigured with the knowledge graph.

[0081] Having described the general process flow of arrangements of the disclosure and having provided examples of hardware and software arrangements for implementing the processes and functions of the disclosure, the sections below provide details and examples of arrangements for data labeling of input data (e.g., performance metrics associated with network nodes 16 in a communication system 10 based on knowledge graphs.

[0082] In some embodiments, an automated solution for labeling data is provided in a cloudbased system (e.g., a management node 21 in a communication system 10) using knowledge extracted from a knowledge graph.

[0083] Some embodiments enable processing of unlabeled data and inferring information and meaning (e.g., labels for machine learning model training), while traversing the input / stored knowledge graph. For example, FIG. 4 illustrates an example method for labeling data, e.g., as implemented in a management node 21.

[0084] As illustrated in the example of FIG. 4, in some embodiments, the labeling process (e.g., as implemented by management node 21) receives inputs (Block S106) including a knowledge graph (which may be a data structure which represents nodes, links, relationships <nodes, links>, etc. of a knowledge base), historical and / or live data collected from the managed communication system 10 (e.g., monitoring metrics, traces, log files, etc., associated with / reported by network nodes 16, WDs 22, other entities in communication system 10, access network 12, core network 14, etc.). This collected data may also represent the unlabeled dataset that the method labels when processing (reasoning over) the input knowledge graph.

[0085] In some embodiments, as illustrated in FIG. 4, the method may determine labels for the data (Block S108), e.g., based on the knowledge graph, and may output (Block SI 10) a labeled dataset, which may include one or more labels (generated by the method), with the labels associated with one or more of the data entries (which were received unlabeled in Block s 106).

[0086] In some embodiments, the labeling process uses historical data (e.g., historical performance metrics associated with one or more network nodes 16 or other entities of communication system 10) to train an Anomaly Detection (AD) model to store the statistical properties of the data characterizing the communication system 10 (e.g., cloud system) under “normal” (e.g., non-anomalous operation, error-free operation, etc.) circumstances. The labeling process may use the data to assign and / or find / determine properties characterizing the knowledge graph nodes. The nodes of the knowledge graph may correspond to network nodes 16 and / or other entities of communication system 10. Given the AD model and the knowledge graph nodes (e.g., analyzed nodes), the labeling system may: process (e.g., reason over) input live data, which is generally (e.g., partially, mostly, entirely, etc.) unlabeled, and maps it to the AD model to find anomalous data (e.g., anomalous performance metrics); map these anomalous performance metrics, if found, to the knowledge graph nodes (e.g., the analyzed nodes) to find possible / candidate paths, score / rank the candidate paths and select the most relevant (e.g., highest scored / ranked) path, generates a label associated with the selected path.

[0087] FIG. 5 illustrates another example flowchart of a labeling process for machine learning model training (e.g., by management node 21) according to one or more embodiments of the present disclosure. The management node 21 (or other entity which provides labeling functionality) labels input data (Block SI 12) based at least in part on the knowledge graph. The obtained labeled dataset may be used to train a machine learning model (Block S 114), which may be represented by / stored in a data structure corresponding to machine learning model weights, parameters, layers, etc., and which may be deployed (Block SI 16) in a communication system 10, such as a cloud system, access network 12, core network 14, one or more network nodes 16, WDs 22, etc. The trained machine learning model may contain a limited set of knowledge from the knowledge base / knowledge graph, and may be used more efficiently, e.g., in distributed deployments, as compared to existing systems.

[0088] FIG. 6 illustrates another example of a labeling process for machine learning model training (e.g., by management node 21) according to one or more embodiments of the present disclosure.

[0089] The management node 21 may receive, or be provided with (Block S128) a knowledge graph that contains information about different relationships that exist between objects, and receives (Block SI 30) historical data, including historical metrics, performance metrics, configuration logs, traces, etc. In some embodiments, the knowledge graph may be obtained from an external Knowledge Graph Generation Model. In some embodiments, the historical data (historical performance metrics), may be collected from a monitoring system or other entity which provides metrics, monitoring information, error information, performance information, etc. In some embodiments, the historical data is used for anomaly detection, which is used to generate labels for live data.

[0090] An Anomaly Detection (AD) process, as implemented by a management node 21 or other entity of communication system 10, may include training (Block SI 32) a model (e.g., machine learning model, statistical model, etc.) that may be configured to identify anomalous metrics in the historical metrics data. The AD process may be implemented using a variety of anomaly detection techniques. An anomaly may be defined as any system state that falls out of a “normal” range, where normal may include, e.g., operation which is error-free, operation having a number of errors which falls below a threshold, operation which does not exhibit security breaches / threats / vulnerabilities, operation which results in performance metrics which fall within a range specified by a technical standard, operation which satisfies a minimum performance threshold, etc.

[0091] In some embodiments, the process assigns (Block S134) the historical metrics data to one or more nodes (e.g., corresponding to network nodes 16 or other entities of communication system 10) in the input Knowledge Graph. In some embodiments, when live data is received (Block S136) by management node 21, the trained AD model (Block SI 38) may be configured to identify lists of anomalous metrics (Step S140). If such a list is empty, the management node 21 assigns a “normal_state” (Step S 141) (e.g., instead of assigning a faulty state or a security threat state) as a label for one or more entries of the live metrics data. If the anomalous metrics list is not empty, the process uses the list of anomalous metrics (Step SI 40) and the knowledge graph and node metrics (Step S142) to identify the anomalous nodes from the knowledge graph (Block S144). In other words, given a live dataset to be labeled (i.e., unlabeled dataset), the trained AD model, the knowledge graph, and the nodes, along with the assigned metrics, are used to identify anomalous nodes (e.g., anomalous network nodes 16) in the knowledge graph. This is done by matching the anomalous metrics onto the metrics assigned to the nodes of the knowledge graph. These nodes are classified as anomalous nodes and represent the candidate nodes (Step S146) to be used for the labeling process, as described herein.

[0092] In some embodiments, possible paths are identified (Block SI 48) between the candidate nodes. Paths may be identified according to a variety of different strategies. For example, an exhaustive search may be performed, where list(s) of all possible paths are identified. Another example approach may include performing a guided search, where different criteria could be identified to exclude some paths. These paths represent the candidate paths. In some embodiments, candidate paths must include at least one anomalous node. In some embodiments, candidate paths must include an anomalous node at the start of the path, and / or at the end of the path. In some embodiments, candidate paths must include only anomalous nodes. In some embodiments, candidate paths may include a mix of anomalous nodes and non-anomalous nodes.

[0093] The generated candidate paths (Step S150) may then be assigned scores (Block SI 52). The scores may be calculated according to a variety of different criteria in various embodiments, such as number of anomalous metrics in a path, a number of anomalous nodes in a path, a ratio of the nodes with anomalous metrics to the total number of nodes in a path, a ratio of nodes with anomalous metrics which exceed a certain threshold (e.g., the number of anomalous metrics, the amount by which one or more anomalous metrics deviates from a preconfigured normal operation range or value, etc.), a weight value assigned to a node, e.g., based on a type of network node 16 corresponding to the node (e.g., some network nodes 16 may be relatively more critical / important for normal operation of the communication system 10), where such weights may be preconfigured in the knowledge graph and / or determined by management node 21, e.g., based on a mapping of network node 16 types and weight values, etc. According to the assigned scores, the candidate paths may be ranked (Step SI 54), and the paths and the rankings / scores may be stored in a corresponding data structure.

[0094] In some embodiments, the path with the highest score may be selected to be analyzed (Step S156). After selecting a path, the process determines / generates a label corresponding to the selected path (Step S158), as described herein. If a label is not determined at this step (Block S162), the process may return back to Step S156, and selects the next path with highest score. If the label was successfully determined, then the live data, the labels, dataset, and / or generated labels are used to train the machine learning model (Block SI 66), e.g., for future anomaly detection / prediction.

[0095] Thus, in some embodiments, the output of Block S160 is the labelled input data, e.g., labeled live metrics / performance metrics / monitoring metrics / etc. In some embodiments, the labels may be characterized as extracting some knowledge from the knowledge graph for the given input data. A set of such labelled input data may be used to train an ML model (e.g., a neural network) using a variety of supervised machine learning approaches. The ML model may then be deployed in the communication system 10, e.g., in the cloud, which may be able to more efficiently handle a variety of inputs (performance metrics / monitoring metrics) that were used for the labeling, as compared to using the knowledge graph itself.

[0096] In some embodiments, the knowledge graph may change over time due to a variety reasons. By re-generating and comparing a subset of labels for previously labelled input data, embodiments of the present disclosure may identify if the change in the knowledge graph affects the labels / answers for a given set of input data. If change is not observed, then there is no need to re-train or re-distribute the ML model. This may advantageously reduce cost, energy, delay, etc., as compared to re-distributing an entire knowledge graph after each change.

[0097] FIG. 7 is a flowchart of an example process for generating a label for a selected candidate path (e.g., Block S160 of FIG. 6) according to one or more embodiments of the present disclosure. The process illustrated in FIG. 7 receives an input (Step S172) of the selected path (e.g., determined in Block S156 of FIG. 6), and identifies (Block S174) the nodes (e.g., corresponding to network nodes 16 and / or other entities of communication system 10) which form the path and the nodes’ assigned metrics (e.g., metric names, corresponding values, etc.) (Step S 176).

[0098] A node for starting the analysis is identified (Block S178). The identification of the starting node or the most relevant node may be performed, e.g., using rules, which may be stored in a Rules Repository (e.g., a cloud-based storage accessible by management node 21, or stored in memory 40 of management node 21). For example, the rules may include a hierarchical ordering of the nodes, which may specify to start the analysis with the underlying physical node, then the dependent nodes, and so on. Another example rule in the context of a multi-layer cloud stack communication system 10 may include ,e.g., starting the analysis with the bare-metal layer, then the virtualization layer (e.g., OpenStack), and then the container layer (e.g., Kubernetes), then the cloud-native network functions (CNFs). Such rules may, for example, be provided as an input by the user, administrator, or by an external system. In some embodiments, the knowledge graph may contain relationships that may be used to define various different hierarchies.

[0099] The relevant nodes / metrics (Step S 182) are identified and the metrics of the relevant node that are deviated more than a specified threshold (e.g., Thresholdl) are identified (Block SI 84). This threshold may be given by an end-user or administrator, and / or may be preconfigured, and may be updated, e.g., based on the measured accuracy of the labeling system.

[0100] After identifying the list of the metrics (Step SI 86) that are deviated more than a threshold, keyword analysis is performed (Block SI 88). In some embodiments, a keyword analysis performs a word2vec computation, which determines all the combinations of these tokens and the frequency of the ones that are mostly present.

[0101] The list of keys (Step S190) with their corresponding frequency may be analyzed and the most frequent key(s) / word(s) are determined (Block S192), which may represent a candidate label (Step S194). In some embodiments, the candidate label may be validated (Block S196), for example, by analyzing one or more other nodes in the selected path to evaluate whether their same metric (e.g., the candidate label) is deviated more than the specified threshold.

[0102] For example, if the percentage of the nodes with the deviated metric is larger than a threshold (i.e., Threshold!) (Block S198), then the label may be considered to be validated (Block S200). For instance, if 50% of the nodes in the selected path are impacted by the same metric deviation, then the candidate label would be validated for that path. The validated label is output (Step S202), e.g., for machine learning model training using the labeled dataset.

[0103] In this example, if the percentage of the nodes with deviated metric is smaller than the identified threshold (i.e., Threshold!), then the process selects all the nodes that are dependent on the relevant node (identified in step 2) to validate the candidate label (Block S204).

[0104] For example, the nodes may be analyzed to evaluate whether they are affected by the same metrics (keys) (Block S206). If these metrics are deviated more than a specified threshold, e.g., Threshold! (note that Thresholdl, Threshold 2, and / or Threshold! could have the same or different values) (Block S208), then the candidate label is validated (Block S200), and process outputs the validated label (Step S202).

[0105] Otherwise, if there are still nodes in the path that are not analyzed (Block S210), the process returns to Block S 178. It starts analyzing the next level of nodes according to the rules in the Rules Repository. Once all the nodes in the path are analyzed and a candidate label is not found, the method returns “No Label” (Step S212). In some embodiments, the process checks only the dependent nodes (e.g., of the start node), and if same metric is deviated from threshold, then the label is validated; if not, then the process proceeds to analyze the next higher level of nodes, e.g., according to the Rules Repository.

[0106] FIG. 8 illustrates an example knowledge graph 54 according to one or more embodiments of the present disclosure. In this example, the knowledge graph 54 is composed of 10 nodes (e.g., representing network nodes 16 of communication system 10), which nodes are labeled:

[0107] LB 1 / 2: Loadbalancerl / 2,

[0108] FE1 / 2: FrontEndl / 2,

[0109] BE1 / 2: BackEndl / 2, DB1: DataBasel, and

[0110] S 1 / 2 / 3: Serverl / 2 / 3.

[0111] The arrows in FIG. 8 correspond to relationships (edges) between the nodes. The directionality of these arrows correspond to hierarchical information (e.g., LB1 is above FE1 and FE2 in the hierarchy), which may be stored in the knowledge graph 54 and / or a separate Rules Repository.

[0112] Each node of the knowledge graph 54 may be characterized by a variety of different metrics and their statistical properties / values. For example, the performance metrics / monitoring metrics data (dataset) may be a collection of time series data.

[0113] By comparing the data to a trained AD model and mapping the data to the knowledge graph 54, embodiments of the present disclosure may identify, for example, the following paths: “[LB1, FE1, LB2, BE1, SI]”, “[LB1, FE1, LB2, BE2, DB1, S3]”, etc.

[0114] Referring to FIG. 8, in one example, BE1 is dependent on SI; thus, if SI is not working, it may be assumed, e.g., based on rules, relationships, or other configuration information, that BE1 will not be running properly either. These rules and relationships may be specified in the Rules Repository and / or the knowledge graph 54.

[0115] In some embodiments, an exhaustive search may be performed in which candidate paths may include both candidate nodes and non-candidate nodes. In embodiments, a guided search may be performed, in which only paths that include only candidate nodes are searched. Other embodiments may utilize a variety of search techniques, e.g., only searching paths which include at least a threshold number of nodes (e.g., 50% of total nodes) which are candidate nodes and / or anomalous nodes, etc.

[0116] In the example of FIG. 8, at a first time (Time 1), the path which is determined is LB1, FE1, LB2, BE1, and SI, and a label node_memoryfault is determined based thereon. At a second time (Time 2), the path which is determined is LB1, FE1, LB2, BE2, DB1, and S3, and a label HDD fault is determined based thereon. At a third time (Time 3), the path which is determined for the dataset is LB1, FE2, LB2, BE2, DB1, and a label DB Fault is determined for the dataset at Time 3 based thereon.

[0117] FIG. 9 illustrates an example path and node metrics corresponding to Timel of FIG. 8. At Time 1, for example, Serverl (SI) may be identified (by management node 21) as the relevant node to analyze in this path. Next, the metrics data are analyzed to determine that, for example, the metrics illustrated in bold typeface in FIG. 9 are the metrics which have deviated more than a preconfigured threshold (e.g., thresholdl of FIG. 7). Finally, the extracted metrics data is processed and the label “node_memory fault” is determined (where ‘node_memory’ is determined to be the most frequent token in this example) in the context of fault management process. In some embodiments, labels may be related to problems other than faults. For example, a high load may be received at LB 1 and LB2. One metric that may be an indication of a high load could be “user_request_count”, for example. Here, high load is an example of a label that is not necessarily related to a fault.

[0118] As will be appreciated by one of skill in the art, the concepts described herein may be embodied as a method, data processing system, computer program product and / or computer storage media storing an executable computer program. Accordingly, the concepts described herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a “circuit” or “module.” Any process, step, action and / or functionality described herein may be performed by, and / or associated to, a corresponding module, which may be implemented in software and / or firmware and / or hardware. Furthermore, the disclosure may take the form of a computer program product on a tangible computer usable storage medium having computer program code embodied in the medium that can be executed by a computer. Any suitable tangible computer readable medium may be utilized including hard disks, CD-ROMs, electronic storage devices, optical storage devices, or magnetic storage devices.

[0119] Some embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems and computer program products. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer (to thereby create a special purpose computer), special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions may also be stored in a computer readable memory or storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0120] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0121] It is to be understood that the functions / acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0122] Computer program code for carrying out operations of the concepts described herein may be written in an object oriented programming language such as Python, Java® or C++. However, the computer program code for carrying out operations of the disclosure may also be written in conventional procedural programming languages, such as the "C" programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0123] Many different embodiments have been disclosed herein, in connection with the above description and the drawings. It will be understood that it would be unduly repetitious and obfuscating to literally describe and illustrate every combination and subcombination of these embodiments. Accordingly, all embodiments can be combined in any way and / or combination, and the present specification, including the drawings, shall be construed to constitute a complete written description of all combinations and subcombinations of the embodiments described herein, and of the manner and process of making and using them, and shall support claims to any such combination or subcombination.

[0124] Abbreviations that may be used in the preceding description include:

[0125] ML Machine Learning

[0126] GPU Graphics Processing Unit

[0127] AD Anomaly Detection

[0128] LB Load Balancer

[0129] FE Front-End

[0130] BE Back-End

[0131] It will be appreciated by persons skilled in the art that the embodiments described herein are not limited to what has been particularly shown and described herein above. In addition, unless mention was made above to the contrary, it should be noted that all of the accompanying drawings are not to scale. A variety of modifications and variations are possible in light of the above teachings and the following claims.

Claims

What Is Claimed:

1. A management node (21) in a communication network, the management node (21) being configured with a knowledge graph comprising a plurality of links for a plurality of nodes (16) in the communication network, the management node (21) comprising processing circuitry 36 configured to: detect anomalous metric data associated with at least one of the plurality of nodes (16) of the knowledge graph based on a first historical dataset comprising metric data representative of a normal state of the plurality of nodes (16) of the knowledge graph; determine at least one anomalous node (16) of the plurality of nodes (16) based on the anomalous metric data; and determine at least one label for the anomalous metric data based on at least one link from the knowledge graph associated with the at least one anomalous node (16).

2. The management node (21) of Claim 1, wherein the processing circuitry (36) is further configured to: train a machine learning (ML) model using the anomalous metric data and the at least one label.

3. The management node (21) of Claim 2, wherein the processing circuitry (36) is further configured to: detect additional anomalous metric data associated with at least one of the plurality of nodes (16) of the knowledge graph; and determine at least one additional label for the additional anomalous metric data based on the trained ML model.

4. The management node (21) of any one of Claims 1-3, wherein the determining of the at least one label for the anomalous metric data further includes:determining a plurality of candidate paths for traversing the knowledge graph, each of the plurality of candidate paths traversing at least one anomalous node ( 16) of the plurality of nodes (16); and determining the at least one label based on metrics associated with at least one of the plurality of candidate paths.

5. The management node (21) of Claim 4, wherein the determining of the at least one label for the anomalous metric data further includes: determining a rank order of the plurality of candidate paths; and determining at least one first label based on at least one corresponding highest-ranked candidate paths of the plurality of candidate paths.

6. The management node (21) of Claim 5, wherein the determining of the rank order of the plurality of candidate paths includes: determining a respective score for each candidate path of the plurality of candidate paths, the respective score being based on at least one of: a number of anomalous nodes (16) of the respective candidate path; a rate of anomaly of the respective candidate path; a severity of anomaly of the respective candidate path; and a number of anomalous metrics of the respective candidate path.

7. The management node (21) of any one of Claims 5 and 6, wherein the at least one highest-ranked candidate path traverses a first set of nodes (16); and the determining of the at least one first label based on the at least one highest-ranked candidate path includes: determining a start node (16) of the first set of nodes (16) based on a hierarchical relationship among the first set of nodes (16); determining a first plurality of anomalous metrics associated with a first number of nodes (16) of the first set of nodes (16), the first plurality of anomalous metrics including a first plurality of metric names and a corresponding first plurality of metric values; determining a first keyword based on the first plurality of metric names; anddetermining the at least one first label based on the first keyword.

8. The management node (21) of Claim 7, wherein the processing circuitry (36) is further configured to: validate the at least one first label based on the first number of nodes (16) associated with the first plurality of anomalous metrics exceeding a preconfigured threshold number of nodes (16).

9. The management node (21) of Claim 8, wherein the processing circuitry (36) is further configured to: responsive to a failure to validate the at least one first label, determine a second candidate path traversing a second set of nodes (16); determine a second plurality of anomalous metrics associated with a second number of nodes (16) of the second set of nodes (16), the second plurality of anomalous metrics including a second plurality of metric names and a corresponding second plurality of metric values; determine a second keyword based on the second plurality of metric names; determine at least one second label based on the second keyword; and validate the at least one second label based on the second number of nodes (16) associated with the second plurality of anomalous metrics exceeding the preconfigured threshold number of nodes (16).

10. The management node (21) of Claim 9, wherein the second candidate path is one of: a second-highest-ranked candidate path of the plurality of candidate paths traversing the second set of nodes (16); and a sub-path of the at least one highest-ranked candidate path, the sub-path traversing the second set of nodes (16), the second set of nodes (16) being a subset of the first set of nodes (16), the second set of nodes (16) including at least one dependent node (16) of the start node (16).

11. The management node (21) of any one of Claims 1-10, wherein the processing circuitry (36) is further configured to: receive, prior to the detecting of the anomalous metric data, the first historical dataset including at least one of monitoring metrics, traces, and log files associated with the plurality of nodes (16) of the knowledge graph.

12. The management node (21) of any one of Claims 1-11, wherein the processing circuitry (36) is further configured to at least one of: receive the knowledge graph from at least one network node (16) in the communication network; and receive the anomalous metric data from at least one network node (16) in the communication network.

13. A method implemented in a management node (21) in a communication network, the management node (21) being configured with a knowledge graph comprising a plurality of links for a plurality of nodes ( 16) in the communication network, the method comprising: detecting anomalous metric data associated with at least one of the plurality of nodes ( 16) of the knowledge graph based on a first historical dataset comprising metric data representative of a normal state of the plurality of nodes (16) of the knowledge graph; determining at least one anomalous node (16) of the plurality of nodes (16) based on the anomalous metric data; and determining at least one label for the anomalous metric data based on at least one link from the knowledge graph associated with the at least one anomalous node (16).

14. The method of Claim 13, wherein the method further comprises: training a machine learning (ML) model using the anomalous metric data and the at least one label.

15. The method of Claim 14, wherein the method further comprises:detecting additional anomalous metric data associated with at least one of the plurality of nodes (16) of the knowledge graph; and determining at least one additional label for the additional anomalous metric data based on the trained ML model.

16. The method of any one of Claims 13-15, wherein the determining of the at least one label for the anomalous metric data further includes: determining a plurality of candidate paths for traversing the knowledge graph, each of the plurality of candidate paths traversing at least one anomalous node ( 16) of the plurality of nodes (16); and determining the at least one label based on metrics associated with at least one of the plurality of candidate paths.

17. The method of Claim 16, wherein the determining of the at least one label for the anomalous metric data further includes: determining a rank order of the plurality of candidate paths; and determining at least one first label based on at least one corresponding highest-ranked candidate paths of the plurality of candidate paths.

18. The method of Claim 17, wherein the determining of the rank order of the plurality of candidate paths includes: determining a respective score for each candidate path of the plurality of candidate paths, the respective score being based on at least one of: a number of anomalous nodes (16) of the respective candidate path; a rate of anomaly of the respective candidate path; a severity of anomaly of the respective candidate path; and a number of anomalous metrics of the respective candidate path.

19. The method of any one of Claims 17 and 18, wherein the at least one highest- ranked candidate path traverses a first set of nodes (16); andthe determining of the at least one first label based on the at least one highest-ranked candidate path includes: determining a start node (16) of the first set of nodes (16) based on a hierarchical relationship among the first set of nodes (16); determining a first plurality of anomalous metrics associated with a first number of nodes (16) of the first set of nodes (16), the first plurality of anomalous metrics including a first plurality of metric names and a corresponding first plurality of metric values; determining a first keyword based on the first plurality of metric names; and determining the at least one first label based on the first keyword.

20. The method of Claim 19, wherein the method further comprises: validating the at least one first label based on the first number of nodes (16) associated with the first plurality of anomalous metrics exceeding a preconfigured threshold number of nodes (16).

21. The method of Claim 20, wherein the method further comprises: responsive to a failure to validate the at least one first label, determining a second candidate path traversing a second set of nodes (16); determining a second plurality of anomalous metrics associated with a second number of nodes (16) of the second set of nodes (16), the second plurality of anomalous metrics including a second plurality of metric names and a corresponding second plurality of metric values; determining a second keyword based on the second plurality of metric names; determining at least one second label based on the second keyword; and validating the at least one second label based on the second number of nodes (16) associated with the second plurality of anomalous metrics exceeding the preconfigured threshold number of nodes (16).

22. The method of Claim 21 , wherein the second candidate path is one of: a second-highest-ranked candidate path of the plurality of candidate paths traversing the second set of nodes (16); anda sub-path of the at least one highest-ranked candidate path, the sub-path traversing the second set of nodes (16), the second set of nodes (16) being a subset of the first set of nodes (16), the second set of nodes (16) including at least one dependent node (16) of the start node (16).

23. The method of any one of Claims 13-22, wherein the method further comprises: receive, prior to the detecting of the anomalous metric data, the first historical dataset including at least one of monitoring metrics, traces, and log files associated with the plurality of nodes (16) of the knowledge graph.

24. The method of any one of Claims 13-23, wherein the method further comprises at least one of: receiving the knowledge graph from at least one network node (16) in the communication network; and receiving the anomalous metric data from at least one network node ( 16) in the communication network.

Images