Systems and methods for data tagging using knowledge graphs in cloud

Abstract

A method, system, and device are disclosed. A method implemented in a management node configured with a knowledge graph comprising a plurality of links for a plurality of nodes in a communication network is provided. In one embodiment, a method includes detecting abnormal metric data associated with at least one of a plurality of nodes of a knowledge graph based on a first historical data set including metric data representing a normal state of the plurality of nodes of the knowledge graph, at least one anomalous node of the plurality of nodes is determined based on the anomalous metric data and at least one tag of the anomalous metric data is determined based on at least one link from a knowledge graph associated with the at least one anomalous node.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to wireless communications, and in particular to performance metric data labeling using knowledge graphs in wireless communication networks. BACKGROUND

[0002] The Third Generation Partnership Project (3GPP) has developed and is developing standards for fourth generation (4G) (also referred to as Long Term Evolution (LTE)) and fifth generation (5G) (also referred to as New Radio (NR)) wireless communication systems. Such systems provide, among other features, wide-area broadband communication between network nodes, such as base stations, and mobile wireless devices (WDs), as well as communication between network nodes and between WDs. 3GPP is also developing standards for sixth generation (6G) wireless communication networks.

[0003] Knowledge bases have been used in telecommunication systems for decades. Knowledge bases store facts about (a subset of) the world, and these facts can for example be used to reason and infer new facts or highlight inconsistencies. Initially, knowledge bases were mainly part of expert systems. As part of machine push solutions, this technology is again gaining momentum. The “knowledge” of a knowledge base is typically stored in a graph form (e.g., a data structure), and thus, the term “knowledge graph” can be used to refer to a knowledge base or some aspect thereof. Expert and machine reasoning systems can use knowledge graphs to infer new facts and answer questions, or for example to pinpoint root causes, identify undesirable system states, etc.

[0004] As knowledge can be complex in nature, knowledge graphs can quickly become complex, e.g., with thousands of nodes and edges. Storing such graphs can consume storage / memory, and traversing them can consume significant amounts of time and computational resources. For real-time applications, decision making over large knowledge graphs through such traversals can not be practical or feasible. In the edge cloud context and / or in IoT environments, there can be significant memory and / or computational limitations, and thus the entire knowledge base can not be available for decision making. As a result, knowledge graph traversal can be more expensive, especially when the knowledge graph needs to be repeatedly traversed to answer different questions (e.g., “is the system in a fault state?”, “is there a security threat?” etc.).

[0005] With the rise of data-centric technologies in telecommunication systems, machine learning (ML) techniques are increasingly being used in various domains including fault, performance, and security management. Such ML techniques can use data collected from the system(s), train detection and / or prediction models, and then use the models for online detection and / or prediction. ML techniques have become an important knock on the door to build automated and intelligent cloud systems. For example, they allow applications running in a cloud system to become more accurate in identifying their outcomes without having to be explicitly programmed to do so. Different ML algorithms have been proposed in the literature, including supervised machine learning, e.g., using labeled datasets to train algorithms to accurately classify data or predict outcomes. For example, some existing supervised machine learning systems have been characterized to answer questions with various levels of accuracy and efficiency (e.g., similar to questions answered by a knowledge graph).

[0006] Labeling is a notable function for configuring supervised machine learning, as the quality of labels can directly impact the performance of supervised learning. Data labeling is often a manual and expensive process and requires prior knowledge about the monitored system. Such manual solutions can not be able to identify all labels in large cloud systems generally, and can also not be able to guarantee full coverage of labels, especially for large datasets. Additionally, such manual solutions can not be possible to identify rare or unknown events when they occur. Traditional label learning methods require labelers to be experts and have prior knowledge about the monitored system to guarantee the correctness of output labels. This leads to expensive labeling costs and requires a large amount of time and effort to provide such manual configuration and knowledge. Additionally, it can not be applicable to large datasets, especially for large systems.

[0007] Some existing systems have built knowledge graphs from extracted facts from large-scale systems. One of the challenges in this process is to consistently infer labels. Some existing systems have considered approaches to generate a consistent set of labels and relations for each node based on resolving co-reference entities in the knowledge graph, and label nodes using classification methods by considering ontology information and neighboring labels.

[0008] In some existing systems, knowledge embedding methods are utilized to infer new relational facts from large-scale knowledge graphs and text corpora. For example, a triple classification task can be performed, which can label a triple (node, relation, node) as true or false.

[0009] In some existing systems, edges on a given directed graph are labeled with integers based on the premise that the edges should be assigned. For example, an edge labeling scheme can seek to balance and / or maximize continuity of each individual vertex, such that local decisions greedily progress toward a global optimum.

[0010] In some existing systems, learning is performed on the labeled graph to measure regularity and similarity of graph nodes. The graph can be partitioned into several well-separated clusters within which labels are approximately constant. Labels can be assigned based on similarity between grouped nodes in each cluster.

[0011] In some existing systems, resolution of a labeled knowledge graph is performed. This can include identification of triples, including source nodes, destination nodes, and links between them, each of which can be labeled. Natural language models launch can be used to provide input functionality to independently developed applications.

[0012] In some existing systems, a plurality of input instances and corresponding matches between nodes in a knowledge graph are identified. For the identified matching objects, the system can identify a common class object represented by the corresponding nodes of the knowledge graph. A certain percentage of the identified matching objects can be descendants of the common class object, and the percentage can be greater than a predefined threshold. A label of the common class object can be associated with the plurality of input instances.

[0013] However, existing systems lack the configuration or ability to label data describing system states. For example, existing systems for inferring labels for nodes, links, subsets of nodes, link nodes, and triples of nodes, etc. in a knowledge graph can use a classification method to label that data (nodes, links, subsets of nodes, etc.), but such existing labeling methods are not automatic and rely on substantial manual input from human operators. Moreover, such existing systems can be computationally expensive and fail to scale to large systems and can result in imbalanced, inconsistent, and / or ineffective labels, which can negatively impact the performance (e.g., training results) of ML models.

[0014] Accordingly, existing systems can suffer from one or more deficiencies or inefficiencies. SUMMARY

[0015] Embodiments disclosed herein can provide an automatic method for performing labeling of ML data in a cloud system using knowledge inferred from a knowledge graph.

[0016] Embodiments of the present disclosure can provide a system or method that can be used to automatically label input data by running inferences using an existing knowledge graph. For example, input data in an input dataset can be paired with inference result(s) from a knowledge base. Labeling, when done manually by humans, can be an extremely time-consuming and resource-consuming process as in some existing systems. Embodiments of the present disclosure can use the knowledge present in the knowledge base for labeling input data, and by training a machine learning model in conjunction with the knowledge graph, the efficiency of existing systems can be improved (e.g., by utilizing only the relevant part (subset) of the captured knowledge base).

[0017] In some embodiments, the labeled dataset can be used to train a supervised machine learning model (e.g., a feedforward neural network). By this approach, the relevant part(s) of the “knowledge” from the knowledge graph (i.e., answers to a certain limited kind of input) can be transferred to the machine learning model. The machine learning model can then be executed more efficiently in real-time or a constrained environment compared to running inferences on the complete knowledge base. This process can be further enhanced with a graphical processing unit (GPU) or other specialized hardware accelerators.

[0018] In some embodiments of the present disclosure, a subset of the knowledge from the knowledge base can be transferred into a trained machine learning model, such as a supervised machine learning model. The transfer can be achieved by means of presenting the knowledge base with example inputs / questions labeled based on the results. The labeled data can then be used to train the machine learning model by means of conventional training methods.

[0019] In one or more embodiments of the present disclosure, one or more entities or nodes of a wireless communication system can: - train an anomaly detection (AD) model using data (e.g., monitoring metrics, log data, traces, etc.) collected from a managed system; - perform analysis of the collected data and assign each data to a node in a knowledge graph; - in case there is data to be labeled, use the trained AD model and the description of the nodes to identify anomalous nodes; - use the knowledge graph description to analyze the set of candidate anomalous nodes and determine a set of paths linking them; - score the obtained paths using different criteria including the number of anomalous nodes, the anomaly rate of each node, etc.; - select the path with the highest score as a candidate path describing the input data to be labeled; - use the knowledge graph and the collected data to generate a label for the selected path; and / or - checking the consistency and / or validity of the generated labels.

[0020] One or more embodiments of the present disclosure can provide one or more of the following advantages over existing systems: - enabling automatic tagging of data using a knowledge graph, which is an improvement over existing systems that require a large amount of manual (human) input for tagging; - enabling extraction of limited knowledge (i.e., a smaller subset of knowledge) from a knowledge graph that can be efficiently used in real-time, limited resources, and / or distributed environments; - enabling efficient upgrade of distributed deployments by recreating and comparing (a subset of) labels after updates to the knowledge base, reducing the computational and network resources required for upgrade / update; - enabling training on a variety of different unlabeled data sets that can be updated as needed and / or periodically; - the consistency and / or validity of the generated labels can be checked in a variety of different system environments; and / or - the generated labels can be stored in a repository (e.g., a cloud-based server, a cloud-based storage device, a network node storage device, a management node storage device, a database, etc.) to be used in the future to tag other data sets, etc.

[0021] According to a first aspect of the present disclosure, there is provided a management node configured to communicate with a plurality of network nodes in a communication system. The management node can be configured with a knowledge graph comprising a plurality of links for a plurality of nodes in the communication network. The management node is configured to detect abnormal metric data associated with at least one node of the plurality of nodes (e.g., network nodes, WDs, etc.) of the knowledge graph based on a first historical data set comprising metric data representative of a normal state of the plurality of nodes of the knowledge graph. The management node is configured to determine at least one abnormal node (e.g., at least one network node and / or at least one WD, etc.) of the plurality of nodes based on the abnormal metric data. The management node is configured to determine at least one label for the abnormal metric data based on at least one link from the knowledge graph associated with the at least one abnormal node.

[0022] According to one or more embodiments of this aspect, the management node is further configured to train a machine learning (ML) model using the anomaly metric data and the at least one label. According to one or more embodiments of this aspect, the management node is further configured to detect additional anomaly metric data associated with at least one node of the plurality of nodes of the knowledge graph and determine at least one additional label for the additional anomaly metric data based on the trained ML model. According to one or more embodiments of this aspect, determining the at least one label for the anomaly metric data further comprises determining a plurality of candidate paths that traverse the plurality of nodes, each candidate path of the plurality of candidate paths traversing at least one anomalous node of the plurality of nodes, and determining the at least one label based on a metric associated with at least one candidate path of the plurality of candidate paths.

[0023] According to one or more embodiments of this aspect, determining the at least one label for the anomaly metric data further comprises determining a ranking order of the plurality of candidate paths and determining the at least one first label based on at least one corresponding highest-ranked candidate path of the plurality of candidate paths. According to one or more embodiments of this aspect, determining the ranking order of the plurality of candidate paths comprises determining a respective score for each candidate path of the plurality of candidate paths, wherein the respective score is determined based on at least one of a number of anomalous nodes of the respective candidate path, an anomaly rate of the respective candidate path, an anomaly severity of the respective candidate path, and / or a number of anomaly metrics of the respective candidate path.

[0024] According to one or more embodiments of this aspect, the at least one highest-ranked candidate path traverses a first set of nodes, and determining the at least one first label based on the at least one highest-ranked candidate path comprises determining a start node of the first set of nodes based on a hierarchical relationship between the nodes of the first set of nodes, determining a first plurality of anomaly metrics associated with a first number of nodes of the first set of nodes, wherein the first plurality of anomaly metrics comprises a first plurality of metric names and corresponding first plurality of metric values, determining a first keyword based on the first plurality of metric names, and determining the at least one first label based on the first keyword. According to one or more embodiments of this aspect, the management node is further configured to validate the at least one first label based on the first number of nodes associated with the first plurality of anomaly metrics exceeding a preconfigured threshold number of nodes.

[0025] According to one or more embodiments of this aspect, the management node is further configured to: in response to a failure to validate the at least one first label, determine a second candidate path traversing the second set of nodes; determine a second plurality of anomaly metrics associated with a second number of nodes in the second set of nodes, the second plurality of anomaly metrics comprising a second plurality of metric names and corresponding second plurality of metric values; determine a second keyword based on the second plurality of metric names; determine at least one second label based on the second keyword; and validate the at least one second label based on the second number of nodes associated with the second plurality of anomaly metrics exceeding a preconfigured threshold number of nodes.

[0026] According to one or more embodiments of this aspect, the second candidate path is one of a second highest ranked candidate path and a sub-path of at least one highest ranked candidate path among a plurality of candidate paths traversing the second set of nodes, wherein the sub-path traverses the second set of nodes, the second set of nodes is a subset of the first set of nodes, and the second set of nodes comprises at least one dependent node of the start node. According to one or more embodiments of this aspect, the management node is further configured to receive, prior to the detecting step, a first historical dataset comprising at least one of monitoring metrics, traces, and log files associated with the plurality of nodes of the knowledge graph.

[0027] According to one or more embodiments of this aspect, the management node is further configured to receive the knowledge graph from at least one network node in the communication network and to receive the anomaly metric data from at least one network node in the communication network. According to one or more embodiments of this aspect, the management node can be preconfigured with the knowledge graph.

[0028] According to another aspect of the disclosure, there is provided a method implemented in a management node configured to communicate with a plurality of network nodes in a communication system. The management node can be configured with a knowledge graph comprising a plurality of links for a plurality of nodes in the communication network. The method comprises detecting anomaly metric data associated with at least one node of the plurality of nodes (e.g., network nodes, WDs, etc.) of the knowledge graph based on a first historical dataset comprising metric data representative of normal states of the plurality of nodes of the knowledge graph. The method comprises determining at least one anomalous node of the plurality of nodes (e.g., at least one network node and / or at least one WD, etc.) based on the anomaly metric data. The method comprises determining at least one label for the anomaly metric data based on at least one link from the knowledge graph associated with the at least one anomalous node.

[0029] According to one or more embodiments of this aspect, the method further includes training a machine learning (ML) model using the anomaly metric data and the at least one label. According to one or more embodiments of this aspect, the method further includes detecting additional anomaly metric data associated with at least one node of the plurality of nodes of the knowledge graph and determining at least one additional label for the additional anomaly metric data based on the trained ML model. According to one or more embodiments of this aspect, determining the at least one label for the anomaly metric data further includes determining a plurality of candidate paths for traversing the knowledge graph, each candidate path of the plurality of candidate paths traversing at least one anomalous node of the plurality of nodes, and determining the at least one label based on a metric associated with at least one candidate path of the plurality of candidate paths.

[0030] According to one or more embodiments of this aspect, determining the at least one label for the anomaly metric data further includes determining a ranking order of the plurality of candidate paths and determining the at least one first label based on at least one corresponding highest-ranked candidate path of the plurality of candidate paths. According to one or more embodiments of this aspect, determining the ranking order of the plurality of candidate paths includes determining a respective score for each candidate path of the plurality of candidate paths, wherein the respective score is determined based on at least one of a number of anomalous nodes of the respective candidate path, an anomaly rate of the respective candidate path, an anomaly severity of the respective candidate path, and / or a number of anomaly metrics of the respective candidate path.

[0031] According to one or more embodiments of this aspect, the at least one highest-ranked candidate path traverses a first set of nodes, and determining the at least one first label based on the at least one highest-ranked candidate path includes determining a start node of the first set of nodes based on a hierarchical relationship between the nodes of the first set of nodes, determining a first plurality of anomaly metrics associated with a first number of nodes of the first set of nodes, wherein the first plurality of anomaly metrics includes a first plurality of metric names and corresponding first plurality of metric values, determining a first keyword based on the first plurality of metric names, and determining the at least one first label based on the first keyword. According to one or more embodiments of this aspect, the method further includes verifying the at least one first label based on the first number of nodes associated with the first plurality of anomaly metrics exceeding a preconfigured threshold number of nodes.

[0032] According to one or more embodiments of this aspect, the method further includes, in response to a failure to validate the at least one first label, determining a second candidate path traversing the nodes of the second set; determining a second plurality of anomaly metrics associated with a second number of nodes of the nodes of the second set, the second plurality of anomaly metrics including a second plurality of metric names and corresponding second plurality of metric values; determining a second keyword based on the second plurality of metric names; determining at least one second label based on the second keyword; and validating the at least one second label based on the second number of nodes associated with the second plurality of anomaly metrics exceeding a preconfigured threshold number of nodes.

[0033] According to one or more embodiments of this aspect, the second candidate path is one of a second highest ranked candidate path of a plurality of candidate paths traversing the nodes of the second set and a sub-path of at least one highest ranked candidate path, wherein the sub-path traverses the nodes of the second set, the nodes of the second set are a subset of the nodes of the first set, and the nodes of the second set include at least one dependent node of the start node. According to one or more embodiments of this aspect, the method further includes receiving, prior to the detecting step, a first historical dataset including at least one of monitoring metrics, traces, and log files associated with the plurality of nodes of the knowledge graph.

[0034] According to one or more embodiments of this aspect, the method further includes receiving the knowledge graph from at least one network node of the communication network and receiving the anomaly metric data from at least one network node of the communication network. According to one or more embodiments of this aspect, the method further includes preconfiguring the management node with the knowledge graph. BRIEF DESCRIPTION OF DRAWINGS

[0035] A more complete understanding of the present embodiments, and the attendant advantages and features thereof, will be more fully understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein: Figure 1 is a schematic diagram illustrating an example network architecture of a communication system in accordance with the principles disclosed herein; Figure 2 is a block diagram of a network node in communication with a wireless device over a wireless connection in accordance with some embodiments of the disclosure; Figure 3 is a flowchart of an example process for data labeling using a knowledge graph in a management node in accordance with some embodiments of the disclosure; Figure 4 is a flowchart of an example process for data labeling using a knowledge graph in accordance with some embodiments of the disclosure; Figure 5 is a flowchart of another example process for data labeling using a knowledge graph in accordance with some embodiments of the disclosure; is a flowchart of another example process for data labeling using a knowledge graph in accordance with some embodiments of the disclosure;Figure 6 is a flowchart of another example process for data labeling using a knowledge graph in accordance with some embodiments of the present disclosure; Figure 7 is a flowchart of an example process for data labeling using a knowledge graph in accordance with some embodiments of the present disclosure; Figure 8 is an illustration of a knowledge graph for which data is labeled in accordance with some embodiments of the present disclosure; and Figure 9 is an illustration of an example traversal of a path of the knowledge graph of Figure 8 DETAILED DESCRIPTION

[0036] Before describing the example embodiments in detail, it is noted that the embodiments primarily reside in combinations of device components and processing steps related to data labeling based on knowledge graph-based monitoring metrics and / or performance metrics, e.g., for training machine learning models for error and anomaly detection in wired and / or wireless communication networks or other distributed computing systems. Accordingly, components have been represented in the drawings by conventional symbols, where appropriate, showing only those specific details that are pertinent to understanding the embodiments so as not to unnecessarily obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0037] As used herein, relational terms, such as“first” and“second,”“top” and “bottom,” and the like, can be used solely to distinguish one entity or element from another entity or element without necessarily requiring or implying any physical or logical relationship or order between such entities or elements. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms“a,”“an” and“the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms“comprises,”“comprising,”“includes” and / or“including,” as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0038] ​In the embodiments described herein, for example, the joining term "in communication with" and the like can be used to indicate electrical or data communication through physical contact, induction, electromagnetic radiation, radio signaling, infrared signaling or optical signaling. It will be appreciated by those skilled in the art that electrical and data communication can be achieved by interconnecting various components and modifications and variations are possible using the principles set forth herein.

[0039] In some embodiments described herein, the terms "coupled," "connected," and the like, can be used herein to indicate a connection, although not necessarily directly, and can include wired and / or wireless connection.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0041] The term "network node" used herein can be any kind of network node included in a distributed computing system such as a communication network, which can be a wired or wireless communication network. The network node can further comprise any of the following: a base station (BS), a radio base station, a base transceiver station (BTS), a base station controller (BSC), a radio network controller (RNC), a g-NodeB (gNB), an evolved Node B (eNB or eNodeB), a Node B, a multi- standard radio (MSR) radio node such as a MSR BS, a multi-cell / multicast coordination entity (MCE), a relay node, a donor node controlling relay, a radio Access Point (AP), a transmission point, a transmission node, a Remote Radio Unit (RRU) a Remote Radio Head (RRH), a core network node (e.g., a mobility management entity (MME), a self-organizing network (SON) node, a coordinating node, a positioning node, an MDT node, etc.), an external node (e.g., a third party node, a node external to the current network), a node in a distributed antenna system (DAS), a spectrum access system (SAS) node, an element management system (EMS), a physical or virtual database, a cloud-based entity / server / device, or any other node or entity in a distributed computing environment, a communication system, etc. The network node can also comprise a test equipment. The term "radio node" used herein can also be used to refer to a wireless device (WD), such as a wireless device (WD) or a radio network node.

[0042] In some embodiments, the non-limiting term management node is used. The management node can be a network node or can be any type of computer or computing entity (e.g., physical devices / servers / etc., virtual devices / servers / etc., cloud-based devices / servers / etc.) that performs one or more management functions in a distributed computing environment or communication network, such as configuration, training and deployment of configurations, such as machine learning models, for network nodes to use for performing anomaly detection, analyzing data (monitoring / performance metrics associated with / from network nodes) and identifying anomalies, etc.

[0043] In some embodiments, the non-limiting terms wireless device (WD) or user equipment (UE) are used interchangeably. A WD herein can be any type of wireless device capable of communicating with a network node or another WD over radio signals, such as a wireless device (WD). The WD can also be a radio communication device, target device, device-to-device (D2D) WD, machine-type wireless device, machine-to-machine (M2M) communication WD, low-cost and / or low-complexity WD, sensor-equipped WD, tablet, mobile terminal, smart phone, laptop embedded equipped (LEE), laptop mounted equipment (LME), USB dongle, customer premises equipment (CPE), an Internet-of-Things (IoT) device, or a Narrowband IoT (NB-IOT) device, etc.

[0044] Also, in some embodiments, the generic term "radio network node" is used. It can be any kind of radio network node, it can comprise any of: base station, radio base station, base transceiver station, base station controller, network controller, RNC, eNodeB, NodeB, gNB, multi-cell / multicast coordination entity (MCE), relay node, access point, radio access point, remote radio unit (RRU) remote radio head (RRH).

[0045] It is noted that while terminology from one particular wireless system, such as, for example, the 3GPP LTE and / or New Radio (NR), can be used in this disclosure, this does not in any way aim to limit the scope of the disclosure to only the aforementioned system. Other wireless systems, including but not limited to Wideband Code Division Multiple Access (WCDMA), Worldwide Interoperability for Microwave Access (WiMax), Ultra Mobile Broadband (UMB), and Global System for Mobile Communications (GSM), can also benefit from exploiting the ideas covered within this disclosure.

[0046] It is further noted that functions described herein as being performed by a wireless device or a network node can be distributed in a number of wireless devices and / or network nodes. In other words, it is contemplated that the functions of the network nodes and wireless devices described herein are not limited to being performed by a single physical device, and can in fact be distributed among several physical devices.

[0047] It is further noted that the functions described herein can be performed by various distributed computing arrangements including but not limited to a communications network comprising one or more cloud-based nodes, entities, servers, etc.

[0048] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms used herein shall be interpreted as having the same meaning as they have in the context of this specification and the relevant field, and that terms used herein shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0049] Referring to the accompanying drawings, in which the same reference numerals denote the same elements, Figure 1 The diagram illustrates a communication system 10 according to an embodiment, such as a 3GPP-type cellular network that may support standards such as LTE and / or NR (5G). It includes an access network 12, such as a radio access network, and a core network 14. The access network 12 includes multiple network nodes 16a, 16b, 16c (collectively referred to as network nodes 16), such as NBs, eNBs, gNBs, or other types of radio access points. Each network node defines a corresponding coverage area 18a, 18b, 18c (collectively referred to as coverage area 18). Each network node 16a, 16b, 16c can be connected to the core network 14 via a wired or wireless connection 20.

[0050] The system may further include a management node 21, which may be a network node 16, such as a cloud-based node and / or server that provides one or more management node 21 functionalities. The management node 21 may communicate and / or receive information associated with one or more of the network nodes 16, such as performance metrics, via one or more of an access network 12, a core network 14, an Internet connection, etc., and the management node 21 may communicate and / or receive information associated with the network nodes 16 directly and / or via one or more intermediate means.

[0051] A first wireless device (WD) 22a located in coverage area 18a is configured to wirelessly connect to or be paged by a corresponding network node 16a. A second WD 22b in coverage area 18b may wirelessly connect to a corresponding network node 16b. Although multiple WDs 22a, 22b (collectively referred to as wireless devices 22) are illustrated in this example, the disclosed embodiments are equally applicable to situations where only one WD is in the coverage area or only one WD is connected to a corresponding network node 16. Note that although only two WDs 22 and three network nodes 16 are shown for convenience, the communication system may include many more WDs 22 and network nodes 16.

[0052] Also, it is contemplated that a WD 22 can be in simultaneous communication and / or configured to separately communicate with more than one network node 16 and / or more than one type of network node 16. As an example, a WD 22 can have dual connectivity with an LTE capable network node 16 and a same or different network node 16 that is NR capable. As a further example, a WD 22 can be in communication with an eNB for LTE / E-UTRAN and a gNB for NR / NG-RAN.

[0053] The management node 21 is configured to comprise a data labelling unit 24 configured to perform one or more management node 21 functions disclosed herein, such as determining labels for input data sets corresponding to performance metrics of network nodes 16 in the communication system 10 based on a knowledge graph of the communication system 10.

[0054] Example implementations of the management node 21 discussed in the preceding paragraphs will now be described with reference to the following Figure 2

[0055] The communication system 10 comprises a management node 21 provided in the communication system 10 and comprising hardware 28 enabling it to communicate with one or more network nodes 16 and / or WDs 22. The hardware 28 can comprise a communication interface 30 for receiving and / or transmitting information (e.g., data sets corresponding to performance metrics, knowledge graphs and related configurations, machine learning models and related parameters, etc.) to / from one or more network nodes 16 via one or more wired and / or wireless connections.

[0056] In the illustrated embodiment, the hardware 28 of the management node 21 further includes processing circuitry 36. The processing circuitry 36 can include a processor 38 and a memory 40. In particular, the processing circuitry 36 can include integrated circuits, such as one or more processors and / or processor cores and / or FPGAs (Field Programmable Gate Arrays) and / or ASICs (Application Specific Integrated Circuits) adapted to execute instructions, in addition to or instead of a processor such as a central processing unit and / or a graphics processing unit (GPU), and memory, in particular for processing and / or control of the management node 21. The processor 38 can be configured to access (e.g., write to and / or read from) memory 40, which can include any kind of volatile and / or non-volatile memory, e.g., cache and / or buffer memory and / or RAM (Random Access Memory) and / or ROM (Read-Only Memory) and / or optical memory and / or EPROM (Erasable Programmable ROM).

[0057] ​Thus, the management node 21 further has software 42 stored internally in, for example, memory 40 or stored in external memory (e.g., databases, storage arrays, network storage devices, etc.) accessible to the network node 16 via an external connection. The software 42 can be executable by the processing circuitry 36. The processing circuitry 36 can be configured to control any of the methods and / or processes described herein, and / or cause such methods and / or processes to be performed, e.g., by the network node 16. The processor 38 corresponds to one or more processors 38 for performing management node 21 functions described herein. The memory 40 is configured to store data, programmatic software code, and / or other information described herein. In some embodiments, the software 42 can include instructions executable by the processor 38 and / or processing circuitry 36 that, when executed, cause the processor 38 and / or processing circuitry 36 to perform processes described herein with respect to the management node 21. For example, the processing circuitry 36 of the management node 21 can execute instructions to perform the functions of the data labeling unit 24 disclosed herein, such as determining labels for input data sets corresponding to performance metrics of network nodes 16 in the communication system 10 based on, for example, a knowledge graph of the communication system 10 stored in the memory 40 and / or in a separate entity (e.g., a cloud database, a server, etc.) accessible via the communication interface 30.

[0058] The communication system 10 further includes the network node 16 already mentioned. The network node 16 can have similar hardware to the managing node 21. In some embodiments, the network node 16 can be the managing node 21. The network node 16 can include hardware 44. The hardware 44 of some network nodes 16 can include a communication interface 46 configured to provide wired and / or wireless connectivity to one or more other network nodes 16, the managing node 21, WDs 22, and / or the like. The network node 16 can include processing circuitry 48, which can be similar in one or more aspects to the processing circuitry 36 of the managing node 21. In some embodiments, one or more network nodes 16 can have processing circuitry 48 including one or more processors 50 and / or memory 52, which can have substantially limited computing resources as compared to the processing circuitry 36 of the managing node 21 (e.g., processor 50 speed, processor 50 core, memory 52 bandwidth, memory 52 size, and / or the like). In some embodiments, the processing circuitry 48 of the first network node 16a can be configured to report and / or determine performance metrics associated with entities of the communication system 10, such as the network node 16a (or other network nodes 16b, 16c, and / or the like), WDs 22, core network 14 entities, cloud-based entities, other nodes, databases, servers, and / or the like. In some embodiments, the performance metrics associated with the network node 16 can be determined by the network node 16 itself and / or by one or more other entities in the system 10 (e.g., the managing node 21, other network nodes 16, core network 14 entities, cloud-based entities / servers, and / or the like). The network node 16 can further include software 53, which can be similar in one or more aspects to the software 42 of the managing node 21 described herein.

[0059] In some embodiments, the internal workings of the managing node 21 and network nodes 16 can be as shown in Figure 2 and independently, the surrounding network topology can be that of Figure 1 .

[0060] The teachings of some of these embodiments can improve data rates, latency, and / or power consumption and thereby provide benefits such as reduced user latency, relaxed restrictions on file size, better responsiveness, extended battery life, and / or the like. In some embodiments, a measurement procedure can be provided for the purpose of monitoring data rates, latency, and other factors improved by one or more embodiments.

[0061] While Figure 1 and Figure 2Various“units” such as the data labeling unit 24 are shown as being within the respective processor, but it is envisaged that the units can be implemented such that part of the unit is stored in a corresponding memory within the processing circuitry. In other words, the units can be implemented in hardware or a combination of hardware and software within the processing circuitry.

[0062] As disclosed herein, Figure 3 is a flowchart of an example procedure for data labeling performance metrics based on a knowledge graph in the management node 21. One or more blocks described herein can be performed by one or more elements of the management node 21, such as by one or more of the processing circuitry 36 (including the data labeling unit 24), the processor 38, and / or the communication interface 30 performing one or more blocks described herein. The management node can be configured with a knowledge graph comprising a plurality of links for a plurality of nodes in a communication network. The management node 21 is configured to detect (block S100) abnormal metric data associated with at least one node of the plurality of nodes (e.g., network nodes 16, WDs 22, etc.) of the knowledge graph based on a first historical data set comprising metric data representative of normal states of the plurality of nodes of the knowledge graph. The management node 21 is configured to determine (block S102) at least one abnormal node of the plurality of nodes (e.g., at least one network node 16 and / or at least one WD 22, etc.) based on the abnormal metric data. The management node 21 is configured to determine (block S104) at least one label for the abnormal metric data based on at least one link from the knowledge graph associated with the at least one abnormal node.

[0063] In some embodiments, the management node 21 is further configured to train a machine learning (ML) model using the abnormal metric data and the at least one label. In some embodiments, the management node 21 is further configured to detect additional abnormal metric data associated with at least one node of the plurality of nodes of the knowledge graph and determine at least one additional label for the additional abnormal metric data based on the trained ML model. In some embodiments, determining the at least one label for the abnormal metric data further comprises determining a plurality of candidate paths for traversing the knowledge graph, each candidate path of the plurality of candidate paths traversing the at least one abnormal node of the plurality of nodes, and determining the at least one label based on a metric associated with at least one candidate path of the plurality of candidate paths.

[0064] In some embodiments, determining the at least one label of the abnormality metric data further comprises determining a ranking order of the plurality of candidate paths and determining the at least one first label based on at least one corresponding highest-ranked candidate path of the plurality of candidate paths. In some embodiments, determining the ranking order of the plurality of candidate paths comprises determining a respective score of each candidate path of the plurality of candidate paths, wherein the respective score is determined based on at least one of a number of abnormal nodes of the respective candidate path, an abnormality rate of the respective candidate path, an abnormality severity of the respective candidate path, and / or a number of abnormality metrics of the respective candidate path.

[0065] In some embodiments, the at least one highest-ranked candidate path (i.e., the most abnormal path) traverses the nodes of the first set, and determining the at least one first label based on the at least one highest-ranked candidate path comprises: determining a start node of the nodes of the first set based on a hierarchical relationship between the nodes of the first set; determining a first plurality of abnormality metrics associated with a first number of nodes of the nodes of the first set, wherein the first plurality of abnormality metrics comprises a first plurality of metric names and corresponding first plurality of metric values; determining first keywords based on the first plurality of metric names; and determining the at least one first label based on the first keywords. In some embodiments, the management node 21 is further configured to validate the at least one first label based on the first number of nodes associated with the first plurality of abnormality metrics exceeding a preconfigured threshold number of nodes. For example, in some embodiments, to determine that a node is affected by the deviation of those nodes, the threshold can be set to be higher than 50%, higher than 75%, higher than 90%, etc. as appropriate. This threshold can be configurable, e.g., by a network operator, an end user, etc.

[0066] In some embodiments, the management node 21 is further configured to: in response to a failure of validating the at least one first label, determine a second candidate path traversing a second set of nodes; determine a second plurality of abnormality metrics associated with a second number of nodes of the second set of nodes, the second plurality of abnormality metrics comprising a second plurality of metric names and corresponding second plurality of metric values; determine second keywords based on the second plurality of metric names; determine at least one second label based on the second keywords; and validate the at least one second label based on the second number of nodes associated with the second plurality of abnormality metrics exceeding a preconfigured threshold number of nodes.

[0067] In some embodiments, the second candidate path is one of: a second highest ranked candidate path (i.e. the most abnormal candidate path traversing the nodes of the second set) of the plurality of candidate paths and a sub-path of at least one highest ranked candidate path, wherein the sub-path traverses the nodes of the second set, the nodes of the second set are a subset of the nodes of the first set, and the nodes of the second set include at least one dependent node of the start node. In some embodiments, the management node 21 is further configured to receive, prior to the detecting step, a first historical dataset comprising at least one of monitoring metrics, traces and log files associated with a plurality of nodes of the knowledge graph.

[0068] In some embodiments, the management node 21 is further configured to receive the knowledge graph from at least one network node 16 in the communication network and to receive the abnormality metric data from at least one network node 16 in the communication network. In some embodiments, the management node 21 can be pre-configured with the knowledge graph.

[0069] Having described the general process flow of the arrangements of the present disclosure and having provided examples of hardware and software arrangements for implementing the processes and functions of the present disclosure, the following sections provide details and examples of arrangements for data labelling of input data (e.g. performance metrics associated with network nodes 16 in the communication system 10 based on the knowledge graph).

[0070] In some embodiments, an automated solution for labelling data is provided in a cloud-based system (e.g. the management node 21 in the communication system 10) using knowledge extracted from the knowledge graph.

[0071] Some embodiments implement processing of unlabelled data and inferring information and meaning (e.g. labels for machine learning model training) while traversing an input / stored knowledge graph. For example, Figure 4 An example method for labelling data, e.g. as implemented in the management node 21, is illustrated.

[0072] As Figure 4 As illustrated in the example of FIG. 1 1, in some embodiments, the labelling process (e.g. as implemented by the management node 21 ) receives input (block S106), the input comprising a knowledge graph (which can be a data structure representing nodes, links, relationships <node, link> etc. of a knowledge base), historical and / or live data collected from the managed communication system 10 (e.g. monitoring metrics, traces, log files etc. associated with / reporting by network nodes 16, WDs 22, other entities in the communication system 10, access networks 12, core networks 14 etc.). This collected data can also represent a set of unlabelled data that the method labels while processing (reasoning) the input knowledge graph.

[0073] In some embodiments, as explained in Figure 4 the method can determine labels for data (block S108), e.g., based on the knowledge graph, and can output (block S110) a labeled dataset that can include one or more labels (generated by the method), where the labels are associated with one or more of the data entries (received unlabeled in block S106).

[0074] In some embodiments, the labeling process uses historical data (e.g., historical performance metrics associated with one or more network nodes 16 or other entities of the communication system 10) to train an anomaly detection (AD) model to store statistical properties of data characterizing the “normal” (e.g., non-anomalous operation, error-free operation, etc.) case of the communication system 10 (e.g., a cloud system). The labeling process can use the data to assign and / or discover / determine properties characterizing the knowledge graph nodes. The nodes of the knowledge graph can correspond to network nodes 16 and / or other entities of the communication system 10. Given the AD model and a knowledge graph node (e.g., an analyzed node), the labeling system can: - process (e.g., reason) input live data that is generally (e.g., partially, mostly, completely, etc.) unlabeled and map it to the AD model to find anomalous data (e.g., anomalous performance metrics); - map these anomalous performance metrics to the knowledge graph node (e.g., the analyzed node) to find possible / candidate paths, if any, - score / rank the candidate paths and select the most relevant (e.g., highest scored / ranked) path, - generate a label associated with the selected path.

[0075] Figure 5Another example flow diagram for a labeling process for machine learning model training (e.g., by the management node 21) is illustrated in accordance with one or more embodiments of the present disclosure. The management node 21 (or other entity providing labeling functionality) labels input data based at least in part on a knowledge graph (block S112). The obtained labeled dataset can be used to train a machine learning model (block S114), which can be represented by / stored in a data structure corresponding to machine learning model weights, parameters, layers, etc., and can be deployed (block S116) in a communication system 10 (such as a cloud system, an access network 12, a core network 14, one or more network nodes 16, a WD 22, etc.). As compared to existing systems, the trained machine learning model can contain a limited set of knowledge from the knowledge base / knowledge graph, and the trained machine learning model can be used more efficiently, e.g., in a distributed deployment.

[0076] Figure 6 Another example of a labeling process for machine learning model training (e.g., by the management node 21) is illustrated in accordance with one or more embodiments of the present disclosure.

[0077] The management node 21 can receive or be provisioned with (block S128) a knowledge graph containing information about different relationships that exist between objects, and receive (block S130) historical data including historical metrics, performance metrics, configuration logs, traces, etc. In some embodiments, the knowledge graph can be obtained from an external knowledge graph generation model. In some embodiments, the historical data (historical performance metrics) can be collected from a monitoring system or other entity that provides metrics, monitoring information, error information, performance information, etc. In some embodiments, the historical data is used for anomaly detection, which is used to generate labels for live data.

[0078] An anomaly detection (AD) process as implemented by the management node 21 or other entity of the communication system 10 can include training (block S132) a model (e.g., a machine learning model, a statistical model, etc.) that can be configured to identify anomalous metrics in historical metric data. The AD process can be implemented using various anomaly detection techniques. An anomaly can be defined as any system state that falls outside of a “normal” range, where normal can include, e.g., error-free operation, operation with a number of errors below a threshold, operation that does not exhibit security vulnerabilities / threats / vulnerabilities, operation that results in performance metrics that belong to a range specified by a technical standard, operation that satisfies a minimum performance threshold, etc.

[0079] In some embodiments, the process assigns (block S134) historical metric data to one or more nodes in the input knowledge graph (e.g., corresponding to network nodes 16 or other entities of communication system 10).

[0080] In some embodiments, when live data is received (block S136) by management node 21, the trained AD model (block S138) can be configured to identify a list of abnormal metrics (step S140). If such a list is empty, then management node 21 assigns a "normal state" (step S141) (e.g., instead of assigning a fault state or a security threat state) as a label for one or more entries of the live metric data. If the list of abnormal metrics is not empty, then the process uses the list of abnormal metrics (step S140) along with the knowledge graph and node metrics (step S142) to identify abnormal nodes from the knowledge graph (block S144). In other words, given a live data set to be labeled (i.e., an unlabeled data set), the trained AD model, the knowledge graph, and the nodes along with the assigned metrics are used to identify abnormal nodes in the knowledge graph (e.g., abnormal network nodes 16). This is done by matching the abnormal metrics to the metrics assigned to the nodes of the knowledge graph. As described herein, these nodes are classified as abnormal nodes and represent candidate nodes to be used for the labeling process (step S146).

[0081] In some embodiments, possible paths are identified (block S148) among the candidate nodes. The paths can be identified according to various different strategies. For example, an exhaustive search can be performed in which a list(s) of all possible paths are identified. Another example approach can include performing a guided search in which different criteria can be identified to exclude some paths. These paths represent candidate paths. In some embodiments, a candidate path must include at least one abnormal node. In some embodiments, a candidate path must include an abnormal node at the beginning of the path and / or at the end of the path. In some embodiments, a candidate path must include only abnormal nodes. In some embodiments, a candidate path can include a mix of abnormal nodes and non-abnormal nodes.

[0082] The generated candidate paths (step S150) can then be assigned scores (block S152). In various embodiments, the scores can be computed according to various different criteria, such as the number of abnormal metrics in the path, the number of abnormal nodes in the path, the ratio of nodes with abnormal metrics to the total number of nodes in the path, the ratio of nodes with abnormal metrics that exceed some threshold (e.g., the number of abnormal metrics, the amount by which one or more abnormal metrics deviate from a preconfigured normal operating range or value, etc.), weight values assigned to nodes, e.g., based on the type of network node 16 corresponding to the node (e.g., some network nodes 16 can be relatively more critical / important to the normal operation of the communication system 10), where such weights can be preconfigured in the knowledge graph and / or determined by the management node 21, e.g., based on a mapping of network node 16 types and weight values. According to the assigned scores, the candidate paths can be ranked (step S154) and the paths and rankings / scores can be stored in a corresponding data structure.

[0083] In some embodiments, the path with the highest score can be selected for analysis (step S156). As described herein, after the path is selected, the process determines / generates a label corresponding to the selected path (step S158). If no label is determined at this step (block S162), the process can return to step S156 and select the next path with the highest score. If a label is successfully determined, the live data, label, data set, and / or generated label are used to train a machine learning model (block S166), e.g., for future anomaly detection / prediction.

[0084] Thus, in some embodiments, the output of block S160 is labeled input data, e.g., labeled live metrics / performance metrics / monitoring metrics / etc. In some embodiments, the label can be characterized as extracting some knowledge from the knowledge graph for a given input data. Such a set of labeled input data can be used to train an ML model (e.g., a neural network) using various supervised machine learning methods. The ML model can then be deployed in the communication system 10 (e.g., in the cloud), which can be able to handle various inputs (performance metrics / monitoring metrics) for labeling more efficiently as compared to using the knowledge graph itself.

[0085] In some embodiments, the knowledge graph can change over time for various reasons. By regenerating and comparing a subset of labels for previously labeled input data, embodiments of the present disclosure can identify whether a change in the knowledge graph would affect the labels / answers for a given set of input data. If no change is observed, there is no need to retrain or redistribute the ML model. This can advantageously reduce cost, energy, latency, etc. as compared to redistributing the entire knowledge graph after each change.

[0086] Figure 7 is an example process for generating a label for a selected candidate path (e.g., in accordance with one or more embodiments of the present disclosure, Figure 6 is a flowchart of an example process of block S160.

[0087] Figure 7 The process described in the Figure 6 receives input of the selected path (step S172) (e.g., determined in block S156) and identifies (block S174) the nodes (e.g., corresponding to network nodes 16 and / or other entities of the communication system 10) forming the path and the assigned metrics (e.g., metric names, corresponding values, etc.) of the nodes (step S176).

[0088] The node to start the analysis is identified (block S178). The identification of the starting node or the most relevant node can be performed, for example, using rules that can be stored in a rules repository (e.g., a cloud-based storage accessible by the management node 21, or stored in the memory 40 of the management node 21). For example, the rules can include a hierarchical ordering of nodes that can specify to start the analysis with the underlying physical nodes, then the slave nodes, and so on. Another example rule in the context of a multi-layer cloud stack communication system 10 can include, for example, starting the analysis with the bare metal layer, then the virtualization layer (e.g., OpenStack), and then the container layer (e.g., Kubernetes), then the cloud-native network functions (CNFs). Such rules can be provided as input, for example, by a user, an administrator, or by an external system. In some embodiments, the knowledge graph can contain relationships that can be used to define various different hierarchies.

[0089] The relevant nodes / metrics are identified (step S182), and the metrics of the relevant nodes that deviate beyond a specified threshold (e.g., threshold 1) are identified (block S184). This threshold can be given by an end user or administrator and / or can be pre-configured and can be updated, for example, based on the measurement accuracy of the labeling system.

[0090] After identifying the list of metrics that deviate beyond the threshold (step S186), keyword analysis is performed (block S188). In some embodiments, the keyword analysis performs a word2vec computation that determines all combinations of these tokens and the frequency of the most frequently occurring tokens.

[0091] The list of keywords with their corresponding frequencies can be analyzed (step S190), and the most frequent keyword(s) / word(s) are determined (block S192), which can represent the candidate label (step S194).

[0092] In some embodiments, the candidate label can be validated, e.g., by analyzing one or more other nodes in the selected path to assess whether their same metric (e.g., candidate label) deviates beyond a specified threshold (block S196).

[0093] For example, if the percentage of nodes with deviating metrics is greater than a threshold (i.e., threshold 2) (block S198), the candidate label can be considered validated (block S200). For example, if 50% of the nodes in the selected path are affected by the same metric deviation, the candidate label will be validated for that path. The validated label is output (step S202), e.g., for use in training a machine learning model using the labeled dataset.

[0094] In this example, if the percentage of nodes with deviating metrics is less than the identified threshold (i.e., threshold 2), the process selects all nodes of the relevant node (identified in step 2) to validate the candidate label (block S204).

[0095] For example, the nodes can be analyzed to assess whether they are affected by the same metric (keyword) (block S206). If these metrics deviate beyond a specified threshold (e.g., threshold 3 (note that threshold 1, threshold 2, and / or threshold 3 can have the same or different values) (block S208), the candidate label is validated (block S200), and the process outputs the validated label (step S202).

[0096] Otherwise, if there are still nodes in the path that have not been analyzed (block S210), the process returns to block S178. It starts analyzing the next level of nodes according to the rules in the rules repository. Once all nodes in the path have been analyzed and no candidate label is found, the method returns "no label" (step S212). In some embodiments, the process only checks dependent nodes (e.g., of the start node), and if the same metric deviates beyond a threshold, the label is validated; if not, the process continues analyzing the next higher level of nodes, e.g., according to the rules repository.

[0097] Figure 8 An example knowledge graph 54 according to one or more embodiments of the present disclosure is illustrated. In this example, the knowledge graph 54 consists of 10 nodes (e.g., representing network nodes 16 of the communication system 10), which are labeled as: LB1 / 2: Loadbalancer 1 / 2, FE1 / 2: FrontEnd 1 / 2, BE1 / 2: BackEnd 1 / 2, DB1: DataBase 1, and S1 / 2 / 3: Server 1 / 2 / 3.

[0098] Figure 8 The arrows in correspond to relationships (edges) between nodes. The directionality of these arrows corresponds to hierarchical information that can be stored in the knowledge graph 54 and / or a separate rules repository (e.g., LB1 is higher in the hierarchy than FE1 and FE2).

[0099] Each node of the knowledge graph 54 can be characterized by various different metrics and their statistical properties / values. For example, performance / monitoring metrics data (data sets) can be a collection of time series data.

[0100] By comparing data to trained AD models and mapping data to the knowledge graph 54, embodiments of the present disclosure can identify, for example, the following paths: “LB1, FE1, LB2, BE1, S1”, “LB1, FE1, LB2, BE2, DB1, S3”,

[0101] Referring to Figure 8 , in one example, BE1 depends on S1; therefore, if S1 is not working, it can be assumed that BE1 will also not be functioning properly, e.g., based on rules, relationships, or other configuration information. These rules and relationships can be specified in the rules repository and / or the knowledge graph 54.

[0102] In some embodiments, an exhaustive search can be performed, where candidate paths can include both candidate nodes and non-candidate nodes. In embodiments, a guided search can be performed, where only paths that include only candidate nodes are searched. Other embodiments can utilize various search techniques, e.g., only search for paths that include at least a threshold number of nodes (e.g., 50% of the total nodes), which are candidate nodes and / or abnormal nodes, etc.

[0103] In the example of Figure 8 , at a first time (time 1), the determined path is LB1, FE1, LB2, BE1, and S1, and based on this, the label node_memoryfault is determined. At a second time (time 2), the determined path is LB1, FE1, LB2, BE2, DB1, and S3, and based on this, the label HDD fault is determined. At a third time (time 3), the determined path for the data set is LB1, FE2, LB2, BE2, DB1, and based on this, the label DB fault is determined for the data set at time 3.

[0104] Figure 9 An example path and node metrics corresponding to time 1 of Figure 8 are illustrated. For example, at time 1, server 1 (S1) can be identified by the management node 21 as a relevant node to analyze in this path. Next, the metric data is analyzed to determine, for example, Figure 9The metrics in bold font are metrics that have deviated beyond a preconfigured threshold (e.g., Threshold 1) of 10%. Finally, the extracted metric data is processed and a label "node_memory fault" is determined in the context of the fault management process (where "node_memory" is determined to be the most frequent token in this example). In some embodiments, the label can be related to a problem other than a fault. For example, high load can be received at LB1 and LB2. For example, one metric that can be an indication of high load can be "user_request_count". Here, high load is an example of a label that is not necessarily related to a fault. Figure 7

[0105] As those skilled in the art will appreciate, the concepts described herein can be embodied as a method, data processing system, computer program product, and / or computer storage media storing executable computer program code. Accordingly, the concepts described herein can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a "circuit" or "module". Any process, step, action and / or functionality described herein can be performed by, and / or associated to, a corresponding module, which can be implemented in software and / or firmware and / or hardware. Furthermore, the disclosure can take the form of a computer program product on a tangible computer usable storage medium having computer program code embodied in the medium that can be executed by a computer. Any suitable tangible computer readable medium can be utilized including a hard disk, CD-ROM, electronic storage, optical storage or magnetic storage.

[0106] Some embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems and computer program products. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0107] These computer program instructions can also be stored in a computer readable memory or storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.​

[0108] Computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0109] It is to be understood that the functions / acts noted in the blocks can occur out of the order noted in the operational illustrations. For example, two blocks noted in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality / acts involved. Although some of the diagrams can include arrows on communication paths to show the primary direction of communication, it is to be understood that communication can occur in the opposite direction to the arrows.

[0110] Computer program code for carrying out operations of the concepts described herein can be written in an object oriented programming language such as Python, Java®, or C++. However, the computer program code for carrying out operations of the disclosure can also be written in a conventional procedural programming language, such as the "C" programming language. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer. In the latter scenario, the remote computer can be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0111] Many different embodiments have been disclosed herein in connection with the above description and the drawings. It will be understood that each of the embodiments described and illustrated herein can be combined with one another in any manner, and the embodiments of the present specification should be construed in such manner that the written description and illustrations used herein serve as a complete description of all such embodiments and the manner and process of making and using them, and that the present specification and drawings should be construed to support claims to all such combinations and sub-combinations.

[0112] Abbreviations that can be used in the preceding description include: ML machine learning GPU graphics processing unit AD anomaly detection LB load balancer FE front end BE back end Those skilled in the art will appreciate that the embodiments described herein are not limited to what has been particularly shown and described herein above. In addition, unless mentioned contrary, it is noted that all percentages (%) herein are by weight. Furthermore, unless otherwise indicated, it should be understood that the drawings are not to scale, the specific embodiments are described herein in order to provide a thorough understanding of the application. It will be readily apparent to one skilled in the art that various modifications and changes can be made to the embodiments described herein without departing from the scope and spirit of the application.

Claims

1. A management node (21) in a communication network, the management node (21) being configured with a knowledge graph including multiple links for multiple nodes (16) in the communication network, the management node (21) including processing circuitry 36 configured to: Based on a first historical dataset including metric data representing the normal state of the plurality of nodes (16) of the knowledge graph, abnormal metric data associated with at least one of the plurality of nodes (16) of the knowledge graph is detected. Based on the anomaly measurement data, at least one abnormal node (16) among the plurality of nodes (16) is identified; and At least one label of the anomaly measurement data is determined based on at least one link from the knowledge graph associated with the at least one anomaly node (16).

2. The management node (21) as claimed in claim 1, wherein the processing circuit (36) is further configured to: The machine learning (ML) model is trained using the anomaly metric data and the at least one label.

3. The management node (21) as claimed in claim 2, wherein the processing circuit (36) is further configured to: Detect additional anomalous metric data associated with at least one of the plurality of nodes (16) of the knowledge graph; and At least one additional label is determined based on the trained ML model for the additional anomaly measurement data.

4. The management node (21) as described in any one of claims 1-3, wherein determining the at least one label of the abnormal measurement data further comprises: Determine multiple candidate paths for traversing the knowledge graph, each of the multiple candidate paths traversing at least one abnormal node (16) among the multiple nodes (16). as well as The at least one label is determined based on a metric associated with at least one of the plurality of candidate paths.

5. The management node (21) as claimed in claim 4, wherein determining the at least one label of the abnormal measurement data further comprises: Determine the ranking order of the multiple candidate paths; as well as At least one first label is determined based on the highest-ranking candidate path among the plurality of candidate paths.

6. The management node (21) as described in claim 5, wherein determining the ranking order of the plurality of candidate paths includes: Determine a score for each of the plurality of candidate paths, the score being based on at least one of the following: The number of abnormal nodes (16) in the corresponding candidate path; The anomaly rate of the corresponding candidate path; The severity of the anomaly in the corresponding candidate path; and The number of anomaly metrics for the corresponding candidate paths.

7. The management node (21) as described in any one of claims 5 and 6, wherein the at least one highest-ranking candidate path traverses the nodes (16) of the first set. as well as Determining the at least one first label based on the at least one highest-ranked candidate path includes: The starting node (16) in the first set is determined based on the hierarchical relationship between the nodes (16) of the first set. Determine a first plurality of anomaly metrics associated with a first number of nodes (16) in the first set of nodes (16), the first plurality of anomaly metrics including a first plurality of metric names and corresponding first plurality of metric values; The first keyword is determined based on the first plurality of metric names; and The at least one first tag is determined based on the first keyword.

8. The management node (21) of claim 7, wherein the processing circuit (36) is further configured to: The at least one first label is verified based on the number of nodes (16) associated with the first plurality of anomaly metrics that exceeds a pre-configured threshold number of nodes (16).

9. The management node (21) of claim 8, wherein the processing circuitry (36) is further configured to: In response to the failure to verify at least one first label, a second candidate path is determined for traversing the nodes (16) of the second set; Determine a second plurality of anomaly metrics associated with a second number of nodes (16) in the second set of nodes (16), the second plurality of anomaly metrics including a second plurality of metric names and corresponding second plurality of metric values; The second keyword is determined based on the second plurality of metric names; At least one second tag is determined based on the second keyword; as well as The at least one second label is verified based on the number of nodes (16) associated with the second plurality of anomaly metrics exceeding the pre-configured threshold number of nodes (16).

10. The management node (21) as claimed in claim 9, wherein the second candidate path is one of the following: Iterate through the nodes (16) of the second set and select the second-highest ranked candidate path from among the plurality of candidate paths; and The sub-path of the at least one highest-ranked candidate path, the sub-path traversing the nodes (16) of the second set, the nodes (16) of the second set being a subset of the nodes (16) of the first set, the nodes (16) of the second set including at least one subordinate node (16) of the starting node (16).

11. The management node (21) as claimed in any one of claims 1-10, wherein the processing circuit (36) is further configured to: Prior to the detection of the abnormal metric data, the first historical dataset is received, which includes at least one of the monitoring metrics, traces, and log files associated with the plurality of nodes (16) of the knowledge graph.

12. The management node (21) as claimed in any one of claims 1-11, wherein the processing circuit (36) is further configured to be at least one of the following: Receive the knowledge graph from at least one network node (16) in the communication network; and The anomaly measurement data is received from at least one network node (16) in the communication network.

13. A method implemented in a management node (21) in a communication network, the management node (21) being configured with a knowledge graph including multiple links for multiple nodes (16) in the communication network, the method comprising: Based on a first historical dataset including metric data representing the normal state of the plurality of nodes (16) of the knowledge graph, abnormal metric data associated with at least one of the plurality of nodes (16) of the knowledge graph is detected. Based on the anomaly measurement data, at least one abnormal node (16) among the plurality of nodes (16) is identified; and At least one label of the anomaly measurement data is determined based on at least one link from the knowledge graph associated with the at least one anomaly node (16).

14. The method of claim 13, wherein the method further comprises: The machine learning (ML) model is trained using the anomaly metric data and the at least one label.

15. The method of claim 14, wherein the method further comprises: Detect additional anomalous metric data associated with at least one of the plurality of nodes (16) of the knowledge graph; as well as At least one additional label is determined based on the trained ML model for the additional anomaly measurement data.

16. The method of any one of claims 13-15, wherein determining the at least one label of the anomalous measurement data further comprises: Determine multiple candidate paths for traversing the knowledge graph, each of the multiple candidate paths traversing at least one abnormal node (16) among the multiple nodes (16). as well as The at least one label is determined based on a metric associated with at least one of the plurality of candidate paths.

17. The method of claim 16, wherein determining the at least one label of the anomalous measurement data further comprises: Determine the ranking order of the multiple candidate paths; as well as At least one first label is determined based on the highest-ranking candidate path among the plurality of candidate paths.

18. The method of claim 17, wherein determining the ranking order of the plurality of candidate paths comprises: Determine a score for each of the plurality of candidate paths, the score being based on at least one of the following: The number of abnormal nodes (16) in the corresponding candidate path; The anomaly rate of the corresponding candidate path; The severity of the anomaly in the corresponding candidate path; and The number of anomaly metrics for the corresponding candidate paths.

19. The method of any one of claims 17 and 18, wherein the at least one highest-ranked candidate path traverses the nodes of the first set (16). as well as Determining the at least one first label based on the at least one highest-ranked candidate path includes: The starting node (16) in the first set is determined based on the hierarchical relationship between the nodes (16) of the first set. Determine a first plurality of anomaly metrics associated with a first number of nodes (16) in the first set of nodes (16), the first plurality of anomaly metrics including a first plurality of metric names and corresponding first plurality of metric values; The first keyword is determined based on the first plurality of metric names; and The at least one first tag is determined based on the first keyword.

20. The method of claim 19, wherein the method further comprises: The at least one first label is verified based on the number of nodes (16) associated with the first plurality of anomaly metrics that exceeds a pre-configured threshold number of nodes (16).

21. The method of claim 20, wherein the method further comprises: In response to the failure to verify at least one first label, a second candidate path is determined for traversing the nodes (16) of the second set; Determine a second plurality of anomaly metrics associated with a second number of nodes (16) in the second set of nodes (16), the second plurality of anomaly metrics including a second plurality of metric names and corresponding second plurality of metric values; The second keyword is determined based on the second plurality of metric names; At least one second tag is determined based on the second keyword; as well as The at least one second label is verified based on the number of nodes (16) associated with the second plurality of anomaly metrics exceeding the pre-configured threshold number of nodes (16).

22. The method of claim 21, wherein the second candidate path is one of the following: Iterate through the nodes (16) of the second set and select the second-highest ranked candidate path from among the plurality of candidate paths; and The sub-path of the at least one highest-ranked candidate path, the sub-path traversing the nodes (16) of the second set, the nodes (16) of the second set being a subset of the nodes (16) of the first set, the nodes (16) of the second set including at least one subordinate node (16) of the starting node (16).

23. The method of any one of claims 13-22, wherein the method further comprises: Prior to the detection of the abnormal metric data, the first historical dataset is received, which includes at least one of the monitoring metrics, traces, and log files associated with the plurality of nodes (16) of the knowledge graph.

24. The method of any one of claims 13-23, wherein the method further comprises at least one of the following: Receive the knowledge graph from at least one network node (16) in the communication network; and The anomaly measurement data is received from at least one network node (16) in the communication network.

Images