Root cause analysis and automation using machine learning

Abstract

A method for discovering and diagnosing network anomalies. The method includes receiving key performance indicator (KPI) data and warning data. The method includes extracting features based on samples obtained by discretizing the KPI data and the warning data. The method includes generating a set of rules based on the features. The method includes identifying the sample as a normal sample or an abnormal sample. In response to identifying the sample as an abnormal sample, the method includes identifying a first rule corresponding to the sample, wherein the first rule indicates a sign and a root cause of the anomaly included in the sample. The method also includes applying the root cause to derive a root cause explanation of the anomaly and performing a corrective action based on the first rule to resolve the anomaly.

Description

Technical Field

[0001] The present disclosure generally relates to machine learning-based root cause analysis of anomalies in cellular networks. More specifically, the present disclosure relates to discovering, diagnosing, and recovering from network anomalies. Background Art

[0002] The number of subscribers to wireless communication services such as cellular networks recently exceeded 5 billion and continues to grow rapidly. Due to the increasing popularity of smartphones and other mobile data devices (such as tablet computers, "notepad" computers, netbooks, e-book readers, and machine-type devices) among consumers and businesses, the demand for wireless data traffic is increasing rapidly.

[0003] The explosive demand for mobile data traffic has created significant operational challenges for mobile network operators given limited bandwidth and infrastructure. Any changes in network conditions, such as higher RF interference, can negatively impact the user experience, such as increased pauses when streaming media content. Therefore, to improve the user experience, service providers must quickly detect anomalies, reveal the underlying root cause of the anomaly, and apply remedial actions. Cellular networks typically require a significant amount of time and manpower to detect and fix network anomalies. This inevitably leads to extended network outages and reduced quality of experience for end users. Summary of the invention

[0004] Technical issues

[0005] The present disclosure provides root cause analysis and automation using machine learning.

[0006] Problem Solution

[0007] In one embodiment, a device for discovering and diagnosing network anomalies is provided. The device includes a communication interface and a processor. The communication interface is configured to receive key performance indicator (KPI) data and warning data. The processor is configured to extract features based on samples obtained by discretizing KPI data and warning data. The processor is configured to generate a set of rules based on the features, wherein a portion of the sample that satisfies the rules corresponds to the anomaly. The processor is configured to identify a sample in the sample as a normal sample or an abnormal sample based on the KPI data and the warning data. In response to identifying the sample as an abnormal sample, the processor is configured to identify a first rule corresponding to the sample, wherein the first rule indicates the signs and root causes of the anomaly included in the sample. The processor is configured to identify the application root cause to derive the root cause explanation of the anomaly based on the KPI associated with the signs and root causes of the anomaly. The processor is configured to perform a corrective action based on the first rule to resolve the anomaly.

[0008] In another embodiment, a method is provided. The method includes receiving KPI data and warning data. The method includes extracting features based on samples obtained by discretizing the KPI data and warning data. The method includes generating a set of rules based on the features, wherein a portion of the sample that satisfies the rules corresponds to an anomaly. The method includes identifying the sample as a normal sample or an abnormal sample. In response to identifying the sample as an abnormal sample, the method includes identifying a first rule corresponding to the sample. The first rule indicates the signs and root causes of the anomaly contained in the sample. The method includes applying the root cause to derive a root cause explanation of the anomaly based on the KPI associated with the signs and root causes of the anomaly. The method includes performing a corrective action based on the first rule to resolve the anomaly.

[0009] Other technical features may be easily apparent to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0011] Figure 1 An example computing system according to various embodiments of the present disclosure is shown;

[0012] Figure 2a-2b and 3a-3b illustrate example devices in a computing system according to various embodiments of the present disclosure;

[0013] Figure 4a A root cause analysis framework for using machine learning according to various embodiments of the present disclosure is shown;

[0014] Figure 4b A block diagram of traffic prediction based on anomaly detection according to various embodiments of the present disclosure is shown;

[0015] Figure 4c A block diagram of performing traffic prediction based on abnormal prediction according to various embodiments of the present disclosure is shown;

[0016] Figure 4d An example framework of a quantile regression forest according to various embodiments of the present disclosure is shown;

[0017] Figure 5a A block diagram for selecting a discretization parameter threshold according to various embodiments of the present disclosure is shown;

[0018] Figure 5b shows an example KPI tree hierarchy according to various embodiments of the present disclosure;

[0019] Figure 5cA flowchart of constructing a KPI tree hierarchy according to various embodiments of the present disclosure is shown;

[0020] Figure 5d shows an example output of a KPI tree hierarchy according to various embodiments of the present disclosure;

[0021] Figure 5e shows a diagram for correlating key quality indicator anomalies with warning data according to various embodiments of the present disclosure;

[0022] Figure 5f A diagram illustrating a time collation of warning and PM data according to various embodiments of the present disclosure;

[0023] Figure 5g A flow chart for sorting regularization and joint processing of warning and PM data using historical data according to various embodiments of the present disclosure is shown;

[0024] Figure 5h A flow chart for real-time sorting regularization and joint processing of warning and PM data according to various embodiments of the present disclosure is shown;

[0025] Figure 6 A process for generating root cause explanations according to various embodiments of the present disclosure is shown;

[0026] Figure 7 shows an example decision tree for root cause analysis associated with a KQI anomaly according to various embodiments of the present disclosure; and

[0027] Figure 8 An example method for discovering and diagnosing network anomalies according to various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Before proceeding to the following detailed description, it may be advantageous to clarify the definitions of certain words and phrases used in this patent document. The term "connection" and its derivatives refer to any direct or indirect communication between two or more elements, whether or not these elements are in physical contact with each other. The terms "transmission", "reception" and "communication" and their derivatives include direct and indirect communication. The terms "include" and "include" and their derivatives mean including but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with..." and its derivatives mean including, included in, interconnected with, included in, connected to, or connected with, connected to, or connected with, can communicate with, collaborate with, crosslink, juxtapose, approach, bind to, or bind with, have, have the property of, have a relationship with, etc. The term "controller" refers to any device, system or part thereof that controls at least one operation. Such a controller can be implemented with hardware or a combination of hardware and software and / or firmware. The functions associated with any particular controller can be centralized or distributed, whether locally or remotely. The phrase "at least one of" when used with a list of items means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C; A and B; A and C; B and C; and A and B and C.

[0029] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a portion thereof that are suitable for implementation with a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and overwrite it later, such as rewritable optical disks or erasable memory devices.

[0030] Definitions for certain other words and phrases are provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most, instances, such definitions apply to prior and future uses of such defined words and phrases.

[0031] Discussed below Figures 1 to 8 The various embodiments used to describe the principles of the present disclosure in this patent document are only for illustration and should not be interpreted in any way as limiting the scope of the present disclosure. Those skilled in the art will appreciate that the principles of the present disclosure can be implemented in any appropriately arranged wireless communication system.

[0032] Since the deployment of 4G communication systems, data traffic has increased. In order to meet the demand for wireless data traffic, efforts have been made to develop improved 5G or pre-5G (pre-5G) communication systems. Therefore, 5G or pre-5G communication systems are also referred to as "beyond 4G networks" or "post-LTE systems."

[0033] 5G communication systems are considered to be implemented in higher frequency (millimeter wave (mmWave), such as 60GHz band) bands to achieve higher data rates. In order to reduce the propagation loss of radio waves and increase the transmission distance, beamforming, massive multiple-input multiple-output (MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, and massive antenna technologies can be used in 5G communication systems.

[0034] Furthermore, in the 5G communication system, development of improvements to the system network based on advanced small cells, cloud radio access networks (RAN), ultra-dense networks, device-to-device (D2D) communication, wireless backhaul, mobile networks, cooperative communications, coordinated multipoint (CoMP), receiving-end interference elimination, etc. is ongoing. In the 5G system, hybrid FSK and QAM modulation (FQAM) and sliding window superposition coding (SWSC) have been developed as advanced coded modulation (ACM), as well as filter bank multi-carrier (FBMC), non-orthogonal multiple access (NOMA), and sparse code multiple access (SCMA) as advanced access technologies.

[0035] Figure 1 An example wireless network 100 according to the present disclosure is shown. Figure 1 The embodiment of the wireless network 100 shown in FIG. 1 is for illustration only. Other embodiments of the wireless network 100 may be used without departing from the scope of the present disclosure.

[0036] Wireless network 100 includes eNodeB (eNB) 101, eNB 102, and eNB 103. eNB 101 communicates with eNB 102 and eNB 103. eNB 101 also communicates with at least one Internet Protocol (IP) network 130 (such as the Internet, a private IP network, or other data network). In some embodiments, wireless network 100 includes a server that maintains the eNB.

[0037] Depending on the network type, the term "base station" or "BS" may refer to any component (or collection of components) configured to provide wireless access to a network, such as a transmission point (TP), a transmission reception point (TRP), an enhanced base station (eNodeB or eNB or gNB), a macro cell, a femto cell, a WiFi access point (AP), or other wireless-enabled devices. A base station may provide wireless access according to one or more wireless communication protocols, for example, 5G 3GPP New Radio Interface / Access (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), High Speed ​​Packet Access (HSPA), Wi-Fi 802.11a / b / g / n / ac, etc. For convenience, in this patent document, the terms "BS" and "TRP" may be used interchangeably to refer to network infrastructure components that provide wireless access to remote terminals. In addition, depending on the network type, the term "user equipment" or "UE" may refer to any component such as a "mobile station", "subscriber station", "remote terminal", "wireless terminal", "reception point", or "user equipment". For convenience, the terms "user equipment" and "UE" used in this patent document refer to a remote wireless device that wirelessly accesses a BS, whether the UE is a mobile device (such as a mobile phone or smartphone) or a device generally considered to be a fixed device (such as a desktop computer or a vending machine).

[0038] The eNB 102 provides wireless broadband access to a network 130 for a first plurality of user equipments (UEs) within a coverage area 120 of the eNB 102. The first plurality of UEs includes a UE 111, which may be located in a small business (SB); a UE 112, which may be located in an enterprise (E); a UE 113, which may be located in a WiFi hotspot (HS); a UE 114, which may be located in a first residence (R); a UE 115, which may be located in a second residence (R); and a UE 116, which may be a mobile device (M) such as a cellular phone, a wireless laptop, a wireless PDA, a server, etc. The eNB 103 provides wireless broadband access to a network 130 for a second plurality of UEs within a coverage area 125 of the eNB 103. The second plurality of UEs includes UE 115 and UE 116. In some embodiments, one or more of the eNBs 101-103 may communicate with each other and with the UEs 111-116 using 5G, long term evolution (LTE), LTE-A, WiMAX, or other advanced wireless communication technologies.

[0039] Dashed lines illustrate the approximate extents of coverage areas 120 and 125, which are shown as generally circular for purposes of illustration and explanation only. It should be clearly understood that coverage areas associated with eNBs, such as coverage areas 120 and 125, may have other shapes, including irregular shapes, depending on the configuration of the eNB and variations in the radio environment associated with natural and man-made obstacles.

[0040] As described in more detail below, cellular networks use a lot of time and manpower to detect and repair various network anomalies. Therefore, human intervention may cause network outages to last longer and reduce the quality of experience for end users. Embodiments of the present disclosure provide analytical methods for efficiently and accurately automating the discovery and diagnosis of network anomalies. For example, embodiments of the present disclosure describe a process for using machine learning to discover network anomalies and reveal the root causes of the anomalies.

[0041] although Figure 1 One example of a wireless network 100 is shown, but may be Figure 1Various changes may be made. For example, the wireless network 100 may include any number of eNBs and any number of UEs in any suitable arrangement. In addition, the eNB 101 may communicate directly with any number of UEs and provide wireless broadband access to the network 130 for these UEs. Similarly, each eNB 102-103 may communicate directly with the network 130 and provide direct wireless broadband access to the network 130 to the UEs. In addition, the eNBs 101, 102, and / or 103 may provide access to other or additional external networks, such as external telephone networks or other types of data networks. In addition, the eNBs 101-103 may transmit data indicating the status of the network, such as warnings and key performance indicators, to the server. The server may detect and diagnose problems in the network and provide instructions on which corrective actions to perform based on the detected and diagnosed problems.

[0042] Figure 2a and 2b Example wireless transmit and receive paths according to the present disclosure are shown. In the following description, transmit path 200 may be described as being implemented in an eNB (such as eNB 102) and receive path 250 may be described as being implemented in a UE (such as UE 116). However, it should be understood that receive path 250 may be implemented in an eNB and transmit path 200 may be implemented in a UE.

[0043] The transmit path 200 includes a channel coding and modulation block 205, a serial to parallel (S-to-P) block 210, an inverse fast Fourier transform (IFFT) block of size N 215, a parallel to serial (P-to-S) block 220, an add cyclic prefix block 225, and an upconverter (UC) 230. The receive path 250 includes a downconverter (DC) 255, a remove cyclic prefix block 260, a serial to parallel (S-to-P) block 265, a fast Fourier transform (FFT) block of size N 270, a parallel to serial (P-to-S) block 275, and a channel decoding and demodulation block 280.

[0044] In the transmit path 200, the channel coding and modulation block 205 receives a set of information bits, applies coding (such as low-density parity check (LDPC) coding), and modulates the input bits (such as using quadrature phase shift keying (QPSK) or quadrature amplitude modulation (QAM)) to generate a frequency domain modulation symbol sequence. The serial to parallel block 210 converts (e.g., demultiplexes) the serial modulation symbols into parallel data to generate N parallel symbol streams, where N is the IFFT / FFT size used in the eNB 102 and the UE 116. The size-N IFFT block 215 performs an IFFT operation on the N parallel symbol streams to generate a time domain output signal. The parallel to serial block 220 converts (e.g., multiplexes) the parallel time domain output symbols from the size-N IFFT block 215 to generate a serial time domain signal. The add cyclic prefix block 225 inserts a cyclic prefix into the time domain signal. The up-converter 230 modulates (e.g., up-converts) the output of the add cyclic prefix block 225 to an RF frequency for transmission over a wireless channel. The signal may also be filtered at baseband before conversion to RF frequency.

[0045] The transmission RF signal from the eNB 102 arrives at the UE 116 after passing through the wireless channel, and the reverse operation of the operation at the eNB 102 is performed at the UE 116. The down converter 255 downconverts the received signal to the baseband frequency, and the cyclic prefix removal block 260 removes the cyclic prefix to generate a serial time-domain baseband signal. The serial-to-parallel block 265 converts the time-domain baseband signal into a parallel time-domain signal. The size-N FFT block 270 performs an FFT algorithm to generate N parallel frequency-domain signals. The parallel-to-serial block 275 converts the parallel frequency-domain signals into a sequence of modulated data symbols. The channel decoding and demodulation block 280 demodulates and decodes the modulation symbols to recover the original input data stream.

[0046] Each of the eNBs 101-103 may implement a transmit path 200 similar to that for transmitting in the downlink to the UEs 111-116 and may implement a receive path 250 similar to that for receiving in the uplink from the UEs 111-116. Similarly, each of the UEs 111-116 may implement a transmit path 200 for transmitting in the uplink to the eNBs 101-103 and may implement a receive path 250 for receiving in the downlink from the eNBs 101-103.

[0047] Figure 2a and 2b Each component in may be implemented using hardware only or a combination of hardware and software / firmware. As a specific example, Figure 2a and 2bAt least some components in can be implemented in software, while other components can be implemented by configurable hardware or a mixture of software and configurable hardware. For example, FFT block 270 and IFFT block 215 can be implemented as configurable software algorithms, wherein the value of size N can be modified according to the implementation.

[0048] In addition, although described as using FFT and IFFT, this is only for illustration and should not be interpreted as limiting the scope of the present disclosure. Other types of transforms can be used, such as discrete Fourier transform (DFT) and inverse discrete Fourier transform (IDFT) functions. It should be understood that for DFT and IDFT functions, the value of variable N can be any integer (such as 1, 2, 3, 4, etc.), and for FFT and IFFT functions, the value of variable N can be any integer that is a power of 2 (such as 1, 2, 4, 8, 16, etc.).

[0049] although Figure 2a and 2b Examples of wireless transmission and reception paths are shown, but Figure 2a and 2b Make various changes. For example, Figure 2a and 2b The various components in may be combined, further subdivided, or omitted, and additional components may be added according to specific needs. Figure 2a and 2b It is intended to illustrate examples of the types of transmission and reception paths that may be used in a wireless network. Any other suitable architecture may be used to support wireless communications in a wireless network.

[0050] Figure 3a An example UE 116 according to the present disclosure is shown. Figure 3a The embodiment of UE 116 shown in FIG. 1 is for illustration only, and Figure 1 UEs 111-115 may have the same or similar configurations. However, UEs may have various configurations, and Figure 3a The scope of the present disclosure is not limited to any particular implementation of the UE.

[0051] In some embodiments, UE 116 receives warning data and key performance indicator data from eNBs 101-103 to detect network anomalies, diagnose anomalies, and provide instructions to correct the detected anomalies. In some embodiments, detecting network anomalies, diagnosing anomalies, and providing instructions to correct the detected anomalies can be performed completely or partially automatically.

[0052] UE 116 includes antenna 305, radio frequency (RF) transceiver 310, transmission (TX) processing circuit 315, microphone 320 and reception (RX) processing circuit 325. UE 116 also includes speaker 330, main processor 340, input / output (I / O) interface (IF) 345, input 350, display 355 and memory 360. Memory 360 includes basic operating system (OS) program 361 and one or more applications 362.

[0053] RF transceiver 310 receives incoming RF signals transmitted by eNBs of network 100 from antenna 305. RF transceiver 310 downconverts the incoming RF signals to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to RX processing circuitry 325, which generates a processed baseband signal by filtering, decoding, and / or digitizing the baseband or IF signal. RX processing circuitry 325 transmits the processed baseband signal to speaker 330 (such as for voice data) or to main processor 340 for further processing (e.g., for web browsing data).

[0054] The TX processing circuit 315 receives analog or digital voice data from the microphone 320 or other outgoing baseband data (such as network data, email, or interactive video game data) from the main processor 340. The TX processing circuit 315 encodes, multiplexes, and / or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The RF transceiver 310 receives the outgoing processed baseband or IF signal from the TX processing circuit 315 and up-converts the baseband or IF signal to an RF signal for transmission through the antenna 305.

[0055] The main processor 340 may include one or more processors or other processing devices and execute a basic OS program 361 stored in the memory 360 to control the overall operation of the UE 116. For example, the main processor 340 may control the reception of forward channel signals and the transmission of reverse channel signals by the RF transceiver 310, the RX processing circuit 325, and the TX processing circuit 315 according to well-known principles. In some embodiments, the main processor 340 includes at least one microprocessor or microcontroller.

[0056] The main processor 340 is also capable of executing other processes and programs resident in the memory 360, such as operations for root cause analysis and automation using machine learning as described in embodiments of the present disclosure. The main processor 340 can move data into or out of the memory 360 as needed for execution. In some embodiments, the main processor 340 is configured to execute applications 362 based on the OS program 361 or in response to signals received from the eNB or operator. The main processor 340 is also coupled to an I / O interface 345, which provides the UE 116 with the ability to connect to other devices such as laptops and handheld computers. The I / O interface 345 is a communication path between these accessories and the main processor 340.

[0057] Main processor 340 is also coupled to input 350 and display 355. An operator of UE 116 may enter data into UE 116 using a keypad such as input 350. Display 355 may be a liquid crystal display or other display capable of presenting text and / or at least limited graphics (such as from a website).

[0058] Memory 360 is coupled to main processor 340. A portion of memory 360 may include random access memory (RAM), and another portion of memory 360 may include flash memory or other read-only memory (ROM). Figure 3a An example of UE 116 is shown, but the Figure 3a Make various changes. For example, Figure 3a The various components in the embodiment may be combined, further subdivided, or omitted, and additional components may be added as required. As a specific example, the main processor 340 may be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, although Figure 3a The UE 116 is shown configured as a mobile phone or smart phone, but the UE may be configured to operate as other types of mobile or stationary devices.

[0059] Figure 3b An example eNB 102 according to the present disclosure is shown. Figure 3b The embodiment of the eNB 102 shown is for illustration only, and Figure 1 Other eNBs may have the same or similar configuration. However, eNBs have various different configurations, and Figure 3b The scope of the present disclosure is not limited to any particular implementation of an eNB. Note that eNB 101 and eNB 103 may include the same or similar structure as eNB 102.

[0060] like Figure 3bAs shown, the eNB 102 includes multiple antennas 370a-370n, multiple RF transceivers 372a-372n, transmission (TX) processing circuitry 374, and reception (RX) processing circuitry 376. In some embodiments, one or more of the multiple antennas 370a-370n include a 2D antenna array. The eNB 102 also includes a controller / processor 378, a memory 380, and a backhaul or network interface 382.

[0061] The RF transceivers 372a-372n receive incoming RF signals from the antennas 370a-370n, such as signals transmitted by a UE or other eNB. The RF transceivers 372a-372n down-convert the incoming RF signals to generate IF or baseband signals. The IF or baseband signals are sent to the RX processing circuit 376, which generates processed baseband signals by filtering, decoding and / or digitizing the baseband or IF signals. The RX processing circuit 376 transmits the processed baseband signals to the controller / processor 378 for further processing.

[0062] The TX processing circuit 374 receives analog or digital data (such as voice data, network data, email, or interactive video game data) from the controller / processor 378. The TX processing circuit 374 encodes, multiplexes, and / or digitizes the outgoing baseband data to generate a processed baseband or IF signal. The RF transceivers 372a-372n receive the outgoing processed baseband or IF signals from the TX processing circuit 374 and up-convert the baseband or IF signals to RF signals that are transmitted through the antennas 370a-370n.

[0063] The controller / processor 378 may include one or more processors or other processing devices that control the overall operation of the eNB 102. For example, the controller / processor 378 may control the reception of forward channel signals and the transmission of reverse channel signals by the RF transceivers 372a-372n, the RX processing circuit 376, and the TX processing circuit 374 according to well-known principles. The controller / processor 378 may also support additional functions, such as more advanced wireless communication functions. For example, the controller / processor 378 may perform a blind interference sensing (BIS) process such as that performed by the BIS algorithm and decode the received signal minus the interference signal. The controller / processor 378 may support any of a variety of different other functions in the eNB 102. In some embodiments, the controller / processor 378 includes at least one microprocessor or microcontroller.

[0064] The controller / processor 378 is also capable of executing programs and other processes resident in the memory 380, such as a basic OS. The controller / processor 378 is also capable of supporting root cause analysis and automation using machine learning as described in the embodiments of the present disclosure. In some embodiments, the controller / processor 378 supports communication between entities, such as web RTC. The controller / processor 378 can move data into or out of the memory 380 as needed for execution. The memory 380 stores various artificial intelligence (AI) algorithms for estimating UE speed and training data sets for training various AI algorithms.

[0065] The controller / processor 378 is also connected to a backhaul or network interface 382. The backhaul or network interface 382 allows the eNB 102 to communicate with other devices or systems through a backhaul connection or through a network. The backhaul or network interface 382 can support communication through (one or more) any suitable wired or wireless connection. For example, when the eNB 102 is implemented as part of a cellular communication system (such as a system supporting 5G, LTE, or LTE-A), the backhaul or network interface 382 can allow the eNB 102 to communicate with other eNBs through a wired or wireless backhaul connection. When the eNB 102 is implemented as an access point, the backhaul or network interface 382 can allow the eNB 102 to communicate through a wired or wireless local area network or through a wired or wireless connection to a larger network (e.g., the Internet). The backhaul or network interface 382 includes any suitable structure that supports communication through a wired or wireless connection, such as an Ethernet or RF transceiver.

[0066] In some embodiments, the eNB 102 transmits the warning data and the key performance indicator data to the server via the backhaul or network interface 382 to detect network anomalies, diagnose the anomalies, and provide instructions to correct the detected anomalies. In some embodiments, detecting network anomalies, diagnosing anomalies, and providing instructions to correct the detected anomalies can be performed completely or partially automatically.

[0067] although Figure 3b An example of an eNB 102 is shown, but the Figure 3b For example, eNB 102 may include any number of Figure 3b . As a specific example, an access point may include multiple interfaces 382, ​​and the controller / processor 378 may support routing functions to route data between different network addresses. As another specific example, although shown as including a single instance of the TX processing circuit 374 and a single instance of the RX processing circuit 376, the eNB 102 may include multiple instances of each (such as one per RF transceiver).

[0068] In certain embodiments of the present disclosure, the BS provides various data sources designated as key performance indicators (KPIs) to an operations support system (OSS). The OSS may be Figure 3b The KPI may be sent periodically, such as every fifteen minutes or another preset time interval. The KPI may include performance management (PM) counters. PM data reflects the state and behavior of the system. For convenience, the terms "KPI data" and "PM data" are used interchangeably in this patent document to refer to information sent periodically from the eNB. A subset of these data is called key quality indicators (KQIs). KQIs provide summary indicators reflecting service accessibility, service maintainability, service availability, service quality, and service mobility levels. In addition to PM data, the BS also reports fault management (FM) data or warnings, which are triggered in response to one or more unexpected events at the BS, such as CPU overload, memory overload, DSP restart, MME failure, etc. For convenience, the terms "FM data" and "warning data" are used interchangeably in this patent document to refer to information sent indicating warnings.

[0069] Troubleshooting is triggered in response to detecting that one or more KQIs fall outside of a threshold or nominal value. The troubleshooting process involves manual or automated reasoning steps to infer the root cause explanation for the KQI degradation. The root cause is obtained by detecting and diagnosing KPI anomalies, thereby providing fine-grained causal information for KQI anomalies. Other sources such as call subscriber tracking, trouble tickets, and customer complaint information can be included during this process.

[0070] For example, when the Quality of Service KQI indicates an abnormal decrease in IP throughput, the possible root cause could be low traffic demand or high RF interference. Once the root cause analysis (RCA) of the anomaly is complete, recovery steps can range from simply resetting the BS, or changing the Operation and Maintenance (OAM) parameters (e.g., transmit power, electrical tilt) at the BS.

[0071] Manual troubleshooting requires human domain experts to participate in each RCA step, including problem detection, diagnosis, and problem recovery. Since each BS reports thousands of KPIs during a single reporting interval (which may occur every 15 minutes or another preset interval), which occurs regularly at different preset time intervals, processing the large amount of data is not an easy task for human experts. There is an additional cost to reproduce the problem (through drive testing), test different solutions, and verify that the final solution fixes the underlying problem.

[0072] Figure 4a400 for using machine learning according to various embodiments of the present disclosure. The RCA framework 400 describes the process of applying machine learning to anomaly detection and root cause analysis. The RCA framework 400 can be used by, for example, Figure 1 and Figure 3b One or more associated servers in the eNB or such as Figure 1 and Figure 3a For example, the RCA framework 400 may be included in the UE electronic device of one of the UEs. Figure 3a In the UE 116. For another example, the RCA framework 400 may be included in Figure 3b A server (having Figure 3a The RCA framework 400 receives data, detects anomalies from the data, discovers the root cause of the detected anomalies, and then performs corrective actions. Figure 4a The embodiments shown are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0073] The RCA framework 400 is a rule-based framework that both derives rules and utilizes derived rules. The rules are in the form of the following equation (1). The terms on the left side of equation (1) are antecedents, while the terms on the right side are consequences. For example, KPI is an antecedent, while KQI is a consequence. That is, equation (1) describes the cause-effect relationship between KPI and KQI.

[0074] Formula (1)

[0075]

[0076] RCA framework 400 receives data from various sources such as source 402. Source 402 may be, for example, Figure 1 The source 402 may also include a core network and a radio access network (RAN). The received data may include KPI data and warnings. KPI data is received periodically at predetermined time intervals (e.g., every 15 minutes), while warnings are events generated at the eNB in ​​response to hardware or software failures.

[0077] The OSS 404 receives data from various sources 402. The OSS 404 may include one or more information repositories for storing warning data (such as FM data), KPIs including PMs, etc. KPIs and warnings indicate the overall network health. KPIs may be in the form of counters (e.g., number of RRC connection attempts) or measurements (meters), such as the average IP throughput per eNB over the last reporting interval. For example, real-time KPIs may come from various eNBs and arrive at periodic reporting intervals (RIs). Warnings are events that indicate problems or conditions caused by hardware or software processes. This information is used to find out if there are anomalies or faults in the E-UTRA network, determine the root cause, and perform corrective actions.

[0078] In some embodiments, the OSS 404 pre-processes the KPI and alert data. In other embodiments, the pre-processing of the KPI and alert data is performed external to the OSS 404. The pre-processing involves manipulating different information sources from the eNB. Since a large amount of data is received, the pre-processing is performed and the data may be in a disparate or different format. Therefore, the pre-processing modifies the data to make it usable in the RCA.

[0079] Preprocessing KPI, PM data, and alert data enables the data to be further processed by the batch processing layer 406 and the speed processing layer 408. Preprocessing includes filling missing values ​​with special entries (e.g., null values ​​(NaN)) to ensure that they are not accidentally included during further downstream processing. Preprocessing can include discretizing KPI data (including PM data), deriving composite PM metrics, and generating KPI hierarchies. The output of the preprocessed data produces data that can be used to detect anomalies and infer the corresponding root causes of the detected anomalies. Preprocessing is described below. Figure 5a-5h for discussion.

[0080] The batch processing layer 406 and the speed processing layer 408 receive the data. The batch processing layer 406 generates rules for RCA from the historical data based on the detected anomalies. The rules identify anomalies from the data and propose the cause of the anomalies. The rules can also propose one or more remedial actions that will resolve the detected anomalies. The speed processing layer 408 uses the rules generated for RCA to detect anomalies from real-time data and resolve the detected anomalies.

[0081] An anomaly is a sign that one or more quality of service metrics of the operator network are outside their normal range, and troubleshooting is required to resolve the underlying cause. Anomaly detection involves detecting anomalies or deviations of one or more of these KQI values ​​relative to their normal operating range. The following formula (2) describes the conditions for identifying anomalies. As shown in formula (2), an anomaly is identified when the KQI is less than a threshold. After detecting the anomaly, the RCA framework 400 then discovers the underlying root cause of the detected anomaly and then performs corrective actions based on the derived rules. That is, the anomaly is a symptom, and RCA is performed to identify the cause of the symptom, and thereby determine the remedy for the detected anomaly so that the operator network returns to normal.

[0082] Formula (2)

[0083] KQI≤T

[0084] The batch processing layer 406 receives data from the OSS 404 and saves the data in the historical data 410. The historical data 410 includes previously processed KPIs and alerts. The previously processed KPIs and alerts are saved in the historical data 410, and machine learning uses the historical data to identify anomalies in the data and generate rules (or conditions) to easily detect anomalies (based on cause and effect) in real time and provide an understanding of the root cause of the problem. The rules can also provide the steps required to resolve the identified anomalies. In contrast, the speed processing layer 408 checks the newly received data (such as real-time data) by applying the rules generated in the batch processing layer 406 to detect anomalies.

[0085] Anomaly detectors 412a and 412b identify one or more anomalies from KPI and warning data indicating that the operator network is outside its normal range. Anomaly detectors 412a and 412b detect anomalies in one or more KQI categories related to accessibility, maintainability, availability, integrity (such as quality), and mobility. Note that anomaly detector 412a detects anomalies from historical data 410, while anomaly detector 412b detects anomalies from real-time data. For example, anomaly detector 412a looks at previous data (via historical data 410) to identify sample patterns corresponding to operator network operations outside its normal range. Anomaly detector 412b looks at real-time data to identify patterns corresponding to operator network operations outside its normal range.

[0086] In some embodiments, there are multiple KQI categories corresponding to anomalies. The KQI category of accessibility provides the probability that an end user will be provided with an E-UTRAN radio access bearer (E-RAB) upon request. For example, service accessibility indicates the ease of establishing a connection with a cell.

[0087] The KQI category of holdability provides a measure of how often an end user abnormally loses an E-RAB during the time the E-RAB is in use. For example, holdability indicates the ability to maintain a connection.

[0088] The KQI category of availability provides a measure of the percentage of time that a cell is available. For example, availability indicates whether the system is unable to detect the presence of a cell.

[0089] The KQI category of integrity provides a measure of IP throughput in both the download and upload directions. For example, integrity indicates the quality of user experience and may be based on download speed.

[0090] The KQI category of mobility measures how well the E-UTRAN mobility functions work. For example, mobility indicates whether a user experiences frequent call drops.

[0091] The KQI category of a traffic measures the proportion of resource utilization at the cell during the RI. A greater amount of resource utilization means that the cell is overutilized and may reduce the quality of service within the cell.

[0092] In some embodiments, the inputs to the anomaly detectors 412a and 412b are time stamped. The inputs may be provided in a streaming manner (such as one sample per RI per eNB) or in a batch manner (such as corresponding to historical data for one or more days / one or more months in a year). For each KQI y∈Y={accessibility, maintainability, availability, integrity, mobility}, if the KQI value of the data is an anomaly sample, the anomaly detectors 412a and 412b output a function AN whose value is non-zero. y For each y, expressed as a KPI j (y), for y the KPI at level j in the KPI hierarchy. Table 1 below describes different KPIs in the tree hierarchy that can be used for anomaly detection for a particular KQI category.

[0093] Table 1

[0094]

[0095]

[0096] Embodiments of the present disclosure provide three different anomaly detection methods for anomaly detectors 412a and 412b. The detection methods include (i) extreme outliers, (ii) standard deviation, and (iii) median absolute deviation (MAD).

[0097] Regarding the extreme outlier approach, if at the KPI level (such as Figure 5bIf at least one KPI in the KPI tree level 520 is below a threshold of a KQI, the anomaly detectors 412a and 412b identify an anomaly. The threshold can be set to a fixed value or calculated based on hourly KPI statistics.

[0098] Regarding the standard deviation method, if at the KPI level (such as Figure 5b If one or more KPIs within the KPI tree level 520 of the KPI tree are lower than k standard deviations of the mean value of the KPI of a KQI, then the anomaly detectors 412a and 412b identify an anomaly. The standard deviation method is described in equation (3) below.

[0099] Formula (3)

[0100]

[0101] With respect to MAD, anomaly detectors 412a and 412b identify anomalies based on the median of a set of differences between each sample point and the sample median. For example, if the KPI value is less than or equal to An anomaly is detected, where Indicates the hourly interval KPI to which the RI belongs j The median value of this MAD j It is a function of the median absolute deviation of the KPI from its median hourly KPI. j It is called the anomaly detection threshold and is used to control the number of anomalies detected for KPIj. The MAD method for anomaly detection is described in the following equation (4).

[0102] Formula (4)

[0103]

[0104] For example, to identify the KPI sequence Y as input and the anomaly detection threshold t The first step is to extract tX = {y: Pr[Y≤y]≤0.2}. It should be noted that values ​​of Y above the 20th quantile do not qualify as outliers. The second step is to solve MAD, where MAD = c×Median[|X-Median[X]|]. In order to identify C, the standard distribution of the identified X is calculated based on So that c = 1 / z, where z is such that Step 3: For each x in X, the MAD method defines M-score = (x i -Median[X]) / MAD. The M-score is compared with a threshold such that if the M-score is less than the threshold t, an anomaly is identified.

[0105] The machine learning training model 414 generates rules for identifying detected anomalies from the historical data 410 through the anomaly detector 412a. The machine learning training model 414 also generates parameters for identifying anomalies from the historical data 410. The rules and parameters can also be used to perform RCA on the detected anomalies and provide one or more remedial actions to resolve the detected anomalies. The following formula (5) describes an example rule used in RCA. The generation of rules and parameters by the machine learning training model 414 is discussed in more detail below.

[0106] Formula (5)

[0107]

[0108] The RCA framework 400 can also evaluate the quality of each derived rule. If S represents the set of historical data 410, then Represents a collection of sample anomalies containing a specific KQI. Similarly, represents the set of relations whose entries satisfy rule j. Equations (6), (7), and (8) describe various measures of the quality of each derived rule. For example, the support of equation (6) is the relative proportion of KPI samples that obey the rule antecedent. For example, a higher confidence score in equation (7) indicates that the derived rule tolerates more and more samples with KQI anomalies compared to normal samples. The confidence score in equation (7), also known as precision or posterior value, refers to the fraction of anomaly samples that obey rule j. This is equivalent to the conditional probability Pr(s∈A KQI |s∈F). A higher confidence rule better distinguishes abnormal samples from normal samples. Similarly, a higher hit rate of (8) indicates that the derived rule can be applied to most KQI abnormal samples. The hit rate of (8), also called recall or likelihood, corresponds to the fraction of KQI anomalies associated with rule j. This is equivalent to the conditional probability Pr(s∈F|s∈A KQI ).

[0109] Formula (6)

[0110]

[0111] Formula (7)

[0112]

[0113] Formula (8)

[0114]

[0115] Once the machine learning training model 414 is generated, the rules and parameters that meet the design goals such as minimum support and confidence (which can be any preset threshold, such as 80%) are saved in the information repository 418. The information repository 418 can be similar to Figure 3a Memory 360 or Figure 3b The information repository 418 may store one or more rules for identifying anomalies and the root causes of anomalies. The rules stored in the information repository 418 may be used as fingerprints to check for (one or more) matching rules, such as checking whether the KPIs of the anomaly samples obey these rules. If a rule match is found, an explanation may be provided at the user interface to explain the physical meaning of the rule so that a human operator can infer the underlying root cause of the anomaly.

[0116] In some embodiments, the information repository 418 also includes a knowledge base derived using human domain expertise. In some embodiments, the data within the information repository 418 is applied to the PM to prepare the data in a format modifiable for performing RCA and corrective actions.

[0117] Once rules are generated that suggest performance degradation with high confidence, the RCA framework 400 identifies which of the generated rules correspond to root causes and which of the generated rules correspond to associated symptoms. The following three examples describe identifying which rules correspond to root causes of KPI degradation.

[0118] In some embodiments, the chain rule r∈R is a set of rules about certain KPIs, which are considered With high confidence. Figure 5c Linking rules together is described in more detail. Thereafter, the relationships between the basic rules in R using a directed graph are established. First, the set of rules in R is divided into basic rules and non-basic rules, and all basic rules are considered as nodes in the graph. Next, if a rule The confidence level is P(KPI j =1-|KPI i =1) is higher than the threshold, then all non-basic rules are satisfied and considered in a pair of basic rules (KPI i =1, KPI j =1). Therefore, directed edges indicate the causal relationship between nodes, i.e., KPI j =1 is KPI i =1 sign, or KPI i =1 leads to KPI j = 1. Nodes with no incoming edges indicate that they are not caused by any other basic rules and are therefore defined as the root causes of the chain rule, while the rest are identified as symptoms.

[0119] In another embodiment, when constructing a KPI hierarchy tree (such as the following Figure 5c As described above, the processor can identify the leaf nodes of the chain rule as the root cause and the other rules as accompanying symptoms.

[0120] In yet another embodiment, all generated rules are considered nodes in a directed graph. Edges between nodes are then added using a method similar to that described above (in the first example). If there is a bidirectional edge between any two nodes, the processor collapses it into one node. Nodes with no incoming edges are defined as the root cause of the KQI. All possible paths from these nodes to the KQI nodes are now a new set of chained / composite rules, where the nodes on the path are symptoms.

[0121] With respect to the speed processing layer 408, after the anomaly detector 412b identifies an anomaly in real time, the RCA 420 identifies the cause of the detected anomaly using information from the information repository 418. For example, the RCA 420 uses machine-learned rules (from the machine-learned training model 414 that are above confidence and hit rate thresholds) to identify the underlying root cause of the detected anomaly.

[0122] In some embodiments, after RCA 420 determines the root cause of the detected anomaly, an explanation of the root cause may be displayed on the user interface along with remedial action 422. When remedial action 422 is displayed, it may include a recommendation for action(s) to be performed to restore the network to its normal operating state. In other embodiments, remedial action 422 performs the necessary actions to automatically restore the network to its normal operating state. Remedial action 422 may apply corrective actions based on the determined root cause that caused the KQI of interest to decrease.

[0123] After determining the root cause and explanation, the next step of the RCA framework 400 is to execute the set of corrective actions (remediation actions 422) to correct the underlying cause of the anomaly and restore the network to its normal operating state.

[0124] In some embodiments, to determine the corrective action, the RCA framework 400 uses a machine learning inference engine (not shown). The machine learning inference engine identifies the correct set of recovery actions based on a set of feature inputs. For example, the machine learning inference engine can be trained using labeled data provided from the historical data 410. For example, the machine learning inference engine is trained using job logs, CM data, etc. stored in the historical data 410. The training labels provide the best corrective action c for each feature vector in the training set x.

[0125] To generate training given job logs and CM data, CM data including eNB configuration stored across multiple dates is compared between one day and the next day and only the changes applicable to configuration parameters are retained. This is sorted in time against the corresponding date when the anomaly was resolved. Thus, labeled data is generated that identifies the nature and magnitude of the eNB configuration parameter changes and what were the underlying root causes / symptoms for applying these changes.

[0126] The input to the machine learning inference engine is the feature vector x = [x 1 , x 2 ,...x n ]. Each entry in this vector is pre-processed PM data, which can be either continuous or discrete values. Examples of entries can include KPIs (such as DLSchedulerMcsRatio, ULSchedulerMcsRatio, RssiPath0Avg, RssiPath1Avg, DLResidualBlerRetrans0, as described below). The output of the machine learning inference engine is the corrective action c∈{C 1 , C 2 , ...C K Possible actions include increasing antenna downtilt (to reduce overshoot), decreasing antenna downtilt (to increase RF coverage), adjusting one or more RRC parameters (such as UE power control parameters, RS-to-data energy per resource element), or adjusting one or more scheduling parameters (to increase the priority set for the GBR radio bearer).

[0127] As described above, PM data and warning data are pre-processed so that the data can be Figure 4a The batch layer 406 and Figure 4a The PM data is further processed by the velocity processing layer 408. The pre-processing of the PM data is based on (i) discretizing the PM data, (ii) deriving a comprehensive PM metric, and (iii) generating a KPI hierarchy. The output of the processed PM data can be used to detect anomalies and their corresponding inferred root causes. Processing the warning data includes associating the PM data and the warning data. Figure 5a-5h Preprocessing of PM data and warning data is described. Figure 5a-5h The embodiments are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0128] KPI discretization improves the efficiency of rule mining and processing of subsequent RCA. In some embodiments, continuous-valued PM data can be used for rule mining and processing of subsequent RCA, however, discretized PM data is more efficient than using continuous-valued PM data.

[0129] Discretization can be performed based on comparing the KPI data to a fixed threshold or a statistical threshold. Discretization using a statistical threshold involves first obtaining the statistical distribution of PM data within a specific time interval (e.g., daily, hourly, etc.). The distribution can be obtained at a pre-specified quantile value (e.g., the 10th quantile, the 70th quantile, or the 100% quantile). Next, the PM data is binned to determine which quantile interval the PM data belongs to. For example, bin 0 may correspond to PM data below the 10th quantile, bin 1 may correspond to PM data between the 10th quantile and the 90th quantile, and bin 2 may correspond to outliers (such as PM data greater than the 90th quantile). It should be noted that for comprehensive PM, special quantization bins are set aside in cases where PM is invalid (such as if no measurements are available, it is set to a null value).

[0130] In some embodiments, the threshold for discretizing PM data for RCA (e.g., extracting rules in the form of equation (2) above) may be selected based on the threshold used for classifying KQI anomalies. For example, if the anomaly detection threshold is selected so that a greater number of samples are declared to experience anomalies at the KQI of interest. Then, the PM threshold for RCA purposes may be changed accordingly so that the RCA framework 400 will attempt to discover the root cause of a greater number of KQI anomaly samples.

[0131] Adaptively adjusting the PM threshold for RCA purposes as a function of the threshold for detecting reduced KQI anomalies provides a reasonable compromise between high rule confidence (based on equation (7), which corresponds to the fraction of samples to which the rule applies that are also declared anomaly samples) and high hit rate (based on equation (8), which corresponds to the fraction of anomaly samples to which the rule applies).

[0132] If the threshold for marking a KQI sample as an anomaly is chosen to be more stringent, fewer KQI event occurrences are classified as anomalies. For example, an anomaly detection algorithm that classifies samples with throughput below 1 Mbps as anomalies will classify fewer throughput anomaly samples than an anomaly detection algorithm that classifies samples with throughput below 5 Mbps.

[0133] In certain embodiments, the RCA framework 400 also uses the historical data 410 to perform KPI predictions based on IP throughput. Figure 4b A block diagram 430 of traffic prediction based on anomaly detection according to various embodiments of the present disclosure is shown. Figure 4c A block diagram 450 of performing traffic prediction based on anomaly prediction according to various embodiments of the present disclosure is shown.

[0134] In some embodiments, the RCA framework 400 can automatically predict data at future times based on a trained prediction model. Once the difference between the predicted data and the actual incoming data is greater than a certain threshold, an anomaly warning will rise. For example, a machine learning framework for traffic prediction based on anomaly detection and anomaly prediction will improve the efficiency of the RCA framework 400.

[0135] KPI prediction based on machine learning uses feature engineering based on historical data. Single feature methods can be based on IP throughput, while multi-feature methods can be based on various parameters, such as hours of the day, the number of active users in a specific geographic area, etc. Based on daily, monthly, quarterly, annually (and various other time intervals), the RCA framework 400 using a long short-term memory (LSTM) neural network or a quantile regression forest can detect and predict IP throughput anomalies in the network. For example, at each timestamp, the RCA framework 400 can predict the amount of data that will appear in the future. When receiving actual data, the RCA framework 400 compares the amount of predicted data with the actual amount of data. If the difference between the predicted data and the actual data exceeds a threshold, the anomaly detector 412b identifies the anomaly.

[0136] An example dataset for time series traffic forecasting is shown in Table (2) below. The dataset in Table 2 describes system log data including different KPIs across the network. KPIs are in the form of counters (e.g., number of RRC connection attempts) or measurements (metrics), such as the average IP throughput per eNB over the last reporting interval. Table 2 gives a description of the features used in anomaly detection and anomaly prediction.

[0137] Table 2

[0138]

[0139] Figure 4b The block diagram 430 of FIG. 4 depicts a machine learning-assisted anomaly detection framework based on traffic prediction. The system first collects historical data, a subset of which is used to train a traffic prediction model based on deep learning / machine learning techniques. The historical data used for traffic prediction can be combined with Figure 4aThe same as the historical data 410 of the previous example. Based on the prediction model, the system predicts the data traffic or IP throughput on the uplink or downlink at a future timestamp. Note that the term "data" can refer to the amount of uplink or downlink traffic, IP throughput, etc. In order to evaluate the accuracy of the prediction, the verification data 436 obtains the predicted data 438a and also compares it with the actual data arriving at the future timestamp. Since the data set is a time series, an error vector can be generated by comparing the difference between the predicted data vector and the reference truth data (ground truth) vector. Once the error vector is obtained, the RCA framework 400 can calculate the statistical parameters of the error, such as the empirical mean, variance, standard deviation, etc., and record them in the database.

[0140] In streaming data scenarios, at each timestamp, the model predicts the data that will appear at a future timestamp. When actual data is received at that timestamp, the predicted data is compared with the ground truth (actual data). If the difference between them exceeds an absolute or relative threshold, the system declares an anomaly. The choice of anomaly detection threshold depends on which anomaly detection method is used.

[0141] Input data is a data set in the past timestamp, which can be a single or multiple features. Actual data is data coming in the future timestamp. Training data 432 includes historical data collected by the system for training the prediction model. Based on the training data 432, the prediction model 434 is trained. For example, the prediction model 434 is constructed based on deep learning (such as LSTM) and machine learning methods (quantile regression forest). The verification data set 436 generates an error vector by calculating the difference between the predicted data vector and the actual data vector. The calculation parameter 438b obtains the error vector from the verification data set 436 and identifies the mean, variance, standard deviation or other parameters related to the statistical distribution of the error vector. The test data set 440 is a data set for real-time testing. The verification data set 436 includes historical data for verifying the prediction model and generating parameters. Detecting anomalies 444 detects whether the actual data is abnormal. Based on which anomaly detection method the system will choose, this function works differently. For the standard deviation method, detecting anomalies 444 detects anomalies by checking whether the error between the predicted data and the actual data is greater than a given threshold, which is determined by the parameters obtained in the verification. For the Tukey method, Detect Anomaly 444 chooses not to use the parameters obtained in the validation dataset, but to detect anomalies based only on the predicted data interval and the actual data.

[0142] Figure 4c The block diagram 450 depicts a machine learning-assisted anomaly prediction framework based on traffic prediction. Figure 4b Anomaly detection based on traffic prediction, Figure 4cThe block diagram 450 first obtains a prediction model based on historical data. The block diagram then groups the historical data based on timestamps in the same time period (such as the same hour, or the same hour-minute). For example, all past data with timestamps in the time interval from 2:00 pm to 3:00 pm belong to the same group. Once the grouped data set is constructed, the system calculates parameters for each group, such as mean, standard deviation. In the anomaly prediction stage, at each timestamp, the system first predicts the data that will arrive at a future timestamp. Then, by comparing the predicted data with the expected value or interval determined by statistical information from the historical data at that particular timestamp, the system predicts whether the predicted traffic constitutes an anomaly.

[0143] Construct a grouped dataset based on the timestamp of the historical data (such as Figure 4a The system groups historical data 410) for each time stamp. For data with the same timestamp in hours and minutes, the system puts them into the same group. Calculation parameters 454 identify the mean, variance, standard deviation and other statistical parameters for each data set group. Predict anomalies 456 predict anomalies based on data that will arrive in future timestamps. This function works differently based on the anomaly prediction method selected by the RCA framework 400. For the standard deviation method, at each timestamp, predict anomalies 456 predicts anomalies by checking whether the error between the predicted data and the mean of the historical data in the next timestamp group is greater than a given threshold, which is determined by the parameter. For the Tukey method, at each timestamp, predict anomalies 456 predicts anomalies by checking whether the mean of the historical data in the next timestamp group falls within the range determined by the prediction interval.

[0144] In some embodiments, the KPI prediction based on IP throughput using historical data 410 includes preprocessing the data using a moving window average. Since the raw traffic data arrives regularly at certain intervals (such as 15-minute intervals), including various bursts, the preprocessing step applies a filter to reduce its variance. In some embodiments, the processor calculates a moving average on the raw data. The moving average is based on the expression It should be noted that is the data after moving window averaging at timestamp i, x(i) is the original data at timestamp i, and T w is the size of the moving window. Note that applying a moving window does not change the size of the output relative to the size of the input. If the KPI is reported every 15 minutes, then T w is set to 1 hour, which covers four sample points in the window. That is, the RCA framework 400 loads the raw data and then takes a moving window average to smooth the network traffic curve.

[0145] In some embodiments, the KPI prediction based on IP throughput using historical data 410 includes preprocessing the data using data binning. Data binning is mainly used for deep learning and multi-feature input. The features used in LSTM prediction other than downlink / uplink throughput are: UEActiveDLAvg / UEActiveULAvg, TotPrbDLAvg / TotPrbULAvg and timestamp. The data is then quantized into multiple bins.

[0146] For example, for the number of active users in DL (UEActiveDLAvg) and UL (UEActiveULAvg), the average number of UEs in downlink / uplink is binned into four groups. For example, (i) bin 0 corresponds to the average number of UEs between 0 and 5, (ii) bin corresponds to the average number of UEs between 5 and 10, (iii) bin 2 corresponds to the average number of UEs between 10 and 20, and (iv) bin 3 corresponds to the average number of UEs over 20.

[0147] For another example, TotPrbDLAvg / TotPrbULAvg, the PRB utilization in the quantized downlink / uplink is binned into four bins. For example, (i) bin 0 corresponds to the average ratio of PRBs between 0 and 30, (ii) bin 1 corresponds to the average ratio of PRBs between 30 and 65, (iii) bin 2 corresponds to the average ratio of PRBs between 65 and 95, and (iv) bin 3 corresponds to the average ratio of PRBs between 95 and 100.

[0148] For another example, the timestamps may be binned by hour. That is, bin 0 may correspond to the hours between 0:00 AM and 6:00 AM, bin 1 may correspond to the hours between 6:00 AM and 12:00 AM, bin 2 may correspond to the hours between 12:00 PM and 6:00 PM, and bin 3 may correspond to the hours between 6:00 PM and 12:00 AM.

[0149] In some embodiments, time series traffic forecasting can be based on LSTM and machine learning, where the input can be (i) a single feature input, (ii) multiple feature inputs, and (iii) clustered single feature inputs. That is, LSTM predicts throughput data at future timestamps. For example, for a given input x, there can be two LSTM layers followed by a fully connected (dense) layer to generate a single output o. The dense layer has a single output corresponding to the predicted traffic. Since the performance of time series traffic forecasting depends on the selection of input features, different inputs (such as a single feature input, multiple feature inputs, clustered single feature inputs) may change the results.

[0150] For a single input feature, only the data of a single feature is input into the LSTM to train the prediction model. A single input feature can be expressed as Among them, x p (i) is the predicted data at time stamp i, x(ij) is the actual data at time stamp ij, is a sequence of size L, and L represents how many past timestamps the LSTM uses to make predictions. In some embodiments, the input is throughput samples in past periods.

[0151] For multiple feature inputs, LSTM takes data from multiple features as input. Multiple feature inputs can be expressed as Among them, x p (i) is the predicted data at timestamp i, x (K) (ij) is the actual data of the kth feature at timestamp ij, K is the total number of features used as input, is the data sequence in the kth feature of size L, and L represents the number of past timestamps used for prediction. Same as the previous section, here LSTM represents the overall deep learning network with a dense layer size of 1.

[0152] It should be noted that the single input feature and the multiple feature inputs are from a single cell. The clustered single input feature performs traffic prediction for multiple cells simultaneously based on past data from all these cells. That is, the clustered single input feature can use data not only in the time domain, but also across the spatial domain (across multiple eNBs and cells). In order to balance the computational cost and the predicted coverage of the cell, the clustered single input feature divides the entire network into multiple clusters and trains the prediction model per cluster. Therefore, the clustered single input feature can be expressed as in, is the prediction result for all N cells at the ith time stamp in a given cluster, x d (j) is the actual data in the dth cell at timestamp j, is a sequence of past data in the dth cell, L represents the number of past timestamps, and N is the number of cells in the cluster. In some embodiments, a cluster may be a single cell or a group of cells within a particular geographic area. In other embodiments, cell clusters may be grouped according to their cell coverage (using downlink transmission power, capabilities such as whether the cell is a macro cell or a pico cell), operating carrier frequency, radio access technology (e.g., 4G LTE, 5G New Radio), etc. The above method may also include additional embodiments, such as multiple feature inputs available from each cell.

[0153] In some embodiments, time series traffic forecasting can be based on traffic forecasting based on quantile regression forests. Quantile regression forests are a modified version of the random forest algorithm for high-dimensional regression and classification. Figure 4d A framework 460 of an example quantile regression forest according to various embodiments of the present disclosure is shown. Input 462 is the input feature. For example, for a given input X=x, the conditional distribution of Y is less than y, which is F(y|X=x)=P(Y≤y|X=x), where Y is the dependent variable corresponding to the predictor variable X. The quantile regression forest approximates the conditional distribution as F(y|X=x)=P(Y≤y|X=x)=E(1 {Y≤y} )|X=x). An estimate of the distribution function is obtained by taking a weighted average of the leaves that x falls into, such as leaves 464a, 464b, and 464c. Since the α quantile is calculated as Q α (x) = inf{y: P(Y≤y|X=x)≥α}, which can identify the prediction interval as I for a given input x and probability α α (x) = [Q (1-α) / 2 (x), Q α (x)]. Note that only past throughput samples are used as input in a single cell, and the output is the prediction interval on the throughput at the next timestamp. This can be expressed as Among them, I α (x) represents the prediction interval at time stamp i, x(ij) is the actual data at time stamp ij, is a data series of size L, and L represents the number of past timestamps used to make predictions. QRF stands for Quantile Regression Forest.

[0154] As mentioned above Figure 4b As described above, the traffic prediction for anomaly detection can be based on standard deviation-based anomaly detection or Tukey-based anomaly detection. For standard deviation-based anomaly detection, after building the traffic prediction model, the first step includes using the validation dataset 436 to measure the prediction ability of the model and calculate the prediction error. The prediction error is represented as err val =(x p (1)-x(1),x p (2)-x(2),...,x p (T)-x(T)), where err val is the error vector on the validation dataset, x(i) is the actual data at timestamp i, and x p (i) is the predicted data at time stamp i, and T is the size of the prediction vector. Based on the error vector err val The second step includes identifying the standard deviation. Block 4389b identifies a parameter such as the standard deviation. The standard deviation is expressed as stdval =STD(err val , T w ), where std val is the standard deviation on the error vector, and STD represents the function for calculating the standard deviation. Here, T w The time window over which the data are used to calculate the standard deviation. For example, if T w =8 and T=24, there are 3 standard deviation values ​​corresponding to their time ranges. Block 430 can store these parameters for future application of anomaly detection. At runtime, at each timestamp i-1, the traffic prediction method automatically predicts the data that will arrive at the next timestamp i based on past data samples. For example, by comparing the predicted data with the actual data, when |x p (i)-x(i)|≥k*std val When an exception exists.

[0155] For Tukey-based anomaly detection, prediction intervals are required before applying anomaly detection. In the first step, prediction intervals in the test dataset are obtained. Next, the intervals are divided into four quarters. Anomalies are detected based on these quarters. For example, when the traffic prediction method makes a prediction for the data that will occur at timestamp i, by running a quantile regression forest, prediction intervals i are obtained. α (x). For a given prediction interval I α (x), dividing it into four quarters, where its boundaries are defined by Q1, Q2, and Q3. For example, if I α (x) = [1, 3], then Q 1 =1.5, Q 2 =2, Q 3 =2.5. When the actual data x(i) arrives at timestamp I, if x(i) 1 -3|Q 3 -Q 1 |or x(i)>Q 3 +3|Q 3 -Q 1 |, serious anomalies may occur. When the actual data x(i) arrives at timestamp I, if Q 1 -3|Q 3 -Q 1 | <x(i)<Q 1 -1.5|Q 3 -Q 1 | or Q 3 +1.5|Q 3 -Q 1 | <x(i)<Q 3 +3|Q 3 -Q​1 |, then there may be a possible exception.

[0156] As mentioned above Figure 4c As described above, the traffic prediction for anomaly prediction can be based on anomaly prediction based on standard deviation, or anomaly prediction based on Tukey. For anomaly prediction based on standard deviation, first, the historical data is grouped by timestamp, then the parameters of each group are identified, and finally the anomaly prediction 456 can predict anomalies from the test data set 440. For example, after collecting historical data, the first step of block diagram 450 groups the data based on the timestamp of the data. That is, for data with the same hour and minute, the system groups them into one group. The group can be expressed as in, is a set of timestamps of size T with the same hour-minute value k, and x k is the kth data group containing all of them. After dividing the data set into groups by timestamp, in the second step, the parameters 438a are calculated to identify the mean and standard deviation on each group. The mean is expressed as mean k = MEAN(x k ), and for each group the standard deviation is expressed as std k =STD(x k , T w ). Please note that T w The same as the time window over which the data is used to calculate the STD. The system stores the mean and STD for each timestamp group. After predicting the data that will appear at the next timestamp i, the third step of block diagram 450 performs anomaly prediction at the same timestamp based on the predicted data. For example, when |x p (i)-mean k |≥k*std k , the anomaly at timestamp I is high.

[0157] For Tukey-based anomaly prediction, the historical data is first grouped by timestamp and the parameters associated with each group are identified. Next, the prediction intervals for the test data set are obtained and divided into four quarters. Finally, predict anomalies 456 predict anomalies. For example, block 450 groups the data by timestamp and calculates the average over each data group. Next, prediction interval i is obtained at timestamp i α (x), and divide the prediction interval into four quarters. The predicted anomaly 456 in mean k 1 -3|Q 3 -Q 1 | or mean k >Q 3 +3|Q 3 -Q 1 ​| when predicting a high probability of anomaly.

[0158] In some embodiments, after performing traffic prediction based on anomaly detection or anomaly prediction, the RCA framework 400 filters the results before reporting them to the system administrator. Filtering reduces the probability of false positives. For example, a post-processing filter can be a throughput threshold, so that if the sample value (i) is below the threshold and the anomaly detection method declares the sample to be an anomaly sample, the system issues an anomaly warning or (ii) exceeds the threshold, the filter declares the sample to be not anomaly (regardless of whether the machine learning model declares the sample to be anomaly). Different thresholds can be applied to different days, different times of the day, different seasonal throughput averages, etc.

[0159] like Figure 5a The block diagram 500 shown depicts discretized KPI data. The block diagram 500 may be composed of Figure 4a The RCA framework 400 of the embodiment of the present invention is associated with a server to perform. Block diagram 500 describes a function or database including a mapping that associates anomaly detection thresholds with quantitative thresholds for discretizing KPI data. In addition, block diagram 500 describes a function or database including a mapping that associates geographic areas, times of day, operators, and quantitative thresholds for discretizing KPI data. For example, for a threshold of 1 Mbps, the discretization threshold can be based on the 90th percentile of the hourly KPI value. For another example, for a threshold of 0.1 Mbps, the discretization threshold can be based on the 99th percentile of the hourly KPI value. For another example, for a threshold of 0.01 Mbps, the discretization threshold can be based on the 99.9th percentile of the hourly KPI value. Note that similar rules can be formulated based on connections of a day, geographic location, etc.

[0160] For example, a threshold parameter for anomaly detection is selected (block 502). The selected threshold parameter (block 502) is received by anomaly detection block 504. Anomaly detection block 504 is similar to Figure 4a Anomaly detectors 412a and 412b are provided. Anomaly detection block 504 detects anomalies based on selected threshold parameters. In block 508, a KQI is obtained from KPI data 506. Anomaly detection block 504 receives the obtained KQI (KQI of block 508) and a threshold to generate a KQI anomaly sample list 510. Block 512 discretizes the KPI based on KPI data 506 and function 514, wherein function 514 is based on the selected threshold parameters (block 502). Discretized PM data 516 is used for RCA 518. RCA 518 is similar to Figure 4a RCA 420.

[0161] In other embodiments, percentile or absolute thresholds for discretized PM data are selected for RCA in an iterative manner. For example, a threshold for one or more PM data can be set to a starting value so that the corresponding confidence score is evaluated based on equation (7) and the hit rate is evaluated based on equation (8). If the confidence score, the hit rate, or both the confidence score and the hit rate do not meet the design constraints, the thresholds are iteratively adjusted accordingly until the design constraints are met. In another embodiment, each time the threshold for detecting KQI anomalies is modified, the iterative process of determining the PM threshold for RCA is repeated.

[0162] Note that for a given network, there are multiple types of geographic areas that will have different underlying patterns and statistics. For example, a residential area will have high traffic demand in the morning and evening, and lower traffic demand in the middle of the day when people are working. In contrast, an industrial or commercial area will have higher traffic demand around weekday business hours (such as 9:00 am to 6:00 pm Monday through Friday), and little demand outside of these hours. Therefore, patterns associated with geographic location (residential compared to industrial / commercial areas) and time constraints can be considered when setting thresholds for anomaly detection or parameter discretization. For example, if commercial and residential areas are combined to calculate hourly statistics and quantiles, their different usage patterns may change the combined distribution in a way that makes outliers more difficult to detect.

[0163] In order to set the threshold to correspond to the geographical location and time constraints, the operator (user) can manually mark the area where each eNB is located. However, manually marking each eNB is a time-consuming task and requires manual input every time a new site is added to the network. Alternatively, marking each eNB can be performed automatically. For example, Figure 4a The root cause analysis framework 400 can identify patterns based on statistics at a day, week, or even month level. Based on the identified patterns, sites that follow similar temporal statistics can be grouped together. Identifying patterns can be performed through clustering or classification techniques. Once a site is classified based on its statistical pattern, it can be grouped with sites in the same class in order to calculate long-term statistics for anomaly detection and parameter discretization.

[0164] Based on how quickly the statistics for a group change, the window size used to calculate the statistics used for anomaly detection and KPI discretization can be adjusted accordingly. For example, if a residential area typically observes the same traffic demand over a 6-hour period, the KPI discretization algorithm can use a larger time window to calculate the quantile level. Using large time windows when traffic demand is constant provides the ability to achieve quantile level accuracy that may be required for anomaly detection. For example, in order to obtain a 0.999 quantile, 1000 samples are required. If the discretization algorithm requires another level of accuracy, the sample count would need to be another order of magnitude. If the local site group is small, there may not be enough samples in just one hour to achieve these desired levels of accuracy. By adjusting the window length based on the site area classification, the algorithm can ensure that it has enough samples to achieve the desired level of accuracy.

[0165] The mapping between geographic location, time of day, and KPI discretization thresholds can be stored in a database. The mapping can be used during the root cause analysis of the root cause analysis framework 400.

[0166] Table 3 below shows example KPIs that may be used during an RCA for uplink integrity associated with a KQI anomaly.

[0167] Table 3

[0168]

[0169]

[0170] Table 4 below shows example KPIs that may be used during RCA for downlink integrity associated with KQI anomalies.

[0171] Table 4

[0172]

[0173]

[0174] Preprocessing PM data also includes deriving synthetic PM data. Synthetic PM data is derived by combining different KPIs. Combining different KPIs provides specific insights that are not already available within the raw PM data. The following examples describe various synthetic PMs that can be used to pinpoint the root cause of IP throughput-related anomalies.

[0175] The first example comprehensive PM is called high resource utilization. In the case of high resource utilization, the CPU load, memory load, and disk utilization of each rack and slot of the processing unit at the eNB can provide insight into the overall health of the network. For example, high CPU load may limit the ability of the eNB to provide downlink and uplink traffic to users, resulting in a decrease in perceived quality of service due to reduced IP throughput and increased IP latency at the end user.

[0176] The second example integrated PM is called the uplink power headroom ratio. The uplink power headroom ratio specifies the ratio of the count of power headroom reports during the most recent RI when the eNB received a power headroom report (PHR) with an index equal to or lower than 18 to the total number of PHRs received in the entire reporting range (between 0 and 63). The PHR is a quantitative measure of the available power headroom (PH, measured in dB), defined as the maximum transmit power P of the UE. c，max Its instantaneous transmission power (derived from its estimated downlink path loss PL, its nominal PUSCH transmission power P 0，PUSCH , partial path loss parameter α and number of allocated resource blocks M PUSCH，RB The power margin is determined by equation (9).

[0177] Formula (9)

[0178] PH=P c，max -[P 0，PUSCH +α·PL+10*log 10 M PUSCH，RB ]

[0179] In the above equation (9), the PHR index j indicates that the power headroom of the UE is within the interval j-23dB≤PH<j-22dB. For example, index 0 indicates that the power headroom is between -23dB and -22dB, while index 63 indicates that the PH exceeds 40dB.

[0180] Equation (10) describes the uplink power headroom ratio. As described in equation (10), a ratio close to 1 indicates that a large number of UEs are power limited (ie, use maximum transmission power for PUSCH transmission), which indicates that the cell of interest has uplink coverage issues.

[0181] Formula (10)

[0182]

[0183] The third example comprehensive PM is called weighted power headroom. The weighted power headroom provides that the reports from different users can be weighted in proportion to their number of occurrences. The following equation (11) describes the driving weighted power headroom.

[0184] Formula (11)

[0185]

[0186] The fourth example comprehensive PM is called the uplink scheduler MCS ratio. The uplink scheduler MCS ratio is the ratio of the cumulative count of resource blocks with uplink MCS values ​​between 0 and 5 (inclusive) during the most recent RI to the cumulative count of resource blocks with uplink MCS values ​​between 0 and 28 (inclusive). The corresponding KPI is named ULSchedulerMcsRatio, as shown in the following formula (12). A ratio close to 1 indicates that a large number of users are provided with uplink MCS values ​​equal to or lower than 5 (corresponding to QPSK modulation symbols with a low coding rate), resulting in fewer payload bits transmitted per scheduling opportunity. This indicates that the uplink throughput is low. It should be noted that formula (12) is calculated only when the denominator is greater than zero, otherwise a special value (such as -1, null value, etc.) is used to mark the ratio.

[0187] Formula (12)

[0188]

[0189] The fifth example comprehensive PM is called the uplink low SINR ratio. The low SINR ratio is the ratio of the cumulative count of uplink SINR values ​​in the range of [-10dB, 2dB] (before outer loop compensation) during the most recent RI to the cumulative count of uplink SINR values ​​in the range of [-10dB, 30dB] (before outer loop compensation). The corresponding KPI is named ULlowSinrRatio and is described in the following formula (13). Cells with ULlowSinrRatio values ​​close to 1 have a large proportion of users experiencing uplink SINR values ​​equal to or below 0dB. It should be noted that formula (13) is calculated when the denominator is greater than zero, otherwise a special value (such as -1, null, etc.) is used to mark the ratio.

[0190] Formula (13)

[0191]

[0192] The sixth example integrated PM is called the weighted uplink SINR. The weighted uplink SINR is the weighted uplink SINR during the latest RI calculated using the uplink SINR measurement (before or after outer loop compensation). The following equation (14) describes the weighted uplink SINR

[0193] Formula (14)

[0194]

[0195] The seventh example comprehensive PM is called the uplink received MCS ratio. The uplink received MCS ratio is the ratio of the cumulative count during the most recent RI when receiving PUSCH with an MCS value between 0 and 5 (inclusive) to the cumulative count when receiving PUSCH with an uplink MCS value between 0 and 28 (inclusive). The corresponding KPI is named ULReceivedMcsRatio and is described in the following formula (15). A ratio close to 1 indicates that a large number of users are provided with uplink MCS values ​​equal to or lower than 5, resulting in fewer bits transmitted per scheduling opportunity. This indicates that the uplink throughput is low. It should be noted that formula (15) is calculated when the denominator is greater than zero, otherwise a special value (such as -1, null, etc.) is used to mark the ratio.

[0196] Formula (15)

[0197]

[0198] The eighth example comprehensive PM is called the downlink scheduler MCS ratio. The downlink scheduler MCS ratio is the ratio of the cumulative count of resource blocks with downlink MCS values ​​between 0 and 5 (inclusive) in the most recent RI to the cumulative count of resource blocks with downlink MCS values ​​between 0 and 28 (inclusive). The corresponding KPI is named DLSchedulerMcsRatio and is described in the following formula (16). A ratio close to 1 indicates that a large number of users are provided with downlink MCS values ​​equal to or lower than 5, resulting in fewer bits transmitted per scheduling opportunity. This indicates that the downlink throughput is low. It should be noted that formula (16) is calculated when the denominator is greater than zero, otherwise a special value (such as -1, null, etc.) is used to mark the ratio.

[0199] Formula (16)

[0200]

[0201] The ninth example comprehensive PM is called weighted downlink MCS. The weighted downlink MCS is a weighted average of the MCS values, obtained by considering the number of times each MCS level was used during the time interval leading to the most recent RI. The corresponding KPI is named WeightedDLScheduledMcs and is described in the following formula (17). It should be noted that formula (17) is calculated when the denominator is greater than zero, otherwise a special value (such as -1, null, etc.) is used to mark the ratio.

[0202] Formula (17)

[0203]

[0204] The tenth example comprehensive PM is called the weighted transmission downlink MCS. The weighted transmission downlink MCS is described in the following equation (18) and includes a weighted average of the modulation and coding scheme used for PDSCH transmission. Small values ​​of the weighted downlink MCS indicate poor radio conditions, resulting in reduced downlink IP throughput. The calculation can be based on a histogram count of the number of occurrences for each MC level, or based on the number of resources (time domain, frequency domain) allocated to each MCS level. Other embodiments may calculate this amount averaged over the codeword and spatial layer domains. It should be noted that equation (18) is calculated when the denominator is greater than zero, otherwise a special value (such as -1, null, etc.) is used to mark the ratio.

[0205] Formula (18)

[0206]

[0207] The eleventh example integrated PM is called weighted downlink CQI. The weighted downlink CQI is described in equation (19) and is calculated by weighted average channel quality, where the weight is proportional to the histogram count for each CQI level between 0 and 15. Small values ​​of weighted CQI indicate poor radio conditions, resulting in reduced downlink IP throughput. In other embodiments, the weight is proportional to the number of radio resources (time domain, frequency domain, and spatial domain) dedicated to transmission of each CQI level.

[0208] Formula (19)

[0209]

[0210] The twelfth example comprehensive PM is called the downlink control channel utilization. The downlink control channel utilization is described in equation (20) and is calculated by the weighted average of the control channel element (CCE) size used to transmit downlink control channel information to the end user. The CCE size varies between 1, 2, 4 and 8, and larger CCE sizes are generally used for users with poor radio quality. Therefore, a weighted average equal to or greater than 4 indicates poor downlink radio quality. Note that other embodiments may include statistics on the percentage of CCE allocation failures for scheduling downlink allocations, uplink grants, etc.

[0211] Formula (20)

[0212]

[0213] The thirteenth example comprehensive PM is called control format utilization. The control format utilization is used to determine the control channel overhead (1, 2 or 3 OFDM symbols), where a larger overhead means reduced resource availability for the data channel. In some embodiments, the CFI ratio can be calculated by the ratio of the histogram count with a CFI value of 3 to the total number of counts, as described in the following equation (21).

[0214] Formula (21)

[0215]

[0216] Pre-processing PM data may also include generating a KPI hierarchy. During each RI, each eNB reports a large number of KPIs. The received KPIs provide detailed information about the health and status of the network between the previous RI and the current RI. The received KPIs may span different protocol layers.

[0217] The KPI tree hierarchy refers to the hierarchical relationship between KPIs. Figure 5b The following table shows the description of different KPIs (e.g. KPI 1 、KPI 2 、KPI 3 、KPI 4 and KPIs 5 ) example KPI tree hierarchy 520 of example hierarchical relationships between ). Figure 5b The different KPIs belong to a certain service category. The root node 521 is the KQI of interest. The service category KQI defines the root node or level 0 KPI. Each KPI of level i provides fine-grained information about the state of the eNB. In some embodiments, the tree may or may not be a binary tree. The levels may be obtained by human domain experts or by a mix of machine learning and EDA tools (e.g., RAPIDMINER).

[0218] The paths linking each parent node to the child nodes of the KPI tree hierarchy 520 are mutually exclusive. For example, node 522 is the parent node of node 524, and node 524 is the child node of node 522. Each child node in the tree corresponds to a KPI or multiple KPIs that satisfy a certain inequality constraint with a confidence exceeding a minimum confidence threshold. In certain embodiments, the minimum confidence threshold is 85%. KPIs that are higher in the tree (such as node 522 compared to node 524) are associated with abnormal signs. The KPIs at the bottom of the tree provide abnormal root cause explanations.

[0219] In some embodiments, the KPI tree hierarchy may include more levels than the two levels shown in KPI tree hierarchy 520. For example, a deeper tree can provide a more detailed explanation, such as listing more symptoms and root causes associated with a KQI anomaly.

[0220] For example, the first level of the tree corresponds to the rules at nodes 522 and 523: The rule at node 522 is based on the relationship And the rule at node 524 is based on the relationship The second level of the KPI tree hierarchy 520 provides root cause explanations. For example, node 524 provides Node 526 provides the root cause explanation when Node 527 provides the root cause explanation when Root cause explanation.

[0221] Figure 5c A flowchart 530 for constructing a KPI tree hierarchy according to various embodiments of the present disclosure is shown. In some embodiments, Figure 4a The RCA framework 400 generates a KPI tree hierarchy, such as the KPI tree hierarchy 520 .

[0222] Once generated, the KPI hierarchy tree explains the underlying root causes and associated symptoms that accompany the anomalies. Typically, the first step involves deriving rules that consist of the form singleton antecedents, subject to a minimum confidence constraint. First, while a base rule may provide high confidence, the accompanying KPI may or may not be the root cause(s) of the anomaly. For example, a rule associating low downlink traffic capacity with an IP throughput anomaly may provide high confidence, but the underlying root cause - causing the low traffic capacity - is either due to poor radio conditions at the user of that cell, or due to no cell accessibility or due to insufficient traffic demand at that cell. Pinpointing the underlying root cause requires a deep understanding of the events that led to the root cause. One way to do this is by linking (e.g., using a logical AND) the antecedents belonging to one or more base rules to form more complex rules. By chaining rules together, the overall confidence of the linked rules is increased. Chained rules also help generate insights into the causal relationships that lead to the IP throughput anomaly. Second, various singleton rules perform poorly in certain specific scenarios. For example, a base rule that provides high confidence May provide a second event KPI 2 <t 2 For example, if the following formula (22) corresponds to the first confidence level, and the following formula (23) corresponds to the second confidence level, according to the probability rule, based on the following formula (24), the overall confidence level may be high on average, but individual items such as The confidence level may be relatively low.

[0223] Formula (22)

[0224]

[0225] Formula (23)

[0226]

[0227] Formula (24)

[0228]

[0229] To generate a KPI tree as described in flowchart 530, is defined as the set of KPIs reported with associated KQI anomalies. That is, for each KQI(s), AN KQI =1, where s∈A KQI The goal is to derive a set of rules R = {r i , i=1,2,...}. For each rule r∈R, let represents the set of all reported KPI data that satisfy the rule antecedent. KPIs can be discretized or continuous. In some embodiments, assume that KPIs are integers ranging from 1 to M. Note that (i) the rule sets in R are mutually exclusive. That is, each PM sample s∈S satisfies either zero rule in R or one rule in R, and (ii) each R i ∈R has a confidence level exceeding a predetermined threshold (such as 85%), and (iii) maximizes the overall hit rate HR based on

[0230] Identify a set of KPIs K = {KPI l , 1≤l≤L}, so that each KPI exceeds the threshold with confidence is the basic rule (i.e., ). In one approach to deriving basic rules, a machine learning algorithm such as association rule mining is applied (such as Figure 4a In some embodiments, a human expert may determine the set of exemplary KPIs for associating a KPI value or value range with a KQI anomaly. In other embodiments, a combination of rule mining and human expert domain knowledge may also be applied.

[0231] In one embodiment, each combination of cascading KPIs may be And the confidence of the chained rules is evaluated to form a chained rule. If the rule does not meet the desired confidence threshold, it is discarded. In some embodiments, if the rule does not meet the desired confidence threshold, the rule and all branches derived from the rule are discarded if q KPIs - q is an algorithm or design parameter - linked to the rule do not meet the confidence threshold. Following this process, the first level of rules in R consists of singleton rules, i.e., rules with a single premise KPI. There are at most possible rules. The second level of rules consists of rules with pairs of KPIs. There are at most possible rules. The third level of rules can be combinations, and so on. Therefore, the maximum number of rules in R is In other embodiments, if a rule does not meet the desired confidence threshold, it is retained until subsequent branches (obtained by linking additional KPIs to the rule) achieve the necessary target confidence. If the rule itself does not meet the desired confidence threshold, such embodiments may include a configurable parameter to control the number of different KPIs linked to the rule.

[0232] An alternative embodiment includes running a rule mining algorithm on progressively smaller data sets. First, as with the previous method, rule mining is used on a large data set (1 or more days) to create basic rules. Once the basic rules are found and sorted using one of the above-mentioned sorting methods, the data set is reduced to include only samples where the basic rules are true. Rule mining is then performed again on this smaller data set to find second-order rules. This process is run in the sorted order of each basic rule. Once second-order rules are found, they can be sorted and the process can be run again. The process will continue until no higher-order rules are found that match the minimum confidence requirements.

[0233] In yet another alternative embodiment, the composite rule is iteratively generated. Flowchart 530 describes this embodiment. For example, the first step includes sorting the KPI group K in order of importance score, from high to low. The importance score can be assigned to the KPI with the largest hit rate starting with the highest importance, and the lowest importance is assigned to the KPI with the smallest hit rate. In an embodiment of the present invention, human domain experts can sort the KPIs in order of the perceived qualitative importance of the KPIs. For example, the highest importance KPI can be set to the availability KPI (or the percentage of time that the cell is available during the time interval leading to the most recent reporting interval), the second highest KPI can be set to the accessibility KPI, and so on.

[0234] For example, the processor evaluates the confidence of each of the following M basic rules: Where KPI1 is the KPI with the highest importance score in K. M basic rules and their confidence values ​​are stored in R. For 1 located between 2 and L and for each r located between R, the rule and all chain rules derived from the rule are discarded from R if the following two conditions are met. The first condition is met when the confidence of the rule is lower than a minimum threshold. The second condition is met when the confidence of the linked rule obtained by linking (all combinations of) the first q KPIs (in order of importance) to the rule is lower than a minimum threshold. Here, q is a configurable parameter between 1 and L-1 (inclusive). For 1 located between 2 and L and for each m located between 1 and M (inclusive), the processor discards the rule KPI by l =m is linked to r to evaluate the confidence of the chain rule. When the chain rule meets the desired confidence threshold, the chain rule and its confidence value are added to R. It is worth noting that pruning

[0235] To generate a KPI tree hierarchy, a set of rules is identified (step 532). In some embodiments, the rule set may be obtained from Figure 4a The machine learning training model 414 recognizes the rules. In other embodiments, the rules are created by human experts. In some embodiments, Figure 4a The machine learning training model 414 is combined with human experts to identify rules. The rules associate a KPI value or value range with a KQI anomaly.

[0236] In step 534, a set of KPIs is identified (collected) = {KPI l , 1≤l≤L}, so that each KPI is a basic rule with a confidence level exceeding a threshold (i.e., ) part.

[0237] In step 536, the group of KPIs is assigned an importance score. The importance score is based on a ranking, where the highest importance is assigned to the KPI with the largest hit rate, and the lowest importance is assigned to the KPI with the lowest hit rate. For example, the highest importance KPI can be set to the availability KPI (or the percentage of time that the cell is available during the time interval leading to the most recent reporting interval), and the second highest KPI can be set to the accessibility KPI, and so on. In other embodiments, human domain experts can sort the KPIs in order of their perceived qualitative importance. In step 538, the KPIs are organized into a list-based list. The order of the KPIs is in descending order of their importance scores. In some embodiments, the KPI list is stored in a rule database.

[0238] In step 540, for each rule, the KPIs are iteratively linked (or chained) starting from the KPI with the highest importance to form the changed rule. In some embodiments, after the KPIs are iteratively linked, the rules with confidence scores below the threshold are removed. The rules are removed after the KPIs are iteratively changed because the confidence of the chained rules may exceed the desired threshold if additional KPIs are subsequently linked to the rule.

[0239] In step 542, if (i) the confidence score of the rule is below a threshold and (ii) the confidence score of the chain rule obtained by linking the previous KPI to the rule is below a threshold, the rule and all chain rules derived from the rule are removed from the tree. For example, starting from each leaf node j, the processor checks whether the support, confidence, and hit rate of the composite rule ending at j are the same as those obtained using the composite rule ending at the parent node of j. All leaf nodes for which the above are TRUE are discarded. In step 544, the chain rules are stored in a memory. In some embodiments, a root cause explanation is generated for each chain rule.

[0240] although Figure 5c An example of a flowchart 530 for constructing a KPI tree hierarchy is shown, but may be used for Figure 5c For example, although shown as a series of steps, Figure 5c The steps in can overlap, occur in parallel, or occur any number of times. As another example, a similar process can be used to create a hierarchical tree for warning-based rules.

[0241] Figure 5d An example output 550 of a KPI tree hierarchy according to various embodiments of the present disclosure is shown. Given an example of a tree 552, a rule such as rule 554 may be associated therewith.

[0242] Pre-processing may also include jointly processing alerts and PM data. It is important to note that alerts are events triggered based on malfunctions at the eNB, while PM data is received on a per-cell basis and at specific intervals. Alerts are critical for troubleshooting and diagnosing hardware and software issues that can lead to degraded network KQI. Therefore, jointly processing alerts and PM data provides the ability to interpret and predict the occurrence of PM anomalies based on the occurrence of alerts to the eNB.

[0243] In certain embodiments, to extract information about the relationship between PM anomaly occurrences and warning occurrences, a time domain correlator processes the time-stamped PM anomaly data and warning data to determine time instances when they occurred together or close to each other.

[0244] In certain embodiments, correlations are presented on a user interface that depict the distribution of the number of occurrences of different types of alerts associated with each anomaly category. For example, the user interface includes an analysis that provides a temporal distribution of alert and anomaly occurrences over a specific duration (e.g., the past hour, day, etc.). End users can use this information to determine the most important alerts that have a service level impact and further determine the root cause of the alert occurrence.

[0245] In some embodiments, there is a causal relationship between multiple alerts. For example, if alert A is activated at time t=T, the causal relationship may indicate the probability that alert B is also activated at t=T. For another example, if alert A is activated at time t=T, the causal relationship may indicate the probability that alert B is activated at a pre-specified time before alert A occurs. Similarly, if alert A is activated at time t=T, the causal relationship may indicate the probability that alert B will be activated at a pre-specified time in the future (after alert A occurs).

[0246] Figure 4a The RCA framework 400 may use correlations between alerts to derive information such as whether the occurrence of a low severity alert can be used to predict the occurrence of a high severity alert within a future time interval.

[0247] In some embodiments, anomalies with timestamps outside a specific time interval may be filtered out when an associated warning occurs and anomalies occur. Thus, the remaining anomalies will have at least one warning (e.g., a single RI) within that time interval, such as Figure 5e After that, the number of warnings within 1 RI of KQI abnormality is counted. For example, Figure 5e A diagram 560 for associating KQI anomalies and warning data according to various embodiments of the present disclosure is shown. As shown in diagram 560, the first 4 warnings have 2 anomalies within one RI, while the remaining 8 warnings have only 1 anomaly within the 1 RI in which they occurred. When the temporal correlation between warnings and KQI anomalies is identified, the warning occurrence provides a root cause explanation for the occurrence of the KQI anomaly. In some embodiments, the user interface can display the relevant output of warnings and KQI anomalies within a time range (e.g., a twenty-four hour period).

[0248] That is, the processor collates warning occurrences with KQI anomalies that occur within a specific time interval of warnings (such as a reporting interval). KQI anomalies with timestamps outside of the warning interval (such as a reporting interval) are not collated with warnings.

[0249] In certain embodiments, Figure 4aThe RCA framework 400 can use machine learning to identify rules that include warnings and one or more KPIs as antecedents and KQI anomalies as consequences. For example, PM data associated with a warning may cause a KQI anomaly at a later point in time. By associating PM data with warning data, Figure 4a The RCA framework 400 is able to preemptively detect KQI anomalies based on alert occurrences.

[0250] After sorting and regularization, machine learning algorithms (such as association rule mining, random forest) are used to identify rules. Rules can be That is, whenever a warning occurs, if the confidence level is greater than a threshold, it indicates that a KQI abnormality has occurred.

[0251] Warning data includes (i) the timestamp when the warning was triggered and (ii) the timestamp when the warning was cleared. Warnings are cleared by a software routine that detects that the initial problem has self-corrected (perhaps due to a system restart) or by a technician working to fix the underlying problem. Regardless of the manner in which the warning is cleared, the time at which the warning is cleared is useful information for determining the root cause of the problem. For example, major problems often have warnings that last for a long time or warnings that recur soon after being cleared. Therefore, embodiments of the present disclosure consider PM reports within 1 RI from the start time to the clearing time to be related to the warning. In certain embodiments, a correlation measurement can be performed between PM metrics and warnings of interest. PM metrics with high correlation coefficients (normalized correlation coefficients close to 1 in magnitude) are identified as being related to warnings.

[0252] Figure 5f A diagram 565 is shown of time-ordered regularization of warnings and PM data according to various embodiments of the present disclosure. Diagram 565 depicts a timeline 566a that increases in time to the right. Each marker 566b indicates a received PM sample. PM samples may be received periodically at certain predetermined time intervals 567 (e.g., every fifteen minutes). For example, each marker 566b is placed fifteen minutes apart along the timeline 566a. Mark 568a identifies the start of a warning and mark 566b indicates the time when the warning is cleared. The processor associates PM data on a range 569. Range 569 starts at a predetermined time before the start of the warning (as shown by marker 568a) and ends at the end of the warning (as shown by marker 568b). In some embodiments, the predetermined time may be 1 RI before the start of the warning. Thus, the processor is able to associate PM data before and during the warning.

[0253] Once the PM data and the sorting rules for each alert type are complete, it is useful to know whether the occurrence of an alert is a good predictor or a sign of an abnormal KQI. One way to do this is to define rules That is, whenever an alert is activated, the rule declares that a KQI anomaly exists. The confidence of this rule (as described in equation (7)) refers to the fraction of alert samples that are also KQI anomalies. If the score is close to 1, the rule is reliable, so the ordered regularized alerts are accompanied by KQI anomalies. In some embodiments, a machine learning algorithm (such as association rule mining, decision trees, etc.) is applied to the set of ordered regularized (timestamped) alerts and corresponding KQI anomalies (taking values ​​0 or 1). The machine learning algorithm identifies a set of alerts whose occurrence leads to KQI anomalies with high confidence and stores them in a database or memory.

[0254] Figure 5g A flowchart 570 for processing alerts and PM data using historical data is shown according to various embodiments of the present disclosure. Flowchart 570 describes processing alerts and PM data using historical data to generate rules based on derived relationships between PM data and alerts. Flowchart 570 may be composed of Figure 4a The RCA framework 400 is implemented.

[0255] In step 572, the RCA framework 400 receives Figure 4a The historical data 410 loads PM data and warning data. In step 574, the RCA framework 400 sorts and regularizes (arranges) the warning data and PM data. For example, the RCA framework 400 identifies the start time and end time (warning clearing time) of a single warning. The RCA framework 400 then identifies a subset of PM data that occurs during the warning period. Since the warning time is a fixed time, the RCA framework 400 identifies a subset of PM data that occurs during the warning period. In some embodiments, the RCA framework 400 identifies PM data that occurs during the warning period and PM data that occurs at a predetermined time interval before the warning. The RCA framework 400 can identify a subset of PM data for the identified warning data.

[0256] In step 576, the RCA framework 400 generates rules based on the subset of PM data and the corresponding alert data. The rules include alerts or KPIs as antecedents and KQI anomalies as consequences. The rules can be Figure 4a The machine learning training model 414 is generated. In step 578, the generated rules are stored in a memory (such as Figure 3a Memory 360 or Figure 3b In some embodiments, only rules having confidence values ​​above a threshold are stored in the memory.

[0257] although Figure 5g One example of a flow chart 570 for processing warnings and PM data using historical data is shown, but may be used for Figure 5g For example, although shown as a series of steps, Figure 5g The steps in a process can overlap, occur in parallel, or occur any number of times.

[0258] Rule mining algorithms often have difficulty creating rules for sets of items that occur infrequently. Since exceptions are not as common as normal operations, and warnings are even less common than exceptions, this creates a problem when using rule mining to find the relationship between warnings and exceptions. To promote the success of the rule miner, embodiments of the present disclosure regularize the ordering of warnings with exceptions rather than with all reported samples. This exaggerates the correlation between rare warning events, making it easier for rule mining algorithms to detect anomalies. In addition, the rules can be validated by testing all discovered rules against historical batch data from multiple days and multiple locations.

[0259] In certain embodiments, Figure 5h Compared to the historical data described in , correlating warnings and PM data can occur in real time. Figure 5h A flow chart 580 is shown for real-time sorting regularization and joint processing of alert and PM data in accordance with various embodiments of the present disclosure.

[0260] To correlate warnings and PM data in real time, first an eNB warning is received (step 582). In step 584, the processor marks the most recent PM data as being regularized with the warning time. The processor then maintains a memory of which eNBs currently have active warnings and which warnings are active. Prior to receiving a warning clear signal, the processor marks all PM data received from the eNB as being regularized with the warning signal. For example, based on correlation measurements derived from historical (batch) data, the memory can store a list of PM metrics associated with the warning signal for each warning.

[0261] Upon receiving the warning signal, the processor loads from memory the stored PM metrics identified as being associated with the warning and tags the data of these PM metrics as being regularized in accordance with the warning signal. In step 586, the processor loads a rule containing warnings as antecedents and KQI anomalies as consequences. For example, the loaded rule includes at least one antecedent including a warning time and a consequence including a KQI.

[0262] In step 588, the processor confirms that the warning occurrence is accompanied by the occurrence of the KQI anomaly. After confirming that the warning occurrence is accompanied by the occurrence of the KQI anomaly, the processor declares the warning occurrence in step 590. Then, when the PM metric satisfies at least one rule that includes a KQI anomaly as a result, the corresponding related PM metric is considered to be the root cause of the KQI anomaly. In some embodiments, the processor then displays the warning and KPI within the premise of the rule as the root cause of the KQI anomaly.

[0263] For example, the processor will load a rule containing an alert as an antecedent and a KQI exception as a result. If an alert occurs and is also accompanied by a KQI exception, the processor can declare that the alert occurred and the corresponding PM metric is the root cause of the KQI exception occurrence.

[0264] although Figure 5h One example of a flow chart 580 for real-time sorting regularization and joint processing of warning and PM data is shown, but may be used for Figure 5h For example, although shown as a series of steps, Figure 5h The steps in a process can overlap, occur in parallel, or occur any number of times.

[0265] Figure 6 FIG. 6 is a diagram showing a process 600 for generating a root cause explanation according to various embodiments of the present disclosure. The process 600 may be performed by Figure 4a The RCA framework 400 is implemented. Figure 6 The embodiments are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0266] Process 600 includes steps similar to Figure 4a The batch processor 602 of the batch processing layer 406 of the process 600 is provided. The batch processor 602 includes a knowledge base 604, a machine learning engine 606, and a filter 610. The batch processor 602 receives historical PM data and alarm data to generate rules. The generated rules are used to diagnose the root cause. The process 600 also includes a memory 612 for storing the generated rules. In some embodiments, the memory stores the generated rules that are above the confidence threshold. The process 600 also includes a rule discovery engine 614 for processing real-time anomaly data 616. When a rule is found that the real-time anomaly data 616 satisfies, a root cause explanation 620 is output. When no rule is found, the post-processing engine 624 attempts to generate a new rule stored in the memory 612.

[0267] The knowledge base 604 maintains various categories for diagnosing uplink and downlink throughput anomalies. The first category is for degraded uplink (coverage holes). Signs or effects of degraded uplink (coverage holes) include a large proportion of users transmitting at maximum power, use of high MCS <= 5 (QPSK with low coding rate) for uplink, and insufficient scheduling of uplink traffic. Throughput anomalies belonging to the degraded uplink category are due to poor radio quality on the uplink caused by coverage holes.

[0268] The second category is based on a degraded uplink (RF interference). Signs or effects of a degraded uplink (RF interference) include high uplink RSSI and interference power, high use of uplink MCS <= 5 (QPSK with low coding rate), and insufficient uplink traffic scheduling. Throughput anomalies belonging to this category are due to high levels of uplink radio frequency interference (RFI). One method for detecting these anomalies is that the comprehensive KPI ULlowSinrRatio takes a value close to 1 for abnormal samples, because high RFI on the uplink will cause an increase in the number of uplink SINR values ​​equal to or below 2dB. In another method for detecting these anomalies, the ratio metric ULSchedulerMcsRatio takes a value close to 1 for abnormal PM samples. This is because high RFI on the uplink causes a large proportion of MCS values ​​to be in the range between 0 and 5 (corresponding to QPSK modulation with low coding rate).

[0269] The third category is based on degraded downlink (RF interference). Signs or effects of a degraded downlink (RF interference) include reported downlink channel quality less than 4, high use of uplink MCS less than or equal to 5 (QPSK with low coding rate), and insufficient uplink traffic scheduling. Throughput anomalies that fall into this category are due to high levels of downlink radio frequency interference (RFI). A degraded downlink can result in the loss of uplink scheduling grants carried over the physical downlink control channel (PDCCH). Due to the loss of scheduling grants, the UE does not transmit its uplink data as instructed by its serving cell. A degraded downlink may also result in lost RLC-AM acknowledgments (generated in response to RLC packets on the uplink) and possible loss of TCP acknowledgments. One method for detecting this anomaly is that the weighted downlink channel quality obtained from the KPI WeightedDLReceivedCQI can be used to assess the downlink radio channel quality. The values ​​of WeightedDLReceivedCQI for abnormal samples tend to be equal to or lower than 4, while the KPI values ​​for normal samples range from 0 to 15. Another method used to detect such anomalies is to use the ratio metric DLSchedulerMcsRatio to take values ​​close to 1 for abnormal PM samples. The reason is that high RFI on the downlink usually results in a large proportion of MCS values ​​in the range between 0 and 5 (corresponding to QPSK modulation with low coding rates).

[0270] The fourth category is based on degraded H-ARQ block error rate. Signs or effects of this category include an uplink H-ARQ block error rate greater than 50% and a downlink H-ARQ block error rate greater than 50%. Throughput anomalies that fall into this category are due to high block error rates in the uplink and / or downlink.

[0271] The fifth category is based on low uplink traffic demand or cell unavailability. In this category, the signs or effects include uplink PRB utilization close to 0, possible availability KQI anomalies, low traffic capacity, and few uplink active users. Anomalies belonging to this RCA category indicate low demand for uplink traffic. They are accompanied by low PRB utilization, few uplink active users, and possible availability anomalies.

[0272] The sixth category is based on low downlink traffic demand or cell unavailability. In this category, symptoms or effects include downlink PRB utilization close to 0, possible availability KQI anomalies, and low traffic capacity and a small number of downlink active users. Anomalies belonging to this RCA category indicate low demand for uplink traffic. They are accompanied by low PRB utilization, a small number of downlink active users, and possible availability anomalies.

[0273] Batch processor 602 loads historical (time-stamped) data via historical data 410a. Historical data 410a can be Figure 4a The historical data 410a includes the KPIs and reported warning occurrences for each RI. The output of the batch processor 602 can be a continuous value or a discrete value. The PM data is labeled by KQI service category based on the output from the anomaly detector 412a, indicating whether an anomaly is observed for that category (i.e., an anomaly is detected for KQI category m, Otherwise 0).

[0274] The machine learning engine 606 (such as association rule mining, decision trees, etc.) generates rules that associate one or more KPIs or alerts with KQI anomalies. A rule is a relationship that associates an outcome (KQI anomaly occurrence) with a cause. The machine learning engine 606 can apply a machine learning algorithm to the pre-processed PM data to generate a set of rules that associate a KPI (or antecedent) with the KQI in question (outcome). An example of a rule that associates two KPIs A and B with a KQIC could be For RCA, antecedents correspond to one or more KPIs that are root causes; the outcome is a Boolean variable equal to true whenever a KQI is anomaly. In certain embodiments, rules are generated by rule mining.

[0275] The filter 610 uses the above equations (6), (7) and (8) to identify the generated rules that meet various thresholds through the machine learning engine 606. For example, equation (7) describes the confidence that a portion of the abnormal sample obeys the rule. The hit rate of equation (8) describes the portion of the KQI associated with the rule that is abnormal. Once the rules are generated, they are output from the batch processor 602 and stored in the memory 612 after meeting the design goals such as minimum support and confidence thresholds. For example, the generated rules can be compared with the confidence threshold and the rules can be saved to the memory 612 based on the comparison results.

[0276] The rule discovery engine 614 receives real-time anomaly data 616. The real-time anomaly data 616 may include input stream KPI (one sample per eNB per RI) data and alert data regularized with timestamp ordering of PM anomaly samples.

[0277] Data is labeled for each RI based on whether a KQI anomaly exists and the anomaly category. For each anomaly sample, the rule discovery engine 614 checks the rules from the memory 612 and identifies anomaly samples corresponding to one or more rules.

[0278] When a rule is discovered, a corresponding root cause explanation 620 is provided on a graphical user interface in decision 618. Providing the root cause explanation 620 may trigger a set of remedial actions, such as adjusting remote electrical tilt at the eNodeB.

[0279] When the rule discovery engine 614 does not find a new rule, the process 600 can perform post-processing of the PM and alert data to identify a new rule in decision 618. When a new rule is identified 626, the new rule is stored in the memory 612.

[0280] although Figure 6 One example of a process 600 for generating a root cause explanation is shown, but may be used for Figure 6 For example, although shown as a series of steps, Figure 6 The steps in a process can overlap, occur in parallel, or occur any number of times.

[0281] Figure 7 An example decision tree 700 for correlating root cause analysis for a certain KQI anomaly according to various embodiments of the present disclosure is shown. Figure 6 The embodiments are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0282] The decision tree 700 is a flow chart-like tool in which each internal node "tests" an attribute under a condition. Each branch represents the result of the test, and the path from the root to the leaf represents the rule. The decision tree 700 identifies the rules associated with RCA for a specific KQI anomaly. In order to generate a decision tree, such as the decision tree 700, the processor identifies various parameters, such as the maximum tree depth, the minimum sample for splitting, and the minimum sample for leaves. The processor first generates N trees. Next, the processor initializes the rule set R={}. : For each tree n∈N, the processor defines a set P: a set of paths to the leaves of the KQI anomaly. For each path p∈P, the processor converts p into a rule r such that R←R∪{r}. For each r∈R, the processor calculates the confidence, support, and hit rate, and discards r if the minimum threshold (based on confidence, support, and hit rate) is not met.

[0283] Figure 8 An example method 800 for discovering and diagnosing network anomalies is shown. The method 800 may be performed by Figure 1 eNB, Figure 3b eNB 102, Figure 1 UE Figure 3a UE 116 or Figure 4a For ease of explanation, the method 800 is described as comprising Figure 4a The electronic device (such as Figure 3a UE 116 or server) is executed. Figure 8 The embodiments are for illustration only. Other embodiments may be used without departing from the scope of the present disclosure.

[0284] In step 802, the electronic device extracts features based on samples obtained by discretizing KPI data and warning data. In step 804, the electronic device generates a set of rules based on the features. The rules can be generated in real time or based on historical data. The rules indicate an explanation of the anomaly. In other words, these rules can be applied to derive the root cause explanation of the anomaly in the wireless cellular network. The rules can include one of the left hand side (LHS) and the right hand side (RHS). A set of KPIs and thresholds can be applied to the LHS and the RHS. It should be noted that if the KPI in the RHS meets the threshold on the RHS, it corresponds to the abnormal sample, and if the KPI in the LHS meets the threshold on the LHS and if the KPI in the RHS corresponds to the abnormal sample, the value of the KPI in the LHS provides a possible sign or root cause explanation.

[0285] In step 806, the electronic device identifies the first sample in the sample as a normal sample or an abnormal sample. The sample can be real-time KPI and warning data. When the first sample is identified as an abnormal sample, in step 808, the electronic device identifies a first rule in a rule set corresponding to the abnormality of the first sample. The first rule indicates the signs and root causes of the abnormality. In step 810, the electronic device applies the root cause to derive a root cause explanation of the abnormality. The root cause explanation can be based on KPIs associated with the signs and root causes of the abnormality included in the sample. In step 812, the electronic device performs a corrective action based on the first rule to resolve the abnormality.

[0286] although Figure 8 An example of a method 800 for discovering and diagnosing network anomalies is shown, but the invention may be used for Figure 8 For example, although shown as a series of steps, Figure 8 The steps in a process can overlap, occur in parallel, or occur any number of times.

[0287] Although the drawings show different examples of user equipment, various changes may be made to the drawings. For example, the user equipment may include any number of each component in any suitable arrangement. In general, the drawings do not limit the scope of the present disclosure to any particular configuration. In addition, although the drawings show operating environments in which various user equipment features disclosed in this patent document may be used, these features may be used in any other suitable system.

[0288] Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to those skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A device for discovering and diagnosing network anomalies, the device include: a communication interface configured to receive key performance indicator (KPI) data and alert data; and a processor operably connected to the communication interface, the processor being configured to: extracting features based on samples obtained by discretizing the KPI data and the warning data, A set of rules is generated based on the features, wherein a portion of the samples that satisfy the rules corresponds to anomalies, Identify samples as normal or abnormal. In response to identifying the sample as an anomalous sample, identifying one or more rules satisfied by the sample, for one or more rules satisfied by the samples identified as anomalous samples, deriving KPIs associated with symptoms and root causes, deriving a root cause explanation for an anomaly included in the sample identified as an anomaly sample based on the KPIs associated with the symptoms and root causes of the anomaly; and performing a corrective action based on the one or more rules to resolve the anomaly, Wherein, the processor is further configured to: receiving, via the communication interface, an indication of a warning from an eNB; In response to receiving the indication, identifying a set of KPI data received from the eNB between the start of the alert and the clearing of the alert; determining whether a key quality indicator (KQI) anomaly is associated with the alert; and When the KQI is associated with the alert, the set of KPI data and the alert are identified as a root cause of the KQI anomaly.

2. The device according to claim 1, in, To generate the rule set, the processor is configured to: identifying a set of KPI data occurring between a predetermined time before the onset of the alert and the clearing of the alert; and Based on the alert data associated with the alert and the set of KPI data, a new rule is generated to identify the anomaly.

3. The device according to claim 1, in: The apparatus further comprises a memory configured to store historical KPI data and historical warning data; and To generate the rule set, the processor is further configured to: Generate new rules from a set of historical KPI data and warnings from historical warning data, deriving a confidence and a hit rate for the new rule, wherein the confidence is based on the fraction of samples satisfying the new rule that are identified as abnormal samples, and the hit rate is based on the fraction of abnormal samples satisfying the new rule, Compare the confidence and hit rate of the new rule with the confidence threshold and hit rate threshold, respectively, and When the confidence associated with the new rule exceeds a confidence threshold and the hit rate associated with the new rule exceeds a hit rate threshold, the new rule is stored.

4. The device according to claim 1, in: In order to discretize the KPI data, the processor is configured to: Selecting a set of features based on the KPI data, deriving additional features based on a composite KPI feature derived from the set of features based on the KPI data, and discretizing the set of features including KPI data and the composite KPI feature; and The processor is further configured to: combining the discretized KPI data with the warning data, and A KPI hierarchy tree is generated based on a portion of the rule set, wherein the portion of the rule set in the KPI hierarchy tree provides a root cause explanation for the first anomaly.

5. The device according to claim 1, further comprising a display, in, The processor is further configured to display on the display at least one of: the one or more rules corresponding to the anomaly, one or more KPI data associated with the symptoms and root causes of the anomaly, An explanation of the root cause of the anomaly, and Corrective action to resolve the anomaly.

6. The device according to claim 1, in, To identify the one or more rules, the processor is configured to: identifying a portion of the rule set associated with the anomaly having a confidence score above a threshold; dividing the portion of the rule set into a basic rule group and a non-basic rule group; When the confidence score of a rule pair in the non-basic rule group is above a threshold, including a directed edge between the rule pair, wherein the directed edge indicates a relationship between the rule pair and a rule in the basic rule group; and A first set of KPI data associated with a first rule of the one or more rules within the base rule set is identified as not including an incoming edge as a root cause, and a second set of KPI data associated with any remaining rules of the one or more rules is identified as a symptom.

7. The device according to claim 1, in, The processor is further configured to: Organize historical data into multiple groups based on timestamps; identifying parameters associated with the plurality of groups; and Future anomalies are predicted based on the parameters.

8. A method for discovering and diagnosing network anomalies, the method include: Receive key performance indicator (KPI) data and warning data; extracting features based on samples obtained by discretizing the KPI data and the warning data; Generate a set of rules based on the features, wherein a portion of the samples that satisfy the rules corresponds to anomalies; Identify samples as normal or abnormal; In response to identifying the sample as an anomalous sample, identifying one or more rules satisfied by the sample; For one or more rules satisfied by the samples identified as abnormal samples, deriving KPIs associated with symptoms and root causes; deriving a root cause explanation for an anomaly included in the sample identified as an anomaly sample based on the KPIs associated with the symptoms and root causes of the anomaly; and performing a corrective action based on the one or more rules to resolve the anomaly, The method further comprises: receiving an indication of a warning from an eNB; In response to receiving the indication, identifying a set of KPI data received from the eNB between the start of the alert and the clearing of the alert; determining whether a key quality indicator (KQI) anomaly is associated with the alert; and When the KQI is associated with the alert, the set of KPI data and the alert are identified as a root cause of the KQI anomaly.

9. The method according to claim 8, further comprising: include: identifying a set of KPI data occurring between a predetermined time before the alarm started and the alarm cleared; and Based on the alert data associated with the alert and the set of KPI data, a new rule is generated to identify the anomaly.

10. The method according to claim 8, further comprising: include: Store historical KPI data and historical warning data; Generate new rules from a set of historical KPI data and warnings from historical warning data; Deriving a confidence and a hit rate of the new rule, wherein the confidence is based on the fraction of samples satisfying the new rule that are identified as abnormal samples, and the hit rate is based on the fraction of abnormal samples satisfying the new rule; Comparing the confidence and hit rate of the new rule with the confidence threshold and hit rate threshold respectively; and When the confidence associated with the new rule exceeds a confidence threshold and the hit rate associated with the new rule exceeds a hit rate threshold, the new rule is stored.

11. The method according to claim 8, in: Discretizing the KPI data includes: Selecting a set of features based on the KPI data, deriving additional features based on a composite KPI feature derived from the set of features based on the KPI data, and discretizing the set of features including KPI data and the composite KPI feature; and The method further comprises: combining the discretized KPI data with the alert data; and A KPI hierarchy tree is generated based on a portion of the rule set, wherein the portion of the rule set in the KPI hierarchy tree provides a root cause explanation for the first anomaly.

12. The method of claim 8, further comprising displaying on a display at least one of: the one or more rules corresponding to the anomaly, one or more KPI data associated with the symptoms and root causes of the anomaly, An explanation of the root cause of the anomaly, and Corrective action to resolve the anomaly.

13. The method according to claim 8, further comprising: include: identifying a portion of the rule set associated with the anomaly having a confidence score above a threshold; dividing the portion of the rule set into a basic rule group and a non-basic rule group; When the confidence score of a rule pair in the non-basic rule group is above a threshold, including a directed edge between the rule pair, wherein the directed edge indicates a relationship between the rule pair and a rule in the basic rule group; and A first set of KPI data associated with a first rule of the one or more rules within the base rule set is identified as not including an incoming edge as a root cause, and a second set of KPI data associated with any remaining rules of the one or more rules is identified as a symptom.

Images