System and method of monitoring network degradation effects on encrypted data stream

Abstract

In a statistical approach to monitoring network performance, the type and extent of network performance degradation, and the concomitant degradation to end-user service quality, e.g., as quantified by QoE for video traffic, are quantified based solely on transport layer data. The technique combines measured network transport layer data and perceived end-user service quality under various types and severities of network performance degradations. Statistical models of each are paired. The methodology estimates service quality degradation in an operating network solely from transport layer metrics, which are available to the Network Management system, even for encrypted traffic flows, such as the QUIC protocol. The method also indicates the likely root cause of the degradation, which enables the Network Management system to automatically apply corrective actions.

Description

[0001] SYSTEM AND METHOD OF MONITORING NETWORK DEGRADATION EFFECTS ON ENCRYPTED DATA STREAMS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to wireless communications, and in particular to a method of estimating end-user service quality for network traffic transported in encrypted form, from statistical analysis of network transport layer metrics.

[0004] BACKGROUND

[0005] Wireless communication networks are ubiquitous in many parts of the world. These networks continue to grow in capacity and sophistication. To accommodate more users, different types of devices, and different use cases, the technical standards governing the operation of wireless communication networks continue to evolve. The fourth generation (4G) of network standards has been deployed, the fifth generation (5G) is in development and early deployment, and the sixth generation (6G) is in design. With each generation, technological advances improve the capacity and spectral efficiency of the wireless communication system.

[0006] One popular model of wireless communication network is referred to as “cellular,” in which generally fixed base stations (BS) provide wireless communication service to a large plurality of fixed and mobile terminals (User Equipment, or UE) within a geographic area called a “cell.” Mobility management techniques, such as handover and cell reselection, provide for transferring service of a UE from one base station to another in response to radio conditions, such as the relative received signal strength of transmissions from the BSs.

[0007] Numerous signals, techniques, and procedures have been developed to optimize the operation of a wireless communication cell. For example, both the BS and UE include reference signals in their respective transmissions. Reference signals are transmissions of known data patterns, which allow the receiver to quantify parameters of the transmission channel. The BS may perform power control over UEs, to avoid transmissions at power levels greater than necessary to receive the data (which are seen by other UEs as interference). In Minimization of Drive Tests (MDT) procedures, the BS directs UEs to perform various measurements of the radio environment and report the results, along with the UEs’ position, allowing the BS to monitor radio performance over the entire cell. The BS may configure, and dynamically alter, a wide array of design and operating parameters in an attempt to optimize network performance.

[0008] Wireless network parameter design and optimization have been widely analyzed in the industry. Monitoring network performance, and dynamically adjusting network operating parameters, are key activities carried out by network engineers with high frequency. Such monitoring and adjustment may be necessary to ensure service according to a Service Level Agreement (SLA). A multitude of tools and algorithms have been developed to support such monitoring and optimization. However, while such tools were helpful in monitoring Transmission Control Protocol (TCP) traffic and the like, those days are gone. A significant proportion of Internet traffic is today transmitted in an encrypted form, and network monitoring tools designed to operate, in part, by monitoring packet headers, and even payloads, have no visibility into encrypted traffic flows.

[0009] To both demonstrate the problems of network monitoring, and to describe an inventive solution, the present disclosure considers a particular, popular, and growing protocol: the Quick UDP Internet Connections, or QUIC. This focus is solely for the purpose of description, and does not limit aspects of the present disclosure, which may be advantageously applied to many other network protocols. The QUIC protocol is a general-purpose transport layer network protocol initially designed by Jim Roskind at Google. It was implemented and deployed in 2012, and since then most Content Delivery Networks (CDN) of Google and Facebook have adopted this protocol. QUIC provides numerous benefits, such as reduced connection establishment latency, improved congestion control, forward error correction, connection migration, and multiplexing without head-of-line blocking.

[0010] However, QUIC provides very limited data to Network Management systems, used by Mobile Network Operators (NMO) to monitor and dynamically adjust the network. A QUIC session is usually initiated by performing a Transport Layer Security (TLS) key exchange, after which packet payloads are sent encrypted. As a result, ascertaining any metrics related to end- user Quality of Service (QoS) or Quality of Experience (QoE) has been difficult.

[0011] Understanding the QUIC protocol mechanism and ensuring QoS and QoE of QUIC transported traffic is important from the MNO’s perspective. There are no monitoring or analytics tools in the prior art capable of providing this information. Many studies and tools focus on performance metrics and improvement of the protocol. Others have investigated QUIC from a cryptographical perspective. Some attempts have been made to address QoE using machine learning, and there are research groups proposing to classify QUIC streams.

[0012] Video is an increasingly important form of content distributed by wired and wireless communication networks. There exists extensive prior art related to video quality assessment. Optimization tools have been developed for TCP-transported streaming video, applying expert rules to transport patterns. However, these rely on parameters which are unavailable when the video is transmitted in encrypted network traffic flows. The present inventors’ earlier work, developing Machine Learning (ML) models for encrypted data streams using network-wide measurable transport parameters as input, is described in International Patent Application Publication No. WO 2024 / 023554, Data collection and test system for delay-critical services, the disclosure of which is incorporated herein by reference, in its entirety.

[0013] What patents and publications do exist in the prior art are either not applicable to QUIC protocol with encrypted data streams, and / or require predefined assumptions and are not data- driven, and / or do not provide possible root cause analysis from which corrective action may be taken. The tools that are documented in publicly available journals are either too specific, or they use black-box like ML algorithms. The accuracy of these methods is completely uncontrollable, and MNOs have no choice but to use them without any possibility to customize them. Such models require substantial time to develop and train. In a different network, or even under traffic conditions, the models may have to be retrained. In particular, the IEEE Explorer database of technical papers was searched with the query “QUIC QoE,” and it yielded only 10 results. This dearth of information related to a very large application of a decade-old protocol is, in itself, an indicator of the difficulty of monitoring QoE flows in QUIC. The papers are briefly summarized:

[0014] • Is QUIC becoming the New TCP? On the Potential Impact of a New Protocol on

[0015] Networked Multimedia QoE The authors investigate the impact of QUIC vs. TCP on QoE for web browsing and video streaming from the perspective of a naive end user. This does not reveal how to monitor QoE once QUIC has been selected as the transport protocol.

[0016] • Media QoE Enhancement With QUIC (https: / / ϴϴϴexplore. eee.org / document / 8743223 /

[0017] The authors of this work focus on a best-effort network use case in which cooperative middle nodes are not prerequisites. They demonstrate that the direct communication of the video app and the QUIC network stack can also result in QoE enhancement. This is a modification of the protocol to improve QoE, and is not a data driven solution to monitoring QoE in QUIC video flows.

[0018] • QUICker or not? - an Empirical Analysis of QUIC vs TCP for Video Streaming QoE Provisioning (https: / / ieeexplore.ieee.org / document / 8685913 / ) In this work, a measurement study was conducted for YouTube video streaming in two mobile and two fixed access networks, in which a defined set of videos was streamed back-to-back with QUIC and TCP in randomized order. This is a comparison of TCP and QUIC, and does not suggest how to monitor QoE in QUIC video flows.

[0019] • Inferring Quality of Experience for Adaptive Video Streaming over HTTPS and QUIC (https: / / ieeexplore.ieee.org / docurnent / 91.48208 / ) The authors of this work propose a machine-learning based solution that leverages a random forest classifier for a better QoE inference accuracy. They use network-and-transport layer information to infer QoE factors such as startup delay and stall events. They inspect and collect various possible set of features from the headers of network-and-transport layers. The primary goal here is to investigate a suit of machine learning classifiers that can boost the classification accuracy much further and analyze their performance in different configurations. Only a few metrics are examined, and they did not use nerdstat.

[0020] • Does QUIC Suit Well With Modern Adaptive Bitrate Streaming Techniques? (https: / / ieeexplore.ieee.org / document / 9084148 / ) The authors of this work perform an analysis on various recent adaptive bitrate streaming (ABR) techniques over the QUIC protocol. With a total of 45 hours of streaming video data, the authors computed three QoE metrics: average playback bitrate, total rebuffering duration, and playback smoothness, while streaming over both TCP and QUIC. The focus here was ABR, which is too specific for a general network monitoring tool.

[0021] • QoE Optimization Technique for Media Delivery in 5G Networks (https: / / ieeexplore.ieee.org / document / 9061469 / ) This paper analyzes different requirements and QoE for broadcast / multicast delivery within the 5G-Xcast project. For enhanced QoE, they develop methodologies that take into account features of very different types of traffic in modern mobile network architectures. The authors propose new methodologies for QoE enhancement; however, their approach is not data driven.

[0022] • YouTube QoE Estimation from Encrypted Traffic: Comparison of Test Methodologies and Machine Learning Based Models (https: / / ieeexplore.ieee.org / document / 8463379 / ) The authors build a dataset to build machine learning based models for estimating QoE and various application-layer KPIs solely from IP-level network traffic features. As such, the approach is applicable in the context of both TLS and QUIC traffic. The study targeted mobile device measurements and a variety of metrics.

[0023] • Real-time Video Quality of Experience Monitoring for HTTPS and QUIC (https: / / ieeexplore.ieee.org / document / 8486321 / ) The authors of this work leverage network and transport layer information as features to train machine learning classifiers for inferring video QoE metrics such as startup delay and rebuffering events. They claim that with the proposed approach, network operators can detect and react to encrypted video QoE impairments in real-time. This work compares HTTPS and QUIC, and infers initial delay, rebuffering events, and video quality only. They do not use live video streams.

[0024] • Empirical study for Dynamic Adaptive Video Streaming Service based on Google Transport QUIC protocol (https: / / ieeexplore.ieee.org / document / 8638062 / ) In this paper, the authors investigate and evaluate the performance of QUIC and traditional transport protocols in the context of video streaming using DASH (Dynamic Adaptive Streaming over HTTP) services. This is a work on Dynamic Adaptive Streaming over HTTP, the effect of DASH parameters is analyzed. This is too specific for a general QoE monitor for QUIC encrypted video flows.

[0025] • Early Online Classification of Encrypted Traffic Streams using Multi-fractal Features (https: / / ieeexplore.ieee.org / document / 8845127 / ) In this paper, the authors present and evaluate a classification framework that combines multi-fractal feature extraction based on time series data (which captures these non-linear characteristics), principal component analysis (PCA) based feature selection, and man-in-the-middle (MITM) based flow labeling. This evaluation shows that the proposed method can quickly and effectively classify traffic belonging to the six most popular traffic types (video streaming, web browsing, social networking, audio communication, text communication, and bulk download) and to distinguish between video-on-demand (VoD) and live streaming sessions delivered from the same services. This work is about machine learning aided classification of Internet traffic, and does not address monitoring QoE.

[0026] Another aspect of some of the present inventors’ prior work in the field is described in European Patent Application EP3821569, Method and system for real-time encrypted video quality analysis, the disclosure of which is incorporated herein by reference, in its entirety. In this application, video quality estimation is made on encrypted video traffic. Only average values of input parameters and also the service quality are considered. Statistical distribution is not taken into account, either at the input or at the output of the model. A method is described for monitoring service quality for a per-user dashboard and present time series QoE data with one of the other network KPIs. No automatic closed loop actions are mentioned. It mentions root cause detection, but it is based on measuring and correlating QoE with other network or node KPIs.

[0027] The Background section of this document is provided to place aspects of the present disclosure in technological and operational context, to assist those of skill in the art in understanding their scope and utility. Approaches described in the Background section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Unless explicitly identified as such, no statement herein is admitted to be prior art merely by its inclusion in the Background section.

[0028] SUMMARY

[0029] The following presents a simplified summary of the disclosure in order to provide a basic understanding to those of skill in the art. This summary is not an extensive overview of the disclosure and is not intended to identify key / critical elements of aspects of the disclosure or to delineate the scope of the disclosure. The sole purpose of this summary is to present some concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

[0030] According to aspects of the present disclosure described and claimed herein, a statistical approach to monitoring network performance quantifies the type and extent of network performance degradation, and the concomitant degradation to end-user service quality, e.g., as quantified by QoE for video traffic, based solely on transport layer data. The technique combines measured network transport layer data and perceived end-user service quality under various types and severities of network performance degradations. Statistical models of each are paired. The methodology estimates service quality degradation in an operating network solely from transport layer metrics, which are available to the Network Management system, even for encrypted traffic flows, such as the QUIC protocol. The method also indicates the likely root cause of the degradation, which enables the Network Management system to automatically apply corrective actions.

[0031] One aspect relates to a method performed by a Network Management system, of monitoring end-user service quality in a wireless communication network. A first phase of the method is performed in a test environment between the wireless communication network and an end-user client, where the Network Management system has access to traffic content. For one or more traffic flows, a plurality of types of performance degradations are introduced into the received traffic flows, each to a plurality of degrees of severity, including zero. For each degree of severity, including zero, of each type of performance degradation, a number N of network transport layer metrics and a number M of end-user service quality metrics are measured; an N- dimensional reference Cumulative Distribution Function (CDF), reflecting a statistical distribution of each of the N network transport layer metrics, is calculated; an M-dimensional reference CDF, reflecting a statistical distribution of each of the M end-user service quality metrics, is calculated; and the reference transport layer CDF, the reference service quality CDF, and the degradation type and severity are associated. A second phase of the method is subsequently performed during operation of the wireless communication network. For one or more traffic flows, the N network transport layer metrics are measured. An N-dimensional operational CDF, reflecting a statistical distribution of each of the N transport layer metrics, is calculated. The operational CDF is compared with each of the reference transport layer CDFs and the reference transport layer CDF closest to the operational CDF is determined. The associated reference service quality CDF is inspected, and predicting values of the M end-user service quality metrics are predicted. If the predicted value of one or more end-user service quality metrics is below a relevant threshold, a corrective action is determined, based on the type and severity of degradation associated with the reference CDFs.

[0032] Another aspect relates to a method, performed by a Network Management system, of monitoring end-user service quality during operation of a wireless communication network. For one or more traffic flows, a number N of network transport layer metrics are measured. An N- dimensional operational CDF, reflecting a statistical distribution of each of the N transport layer metrics, is calculated. A set of N-dimensional reference CDFs reflecting a statistical distribution of each of N network transport layer metrics measured under a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero, and an associated set of M-dimensional reference CDFs reflecting a statistical distribution of each of M end-user service quality metrics measured under the same performance degradations are retrieved. The operational CDF is compared with each of the reference transport layer CDFs and the reference transport layer CDF closest to the operational CDF is determined. The associated reference service quality CDF is inspected, and values of the M end-user service quality metrics are estimated. If the estimated value of one or more end-user service quality metrics is below a relevant predetermined threshold, a corrective action is determined, based on the type and severity of degradation associated with the reference CDFs.

[0033] Yet another aspect relates to a computer implementing at least part of a Network Management system. The computer includes communication circuitry configured to receive and / or transmit traffic flows with a wireless communication network, and processing circuitry processing circuitry operatively connected to the communication circuitry. The processing circuitry is configured to: for one or more received traffic flows, introduce into the received traffic flows a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero; for each degree of severity, including zero, of each type of performance degradation: measure a number N of network transport layer metrics and a number M of end- user service quality metrics; calculate an N-dimensional reference CDF reflecting a statistical distribution of each of the N network transport layer metrics; calculate an M-dimensional reference CDF reflecting a statistical distribution of each of the M end-user service quality metrics; and associate the reference transport layer CDF, the reference service quality CDF, and the degradation type and severity.

[0034] Still another aspect relates to a network node operative in a wireless communication network and implementing at least part of a Network Management system. The node includes communication circuitry configured to receive and / or transmit traffic flows with a wireless communication network, and processing circuitry processing circuitry operatively connected to the communication circuitry. The processing circuitry is configured to: for one or more traffic flows, measure a number N of network transport layer metrics; calculate an N-dimensional operational CDF reflecting a statistical distribution of each of the N transport layer metrics; retrieve a set of N-dimensional reference CDFs reflecting a statistical distribution of each of N network transport layer metrics measured under a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero and an associated set of M-dimensional reference CDFs reflecting a statistical distribution of each of M end-user service quality metrics measured under the same performance degradations; compare the operational CDF with each of the reference transport layer CDFs and determine the reference transport layer CDF closest to the operational CDF; inspect the associated reference service quality CDF and estimate values of the M end-user service quality metrics; and if the estimated value of one or more end-user service quality metrics is below a relevant predetermined threshold, determine a corrective action based on the type and severity of degradation associated with the reference CDFs.

[0035] BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which aspects of the disclosure are shown. However, this disclosure should not be construed as limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like numbers refer to like elements throughout.

[0037] FIG. 1 is a block diagram of a wireless communication network test environment.

[0038] FIG. 2 is a flow diagram of a method of monitoring end-user service quality in a wireless communication network.

[0039] FIG. 3 is a set of graphs depicting Cumulative Distribution Functions (CDF) of QUIC downlink bitrate for different levels of severity of bandwidth limitation.

[0040] FIG. 4 is a set of graphs depicting CDFs of QUIC buffer health for different levels of severity of bandwidth limitation.

[0041] FIG. 5 is a set of graphs depicting CDFs of QUIC connection speed for different levels of severity of bandwidth limitation.

[0042] FIG. 6 is a set of graphs depicting CDFs of QUIC buffer health for different levels of severity of bitstream corruption.

[0043] FIG. 7 is a set of graphs depicting CDFs of QUIC connection speed for different levels of severity of bitstream corruption.

[0044] FIG. 8 is a set of graphs depicting CDFs of QUIC buffer health for different levels of severity of packet loss.

[0045] FIG. 9 is a set of graphs depicting CDFs of QUIC connection speed for different levels of severity of packet loss.

[0046] FIG. 10 is a set of graphs depicting CDFs of QUIC network activity for different levels of severity of packet loss.

[0047] FIG. 11 is a block diagram of an operational wireless communication network.

[0048] FIG. 12 is a flow diagram of a method of monitoring end-user service quality during operation of a wireless communication network.

[0049] FIG. 13 is a hardware block diagram of a computer.

[0050] FIG. 14 is a functional block diagram of a computer.

[0051] FIG. 15 is a hardware block diagram of a network node.

[0052] FIG. 16 is a functional block diagram of a network node.

[0053] DETAILED DESCRIPTION

[0054] For simplicity and illustrative purposes, the present disclosure is described by referring mainly to an exemplary aspect thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be readily apparent to one of ordinary skill in the art that the present disclosure may be practiced without limitation to these specific details. In this description, well known methods and structures have not been described in detail so as not to unnecessarily obscure the present disclosure. The inventive method predicts values of end-user service quality (e.g., QoE) metrics in an operating wireless communication network, based on measuring only network transport layer metrics. Furthermore, the method inherently discovers the probable root cause of sub-par service quality metrics, suggesting automatic corrective actions. As discussed above, the method is applicable to any network traffic and any end-user quality parameters. However, to clearly describe the method, discussion herein focuses on video transported in encrypted QUIC protocol traffic flows. This illustrates a major problem in performance management of modern networks: most traffic is now encrypted, and the Network Management system has very limited visibility into it. The methods described and claimed herein are fully applicable to other types of network traffic, although they are most acutely needed to monitor encrypted traffic flows. For example, the methodology is fully applicable to User Datagram Protocol (UDP) traffic, which is a connectionless protocol (quick, but less reliable), and Transmission Control Protocol (TCP) traffic, which is a connection-based protocol (more reliable, but slower).

[0055] The methodology comprises two parts, or phases. A first phase is performed in a test environment, where the Network Management system has encryption keys, and thus has full access to the encrypted video. FIG. 1 depicts one example of the overall test environment 10. A computer 12, such as consumer laptop or notebook, measures network transport layer metrics and end-user service quality metrics. The computer 12 has sufficiently powerful hardware that neither video encoding nor cryptography constitutes a bottleneck. Internet connection is established using a wireless USB stick 14 that connects to the wireless communication network 16 (e.g., 4G, 5G, etc.). The data plan of the wireless USB stick 14 is unlimited, to ensure that the network provider does not impose limitations on the connection. The computer 12 is connected to a source of video content 18 via the wireless communication network 16, and all routing algorithms or other performance related metrics of the connection establishment are beyond control of the Network Management system. The source of video content 18 may be any video source. YouTube® was selected, according to one aspect of the present disclosure, to take advantage of the “stats for nerds” source of video service quality (QoE) metrics.

[0056] While streaming video from YouTube servers 18, across the wireless communication network 16, and into the computer 12 via the USB stick 14, baseline measurements of network transport layer metrics and end-user service quality metrics are measured and recorded. The network transport layer metrics may be measured in the core network of the wireless communication network 16, as described in greater detail herein, or alternatively by the computer 12. In one aspect of the present disclosure, the computer 12 runs the latest Ubuntu operating system, which is a variant of Unix. Most modern Unix distributions include the tshark utility, which is a network protocol analyzer that can monitor and report various I / O parameters, such as properties of the data flowing into and out of the wireless USB stick 14. A Mozilla Firefox 108.0.1 web browser client is used during measurement when YouTube videos are opened. At each measurement point, a 1 -minute 1080p YouTube video is played, from the same YouTube channel to ensure that the content of the videos is similar. End-user service quality metrics of the decrypted and rendered video are collected using the “stats for nerds” feature of YouTube.

[0057] A number N of network transport layer metrics are measured. These are metrics which are ascertainable for encrypted traffic flows, without decryption keys. In one aspect of the present disclosure, N=3, and the metrics include the number of QUIC packets, bitrate, and interarrival time. Generally, packet size may be a useful network transport layer metric; however, because the QUIC protocol pads all packets to have the same length, packet size does not provide any useful information in this context.

[0058] A number M of end-user service quality metrics are measured. The M end-user service quality metrics are QoE metrics available from the “stats for nerds” utility - which requires that the Network Management system have appropriate decryption keys to decrypt and render the video. In one aspect of the present disclosure, M=6, and the metrics include viewport / frames, connection speed, network activity, buffer health, live latency, and video resolution.

[0059] While the end-user service quality metrics can only be measured in downlink (DL) traffic ( / '.e., data flowing into the computer 12), it is important to measure network transport layer metrics in both the DL and uplink (UL). In general, for any given test session, DL and UL metrics may be measured in the same or in different traffic flows using the same data path between the endpoints. An example of the same flow is a video stream in DL and associated control messages in UL. An example of different flows is an audio-video conference call with an audio and a video flow in both directions. The end-user service quality metrics are measured in the DL for the video stream, and both DL and UL metrics are measured for the voice flow. Measuring UL metrics is important for later monitoring of real-time traffic. As discussed below, in the second phase of the inventive methodology, measurements of network transport layer metrics are taken in the wireless communication network 16. If network traffic degradation occurs on the user side of the observation point, it may not be detectable from the DL metrics alone.

[0060] After a sufficient number of baseline measurements of both network transport layer metrics and end-user service quality metrics have been measured and recorded, a variety of types of performance degradations are introduced into the QUIC video traffic flows into the wireless USB stick 14. The Unix utilities tc (traffic control) and tcset provide the means to control the traffic flows. Performance degradations that can be introduced include, for example, bandwidth limitation, bit corruption, packet loss, packet duplication, and delay. For each type of network performance degradation, and at a plurality of degrees of severity of the performance degradation, both network transport layer metrics and end-user service quality metrics are measured and recorded. The data sets are tagged with the performance degradation type and severity, so that they can be associated. Statistical analyses are performed on both the baseline metrics, and on those measured under each type of performance degradation and at each level of severity. As used herein, a baseline metric may also be described as a metric measured under a performance degradation of zero severity ( / '.e., not enabled). In particular, a Cumulative Distribution Function (CDF) is computed for each metric under each condition. As well known in the art, the CDF of a real- valued random variable X (the measured metric), evaluated at x, is the probability function that X will have a value less than or equal to x. This generates two associated sets of reference CDFs (also referred to as two multi-dimensional CDFs) - a set of reference transport layer CDFs and a set of reference end-user service quality CDFs. For each type of network performance degradation, at each level of severity (including zero, which is the baseline), the reference CDFs from these two sets are associated with each other, and also associated ( / '.e., tagged) with the performance degradation type and severity.

[0061] FIG. 2 depicts a method 100 implementing this first phase of the inventive methodology, according to one aspect of the present disclosure. The method 100 is implemented in a test environment between the wireless communication network 16 and an end-user client (web browser), where the Network Management system performing the method 100 has access to traffic content ( / '.e., it has the relevant decryption keys). An initial performance degradation type is selected, and the severity of that performance degradation is set to zero (block 102). That is, the performance degradation is not applied for the first time through the loop, yielding a baseline set of reference CDFs. Within a nested loop, for each selected type of performance degradation and for each level of severity of the currently selected performance degradation type, N network transport layer metrics and M end-user service quality (e.g., QoE) metrics are measured (block 104). An N-dimensional reference CDF, which reflects the statistical distribution of each of the N network transport layer metrics, is calculated (block 106). An M-dimensional reference CDF, which reflects the statistical distribution of each of the M end-user service quality metrics, is calculated (block 108). The reference transport layer CDF, the reference service quality CDF, and the degradation type and severity are all associated (block 112).

[0062] If all severity levels of the current performance degradation type have not yet been applied (block 114), the severity level is increased (block 116), and new metrics are measured and new reference CDFs are calculated (blocks 104-112). When all desired severity levels of the current performance degradation type have been applied (block 114), the severity level is reset to 0 (block 118). If all desired types of performance degradation have not yet been applied (block 120), the performance degradation type is changed (block 122), and the inner loop (incrementing severity levels) is repeated for the new performance degradation type (blocks 104-118). When all desired types of performance degradation have been applied (block 120), the method 100 ends.

[0063] FIGs. 3-10 depict examples of reference CDFs for both network transport layer metrics and end-user QoE CDFs. Although presented as a set of CDF graphs for each metric for a plurality of degrees of severity of the same applied performance degradation, these individual graphs can each be considered a projection of a multi-dimensional CDF including statistical distributions of all N or M metrics for a single severity level of the same performance degradation.

[0064] FIG. 3 depicts a set of reference CDFs of QUIC downlink (DL) bitrate, in bits / sec. (a network transport layer metric), for each of four levels of severity of bandwidth (BW) limitation, ranging from 500-3000 Kbps.

[0065] FIG. 4 depicts a set of reference CDFs of QUIC buffer health (an end-user service quality (QoE)) metric for the same set of bandwidth limitations.

[0066] FIG. 5 depicts another set of reference service quality CDFs, here the QUIC connection speed, again for four levels of severity of bandwidth restriction.

[0067] FIG. 6 depicts a set of reference CDFs of QUIC buffer health for a different applied network degradation: inflicting bit corruption on the data stream. For example, the command $ tset wlp0s20f3 -direction incoming -corrupt ${ii} was used to generate perturbations from 0- 50.

[0068] FIG. 7 depicts a set of reference CDFs of QUIC connection speed for five levels of severity of bitstream corruption.

[0069] FIG. 8 depicts a set of reference CDFs of QUIC buffer health under another performance degradation: packet loss.

[0070] FIG. 9 depicts a set of reference CDFs of QUIC connection speed for five levels of severity of packet loss.

[0071] FIG. 10 depicts a set of reference CDFs of QUIC network activity under various levels of severity of packet loss.

[0072] A second phase of the inventive methodology is performed during operation of the wireless communication network, where the Network Management system does not (necessarily) have encryption keys to the traffic flows, and thus may have no visibility into, e.g., packet headers. FIG. 11 depicts one example of the wireless communication network 16 during operation. The second phase of the inventive method is performed in the Network Data Analytics Function (NWDAF). During operation, the NWDAF receives traffic probe reports per traffic flow from User Plane Function (UPF) Network Functions (NF), typically with 1s time granularity. These are used for calculating the network transport layer metrics, also known as Key Performance Indicators (KPI). As used herein, the NWDAF “measures” these metrics by receiving the traffic probe reports from the UPF and calculating the metrics. For QUIC video, the network transport layer metrics include: number of QUIC packets, bitrate and interarrival time. In other aspects, instead of UPF node reports, any other network probing can also be used for obtaining the traffic data and calculating these metrics.

[0073] The network transport layer metrics are calculated per flow for a time window, which can be, for example, 10 sec. to 5 min. In some applications, e.g., mobility scenarios, 1 sec. granularity may be applicable as well. The network transport layer metrics can be aggregated to different NFs, cells, gNBs, or slices, service types, user groups, or combination of these, depending on the use cases. In this case, the estimated service quality and degradation refers to the aggregated entity.

[0074] The statistical distribution of the network transport layer metrics is obtained for each aggregation time window. As in the test environment, an N-dimensional CDF is calculated, referred to as the operational CDF. The is compared to all of the reference transport layer CDFs obtained for the network transport layer metrics under different network degradation scenarios in the test environment ( / '.e., during the execution of the method 100). First, it is determined whether there are degradations or not. If the operational CDF, reflecting the actual distributions of the network transport layer metrics, is the same (within a tolerance) as the reference transport layer CDFs created in the test environment at zero severity of any performance degradation, it is assumed that the distribution of the end-user service quality (e.g., QoE) metrics in the operating network is the same as the ones in the test network.

[0075] Network degradation is detected if the operational CDF is different from the severity=0 reference transport layer CDFs. In this case, the closest reference transport layer CDF is determined. This comparison may comprise computing a Euclidean distance between a selected features of the CDF, and selecting the closest reference transport layer CDF as the one with the least Euclidean distance.

[0076] Alternatively, a vector of measurement points or quantiles of the CDFs may be constructed, and a cosine similarity operation applied to determine the closest CDF. The cosine similarity measures the cosine of the angle between two vectors projected in a multi- dimensional space. The idea that similar vectors will likely point in the same direction, so the angle between them will be reduced. When using this method, n-long vectors are formed based on the CDF data points of the N transport metrics (n = N * the number of data points of the 1- dimensional CDF). Formally, comparing the distance of two vectors A and B , and , the cosine distance of these two N dimensional CDFs will be: cosine_distance(A,B) = 1 - cosine_similarity(A,B).

[0077] The smaller the cosine_distance between A and B, the closer the N-dimensional CDFs are to each other.

[0078] Those of skill in the art will recognize that other methods of comparison may be employed, such as training an ML model to map the operational CDF to the closest reference transport layer CDF. Once the closest reference transport layer CDF is determined, both the type of performance degradation and the severity of that degradation are directly indicated (i.e., by which curve matches). With this information in hand, the reference end-user service quality CDFs are inspected. A given degradation type and severity corresponds to one of the curves in the graphs of FIGs. 4-10. From these graphs the end-user service quality metrics - which cannot be directly assessed in the operating environment due to encryption - are estimated. If these estimates fall below respective thresholds - for example, guaranteed values in an SLA - then the degradation type and severity information directly points to a root cause, and suggests network operating parameters that may be adjusted to alleviate the degradation.

[0079] FIG. 12 depicts a method 200 implementing this second phase of the inventive methodology, according to one aspect of the present disclosure. The method 200 is performed by the Network Management system, which monitors network transport layer metrics during operation of a wireless communication network, when in general traffic flows may be encrypted and the Network Management system has no visibility into packets. For one or more selected traffic flows, a number N of network transport layer metrics are measured (block 202). An N- dimensional operational CDF, reflecting the statistical distribution of each of the N transport layer metrics, is calculated (block 204). A set of N-dimensional reference CDFs is retrieved (block 206). The N-dimensional reference CDFs reflect the statistical distribution of each of N network transport layer metrics measured under a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero. For example, the reference transport layer CDFs may be those generated during execution of the method 100 in the test environment 10. Additionally, an associated set of M-dimensional reference CDFs is retrieved (also block 206). The M-dimensional reference CDFs reflect the statistical distribution of each of M end-user service quality metrics measured under the same performance degradations. The operational CDF is compared to each of the reference transport layer CDFs, and the reference transport layer CDF closest to the operational CDF is determined (block 208). From the degradation type and severity (i.e., which curve best matches), the associated reference service quality CDF is inspected, and values of the M end-user service quality metrics are taken as estimates of the current values of these metrics in the operational network (block 210). If the estimated value of one or more end-user service quality metrics is below a relevant threshold, a corrective action is determined, based on the type and severity of degradation associated with the reference CDFs (block 212).

[0080] The methods 100, 200 described herein directly suggest remedial action that can be taken in a wireless communication network when a low end-user service metric is identified, as the mechanism of prediction identifies a particular network performance degradation, at a specific level of severity. According to one aspect of the present disclosure, when degraded service quality is estimated, Network Slice Management Function (NSMF) and / or NWDAF closed loop correct actions are triggered. For example, possible end-to-end (e2e) slice relation actions in the NSMF include: setting up a new slice for the affected traffic flows; modifying the slice of the affected traffic; moving low priority traffic to different slice (to reduce the load of the affected slice); and modifying radio resource partitioning (RRP) among slices. Possible NWDAF actions include: allocating the affected traffic to different 5QI that improves e2e latency; allocating the affected traffic to different 5QI that improves e2e loss; allocating the affected traffic to different 5QI that increases scheduling priority; moving other traffic than the affected one to a lower priority 5QI, which improves the QoS of the affected traffic; applying admission control to affected traffic; applying admission control to other traffic types; and applying traffic shaping to the affected traffic or other traffic types.

[0081] According to some aspects of the present disclosure, the identified root cause of the degraded end-user service quality is used for selecting further actions. For example, if the issue is packet loss and the e2e path and the serving cell is not congested, a corrective action may be to allocate affected traffic to a 5QI which improves e2e latency. As another example, if a slice of the affected traffic is overloaded and there is no QoS or QoE issue in the other slices, the RRP may be modified to increase the capacity of the affected slice. As still another example, if a cell is overloaded by the high volume of the MMB traffic, applying traffic shaping to MMB in that cell may alleviate the degradation. Those of skill in the art will readily recognize other remedial network actions that may automatically be taken based on the detected network performance degradation and its severity, given the teachings of the present disclosure.

[0082] Apparatuses described herein may perform the methods 100, 200, herein and any other processing by implementing any functional means, modules, units, or circuitry. In one aspect of the disclosure, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several aspects of the disclosure. In aspects of the disclosure that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

[0083] FIG. 13 for example illustrates a hardware block diagram of the computer 12 that operates in the test environment for execution of the method 100, as implemented in accordance with one or more aspects of the disclosure. As shown, the computer 12 includes processing circuitry 22 and communication circuitry 26. The communication circuitry 26 is configured to transmit and / or receive information to and / or from one or more base stations in the wireless communication network 16. As depicted in FIG. 1 , the communication circuitry 26 may comprise a wireless USB stick 14 removably connected to a USB port of the computer 12, in which case an antenna 28 extends exteriorly of the computer 12. Alternatively, as indicated by dashed lines, the computer 12 may have integral radio circuitry 26, with an internal antenna 28. The processing circuitry 22 is configured to perform processing described above, such as by executing instructions stored in memory 24. The processing circuitry 22 in this regard may implement certain functional means, units, or modules. When equipped with communication circuitry 26, the computer 12 may be considered a User Equipment (UE) in the wireless communication network 16.

[0084] FIG. 14 illustrates a functional block diagram of a computer 30 according to still other aspects of the disclosure. As shown, the computer 30 implements various functional means, units, or modules, e.g., via the processing circuitry 22 in FIG. 13 and / or via software code. These functional means, units, or modules, e.g., for implementing method 100 herein, include for instance: network performance degrading unit 32; metric measuring unit 34; CDF calculating unit 36; and associating unit 38. Those of skill in the art will readily recognize that some or all of the units 32-38 may be combined in any given implementation, and the organization of FIG. 14 is representative only and not limiting.

[0085] Network performance degrading unit 32 is configured to, for one or more traffic flows, introduce into the received traffic flows a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero. Metric measuring unit 34 is configured to measure a number N of network transport layer metrics and a number M of end-user service quality metrics, at each degree of severity of each type of performance degradation. CDF calculating unit 36 is configured to, for each degree of severity, including zero, of each type of performance degradation, calculate an N-dimensional reference CDF reflecting the statistical distribution of each of the N network transport layer metrics and calculate an M-dimensional reference CDF reflecting the statistical distribution of each of the M end-user service quality metrics. Associating unit 38 is configured to, for each degree of severity, including zero, of each type of performance degradation, associate the reference transport layer CDF, the reference service quality CDF, and the degradation type and severity.

[0086] FIG. 15 illustrates a hardware block diagram of a network node 40 as implemented in accordance with one or more aspects of the disclosure. The network node 40 may implement a NWDAF, or similar analytics function, in the wireless communication network 16. As shown, the network node 40 includes processing circuitry 42 and communication circuitry 46. The communication circuitry 46 is configured to transmit and / or receive information to and / or from one or more other network nodes, e.g., via any communication technology. The processing circuitry 42 is configured to perform processing described above, such as by executing instructions stored in memory 44. Although represented as being within the network node 40, those of skill in the art understand that some or all of the processing circuitry 42 may be implemented as virtualized servers in a data center, e.g., in the so-called cloud. The processing circuitry 42 in this regard may implement certain functional means, units, or modules.

[0087] FIG. 16 illustrates a functional block diagram of a network node 50 in the wireless communication network 16 according to still other aspects of the disclosure, in which the network node 50 implements a NWDAF or similar analytics function. As shown, the network node 50 implements various functional means, units, or modules, e.g., via the processing circuitry 42 in FIG. 15 and / or via software code. These functional means, units, or modules, e.g., for implementing the method 200 herein, include for instance: metric measuring unit 52; operational CDF calculating unit 54; reference CDF retrieving unit 56; transport layer CDF comparing unit 58; operational CDF inspecting unit 60; and corrective action determining unit 62. Those of skill in the art will readily recognize that some or all of the units 52-62 may be combined in any given implementation, and the organization of FIG. 16 is representative only and not limiting.

[0088] Metric measuring unit 52 is configured to, for one or more traffic flows, measure a number N of network transport layer metrics. Operational CDF calculating unit 54 is configured to calculate an N-dimensional operational CDF reflecting the statistical distribution of each of the N transport layer metrics. Reference CDF retrieving unit 56 is configured to retrieve a set of N- dimensional reference CDFs reflecting the statistical distribution of each of N network transport layer metrics measured under a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero. Reference CDF retrieving unit 56 is further configured to retrieve an associated set of M-dimensional reference CDFs reflecting the statistical distribution of each of M end-user service quality metrics measured under the same performance degradations. Transport layer CDF comparing unit 58 is configured to compare the operational CDF with each of the reference transport layer CDFs and determine the reference transport layer CDF closest to the operational CDF. Operational CDF inspecting unit 60 is configured to inspect the associated reference service quality CDF and estimate values of the M end-user service quality metrics. Corrective action determining unit 62 is configured to, if the estimated value of one or more end-user service quality metrics is below a relevant predetermined threshold, determine a corrective action based on the type and severity of degradation associated with the reference CDFs.

[0089] Those skilled in the art will also appreciate that aspects of the disclosure herein further include corresponding computer programs.

[0090] A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above. Aspects of the disclosure further include a carrier containing such a computer program. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0091] In this regard, aspects of the disclosure herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

[0092] Aspects of the disclosure further include a computer program product comprising program code portions for performing the steps of any of the aspects of the disclosure herein when the computer program product is executed by a computing device. This computer program product may be stored on a computer readable recording medium.

[0093] Aspects of the present disclosure present numerous advantages over the prior art. The methodology is applicable to encrypted data streams, for which very limited information is available by the mere inspection of transmitted packets. The methodology recognizes that in case of different network issues, different transport layer metrics are important to estimate the QoE degradation. The methodology inherently suggests root cause identification, which makes it possible for the operator to select and trigger the appropriate corrective action. The methodology is based on a statistical approach, and utilizes mathematical tools from probability theory. Accordingly, it avoids reliance on exotic and heavy-weight ML libraries. The end-user service quality metrics are characterized using technically measurable probabilistic variables, and do not rely on subjective measurements. This enables easy verification of the methodology in different networks. It also facilitates automation of the methodology.

[0094] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the aspects disclosed herein may be applied to any other aspect, wherever appropriate. Likewise, any advantage of any of the aspects may apply to any other aspects, and vice versa. Other objectives, features and advantages of the enclosed aspects will be apparent from the description.

[0095] The term “unit” may have conventional meaning in the field of electronics, electrical devices and / or electronic devices and may include, for example, electrical and / or electronic circuitry, devices, modules, processors, memories, logic solid state and / or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and / or displaying functions, and so on, as such as those that are described herein.

[0096] As used herein, the term “configured to” means set up, organized, adapted, or arranged to operate in a particular way; the term is synonymous with “designed to,” or with respect to processing circuitry, “programmed to.”

[0097] Some of the aspects contemplated herein are described more fully with reference to the accompanying drawings. Other aspects, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the aspects set forth herein; rather, these aspects are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0098] The present disclosure may, of course, be carried out in other ways than those specifically set forth herein without departing from essential characteristics of the disclosure. The present aspects are to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended aspects are intended to be embraced therein.

Claims

CLAIMSWhat is claimed is:1 . A method (100), performed by a Network Management system, of monitoring end-user service quality in a wireless communication network (16), the method (100) characterized by, in a test environment between the wireless communication network (16) and an end-user client, where the Network Management system has access to traffic content: for one or more received traffic flows, introducing (102) into the received traffic flows a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero; for each degree of severity, including zero, of each type of performance degradation: measuring (104) a number N of network transport layer metrics and a number M of end-user service quality metrics; and calculating (106) an N-dimensional reference Cumulative Distribution Function, CDF, reflecting a statistical distribution of each of the N network transport layer metrics; calculating (108) an M-dimensional reference CDF reflecting a statistical distribution of each of the M end-user service quality metrics; and associating (112) the reference transport layer CDF, the reference service quality CDF, and the degradation type and severity.

2. The method (100) of claim 1 , further characterized by, during operation of the wireless communication network (16): for one or more traffic flows, measuring (202) the N network transport layer metrics; calculating (204) an N-dimensional operational CDF reflecting a statistical distribution of each of the N transport layer metrics; comparing (208) the operational CDF with each of the reference transport layer CDFs and determining the reference transport layer CDF closest to the operational CDF; inspecting (210) the associated reference service quality CDF and estimating values of at least one end-user service quality metric; and if the estimated value of at least one end-user service quality metrics is below a relevant predetermined threshold, determining (212) a corrective action based on the type and severity of degradation associated with the reference CDFs.

3. The method (100) of any preceding claim wherein the traffic content is encrypted video, and wherein the service quality metrics are Quality of Experience, QoE, metrics.

4. The method (100) of claim 3 wherein the Network Management system has access to the encrypted video in the test environment.

5. The method (100) of claim 2 wherein the traffic content is encrypted video and wherein the Network Management system does not have access to the video in the operating wireless communication network (16) due to the traffic flows being encrypted.

6. The method (100) of claim 5 wherein the traffic flows conform to the Quick UDP Internet Connections, QUIC, protocol.

7. The method (100) of claim 1 wherein the traffic flows conform to the User Datagram Protocol, UDP.

8. The method (100) of claim 1 wherein the traffic flows conform to the Transmission Control Protocol, TCP.

9. The method (100) of any preceding claim wherein at least some of the N network transport layer metrics are measured in both uplink and downlink.

10. The method (100) of claim 9 wherein the uplink and downlink measurements are taken in the same traffic flow.11 . The method (100) of claim 9 wherein the uplink and downlink measurements are taken in different traffic flows.

12. The method (100) of any preceding claim wherein the network transport layer metrics include one or more of: a number of packets, a bitrate, and an interarrival time.

13. The method (100) of any of claims 3-12 wherein the QoE metrics include one or more of: viewport / frames, connection speed, network activity, buffer health, live latency, and video resolution.

14. The method (100) of any preceding claim wherein the types of performance degradation include one or more of: bandwidth limitation, bit corruption, packet loss, packet duplication, and delay.

15. The method (100) of any preceding claim wherein the test environment comprises a computer (12) with an interface to the wireless communication network and running a Unix or Unix-like operating system.

16. The method (100) of claim 15 wherein performance degradations are introduced into the received traffic flows by use of the tc and tcset commands of the Unix or Unix-like operating system.

17. The method (100) of claim 15 wherein, in the test environment, the network transport layer metrics are measured by use of the tshark network protocol analyzer.

18. The method (100) of claim 3 wherein the video traffic is obtained from YouTube, and the QoE metrics are measured by use of the “Stats for nerds” statistics of a YouTube web application.

19. The method (100) of claim 2 wherein, during operation of the wireless communication network (16), the network transport layer metrics are measured by a User Plane Function of a Core Network of the wireless communication network and the operational CDFs are calculated in a Network Data Analytics Function of the wireless communication network.

20. The method (100) of any preceding claim wherein comparing (208) the operational CDF with each of the reference transport layer CDFs and determining the reference transport layer CDF closest to the operational CDF comprises computing a Euclidean distance between a selected features of the CDF, and selecting the closest reference transport layer CDF as the one with the least Euclidean distance.21 . The method (100) of any of claims 1-19 wherein comparing (208) the operational CDF with each of the reference transport layer CDFs and determining the reference transport layer CDF closest to the operational CDF comprises constructing a vector of measurement points or quantiles of the CDFs and performing a cosine similarity operation.

22. A method (200), performed by a Network Management system, of monitoring end-user service quality during operation of a wireless communication network (16), the method (200) characterized by: for one or more traffic flows, measuring (202) a number N of network transport layer metrics; calculating (204) an N-dimensional operational Cumulative Distribution Function, CDF, reflecting a statistical distribution of each of the N transport layer metrics;retrieving (206) a set of N-dimensional reference CDFs reflecting a statistical distribution of each of N network transport layer metrics measured under a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero and an associated set of M-dimensional reference CDFs reflecting a statistical distribution of each of M end-user service quality metrics measured under the same performance degradations; comparing (208) the operational CDF with each of the reference transport layer CDFs and determining the reference transport layer CDF closest to the operational CDF; inspecting (210) the associated reference service quality CDF and estimating values of the M end-user service quality metrics; and if the estimated value of one or more end-user service quality metrics is below a relevant predetermined threshold, determining (212) a corrective action based on the type and severity of degradation associated with the reference CDFs.

23. The method (200) of claim 22 wherein the traffic flows for which network transport layer metrics are measured carry video, and wherein the service quality metrics are Quality of Experience, QoE, metrics.

24. The method (200) of claim 23 wherein the traffic flows conform to the Quick UDP Internet Connections, QUIC, protocol.

25. The method (200) of claim 22 wherein the traffic flows conform to the User Datagram Protocol, UDP.

26. The method (200) of claim 22 wherein the traffic flows conform to the Transmission Control Protocol, TCP.

27. The method (200) of any preceding claim wherein at least some of the N network transport layer metrics are measured in both uplink and downlink.

28. The method (200) of claim 27 wherein the uplink and downlink measurements are taken in the same traffic flow.

29. The method (200) of claim 27 wherein the uplink and downlink measurements are taken in different traffic flows.

30. The method (200) of any of claims 22-29 wherein the network transport layer metrics include one or more of: a number of packets, a bitrate, and an interarrival time.31 . The method (200) of claim 23 wherein the QoE metrics include one or more of: viewport / frames, connection speed, network activity, buffer health, live latency, and video resolution.

32. The method (200) of any of claims 22-31 wherein the types of performance degradation include one or more of: bandwidth limitation, bit corruption, packet loss, packet duplication, and delay.

33. A computer (12, 30) implementing at least part of a Network Management system, characterized by: communication circuitry (26) configured to receive and / or transmit traffic flows with a wireless communication network (16); and processing circuitry (22) operatively connected to the communication circuitry (26), the processing circuitry (22) configured to: for one or more traffic flows, introduce (102) into the received traffic flows a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero; for each degree of severity, including zero, of each type of performance degradation: measure (104) a number N of network transport layer metrics and a number M of end-user service quality metrics; and calculate (106) an N-dimensional reference Cumulative Distribution Function, CDF, reflecting a statistical distribution of each of the N network transport layer metrics; calculate (108) an M-dimensional reference CDF reflecting a statistical distribution of each of the M end-user service quality metrics; and associate (110) the reference transport layer CDF, the reference service quality CDF, and the degradation type and severity.

34. The computer (12, 30) of claim 33 wherein the traffic content is encrypted video, and wherein the service quality metrics are Quality of Experience, QoE, metrics.

35. The computer (12, 30) of claim 34 wherein the Network Management system has access to the video.

36. The computer (12, 30) of claim 34 wherein the traffic flows conform to the Quick UDP Internet Connections, QUIC, protocol.

37. The computer (12, 30) of claim 33 wherein the traffic flows conform to the User Datagram Protocol, UDP.

38. The computer (12, 30) of claim 33 wherein the traffic flows conform to the Transmission Control Protocol, TCP.

39. The computer (12, 30) of any of claims 33-38 wherein the processing circuitry is configured to measure at least some of the N network transport layer metrics in both uplink and downlink.

40. The computer (12, 30) of claim 39 wherein the processing circuitry is configured to take uplink and downlink measurements in the same traffic flow.41 . The computer (12, 30) of claim 39 wherein the processing circuitry is configured to take uplink and downlink measurements in different traffic flows.

42. The computer (12, 30) of any of claims 33-41 wherein the network transport layer metrics include one or more of: a number of packets, a bitrate, and an interarrival time.

43. The computer (12, 30) of any of claim 34 wherein the QoE metrics include one or more of: viewport / frames, connection speed, network activity, buffer health, live latency, and video resolution.

44. The computer (12, 30) of any of claims 33-43 wherein the types of performance degradation include one or more of: bandwidth limitation, bit corruption, packet loss, packet duplication, and delay.

45. The computer (12, 30) of any of claims 33-44 wherein the processing circuitry is further configured to run a Unix or Unix-like operating system.

46. The computer (12, 30) of claim 45 wherein the processing circuitry is configured to introduce performance degradations into the received traffic flows by use of the tc and tcset commands of the Unix or Unix-like operating system.

47. The computer (12, 30) of claim 45 wherein the processing circuitry is configured to measure network transport layer metrics by use of the tshark network protocol analyzer.

48. The computer (12, 30) of claim 34 wherein the video traffic is obtained from YouTube, and wherein the processing circuitry is configured to measure the QoE metrics by use of the “Stats for nerds” statistics of a YouTube web application.

49. The computer (12, 30) of any of claims 33-48 wherein the processing circuitry is configured to compare (208) the operational CDF with each of the reference transport layer CDFs and determine the reference transport layer CDF closest to the operational CDF by computing a Euclidean distance between a selected features of the CDF, and selecting the closest reference transport layer CDF as the one with the least Euclidean distance.

50. The computer (12, 30) of any of claims 33-48 wherein the processing circuitry is configured to compare (208) the operational CDF with each of the reference transport layer CDFs and determine the reference transport layer CDF closest to the operational CDF comprises constructing a vector of measurement points or quantiles of the CDFs and performing a cosine similarity operation.51 . A network node (40, 50) operative in a wireless communication network (16) and implementing at least part of a Network Management system, characterized by: communication circuitry (46) configured to communicate with other network nodes; and processing circuitry (42) operatively connected to the communication circuitry (46), the processing circuitry (42) configured to: for one or more traffic flows, measure (202) a number N of network transport layer metrics; calculate (204) an N-dimensional operational Cumulative Distribution Function, CDF, reflecting a statistical distribution of each of the N transport layer metrics; retrieve (206) a set of N-dimensional reference CDFs reflecting a statistical distribution of each of N network transport layer metrics measured under a plurality of types of performance degradations, each to a plurality of degrees of severity, including zero and an associated set of M- dimensional reference CDFs reflecting a statistical distribution of each of M end-user service quality metrics measured under the same performance degradations;compare (208) the operational CDF with each of the reference transport layer CDFs and determine the reference transport layer CDF closest to the operational CDF; inspect (210) the associated reference service quality CDF and estimate values of the M end-user service quality metrics; and if the estimated value of one or more end-user service quality metrics is below a relevant predetermined threshold, determine (212) a corrective action based on the type and severity of degradation associated with the reference CDFs.

52. The network node (40, 50) of claim 51 wherein the traffic flows for which network transport layer metrics are measured carry encrypted video, and wherein the service quality metrics are Quality of Experience, QoE, metrics.

53. The network node (40, 50) of claim 52 wherein the traffic flows conform to the Quick UDP Internet Connections, QUIC, protocol.

54. The network node (40, 50) of claim 51 wherein the traffic flows conform to the User Datagram Protocol, UDP.

55. The network node (40, 50) of claim 51 wherein the traffic flows conform to the Transmission Control Protocol, TCP.

56. The network node (40, 50) of any of claims 51-55 wherein at least some of the N network transport layer metrics are measured in both uplink and downlink.

57. The network node (40, 50) of claim 56 wherein the uplink and downlink measurements are taken in the same traffic flow.

58. The network node (40, 50) of claim 56 wherein the uplink and downlink measurements are taken in different traffic flows.

59. The network node (40, 50) of any of claims 51 -58 wherein the network transport layer metrics include one or more of: a number of packets, a bitrate, and an interarrival time.

60. The network node (40, 50) of claim 52 wherein the QoE metrics include one or more of: viewport / frames, connection speed, network activity, buffer health, live latency, and video resolution.61 . The network node (40, 50) of any of claims 51 -60 wherein the types of performance degradation include one or more of: bandwidth limitation, bit corruption, packet loss, packet duplication, and delay.

Images