Estimating quality of experience for cloud gaming service

Abstract

A machine learning model and method of training a machine learning model is provided to predict the QoE of a cloud gaming service, defined as in-game performance, based on measurable, QoS metric in the communication network. During training, an objective, measurable, in-game performance metric is used as a target. QoS metrics covering different degradation scenarios ranging from good network conditions to poor network conditions are collected and used as training data. A selected set of measurable QoS metrics predictive of in-game performance is determined by comparing the performance of a constrained QoS model trained using only selected QoS metrics measurable by in the network with the performance of a reference QoS model trained using all significant QoS metrics. In cases where the constrained QoS model provides similar performance to the reference QoS model, the selected set of measurable QoS metrics is validated. The constrained QoS model can be deployed in the network to estimate QoE of the cloud-gaming service and to manage network resources to ensure that QoS guarantees are met. Additionally, the QoS model can be used to provide a performance boosting service to users of the cloud-gaming service.

Description

[0001] ESTIMATING QUALITY OF EXPERIENCE FOR CLOUD GAMING SERVICE TECHNICAL FIELD The present disclosure relates generally to Quality of Service (QoS) assurance for a cloud gaming service delivered over a wireless communication network and, more particularly, to methods of training machine learning models to predict quality of experience (QoE) of a cloud gaming service based on measurable, QoS metrics in the communication network. BACKGROUND The gaming landscape has undergone a significant transformation in recent years with the emergence of cloud gaming services. These services have revolutionized how games are delivered and played, enabling gamers to access and enjoy high-quality gaming experiences without the need for expensive gaming hardware. The cloud gaming market, estimated to be 1.69 billion USD in 2023, is expected to reach 6.80 billion USD by 2028. The growth and success of the cloud-gaming market can be attributed, at least in part, to the Quality of Experience (QoE), provided by the cloud- gaming services. Unlike traditional online games that run on local hardware, cloud gaming applications run on remote servers that are accessed over a communication network. Video, audio, and user inputs are transmitted over the network in real time to and from the remote server. Thus, network conditions can have a significant impact on gamers’ QoE. Internet service providers (ISPs) and other network operators, referred to herein as communication service providers (CSPs), face the challenge of ensuring the (QoS) of different real-time services, each with its unique network requirements. Service providers of cloud gaming services and CSPs often establish service level agreements (SLAs) that specify necessary network and quality of service (QoS) parameters. These parameters must be monitored and maintained within the network, tailored to the specific requirements of each service. W3C has standardized QoS parameters for cloud gaming services using WebRTC technology, such as NVIDIA’s GeoForce Now cloud gaming platform. However, these parameters are not readily available to the CSPs. The relationship between gaming experience and network conditions has been extensively studied. Some studies have been performed to examine the behavior of cloud gaming platforms under diverse network conditions by assessing various QoS metrics. In P. Graff et al., “An analysis of cloud gaming platforms behavior under different network constraints,” in 17th International Conference on Network and Service Management (CNSM). IEEE, 2021 and X. Xu and M. Claypool, “Measurement of the responses of cloud-based game streaming to network congestion,” in Proceedings of the 32nd Workshop on Network and Operating Systems Support for Digital Audio and Video, 2022, pp.22–28, the authors conducted a network traffic analysis to investigate the behavior of commercial services under network constraints. Marchal et al, An Analysis of Cloud Gaming Platforms Behaviour Under Synthetic Network Constraints and Real Cellular Networks Conditions |Journal of Network and Systems Management (springer.com), showed that cloud gaming services change their network traffic under both sudden and initial network degradations. They demonstrate that services reduce the bitrate by increasing the packet inter-arrival time (IAT) and / or reducing the packet size. Xu et al. analyzed the changes in bitrate, network latency and packet loss of cloud gaming services under network congestion, achieved by bandwidth reduction and background traffic increase. They found that when network disturbances occur, the GFN reduces its bitrate significantly and becomes very conservative with resources. In O. S. Penaherrera-Pulla, C. Baena, S. Fortes, E. Baena, and R. Barco, ˜ “Measuring key quality indicators in cloud gaming: Framework and assessment over wireless networks,” Sensors, 2021. the authors investigated on their test bed (using the Moonlight platform) how Key Quality Indicators (KQIs) are affected by different transport networks (LTE, Ethernet, and WiFi). The metrics described latency, frame rate, rendering time and rendering loss. They found Ethernet to be the most viable, LTE to be an adequate technology with high response time, and WiFi is deemed impractical. Chen et al., “On the quality of service of cloud gaming systems,” IEEE Transactions on Multimedia, vol. 16, no.2, pp.480–495, 2013 examined the commercial OnLive and self-hosted StreamMyGame platforms. They conclude that commercial OnLive performs better, because it provides adaptable frame rates, and better graphic quality. Studies have also been made of the connection between various types of network degradations and Quality of Experience. Generally, the focus of these studies is on the impact of one or more network characteristics (i.e., latency, packet loss, jitter and bandwidth) on perceived QoE, either using commercial cloud gaming services or their own test beds. Wahab et al, “Subjective quality assessment for cloud gaming,” (https: / / www.mdpi.com / 2571-8800 / 4 / 3 / 31 / review_report) and Clincy et al. “Subjective evaluation of latency and packet loss in a cloud-based game,” in 201310th International Conference on Information Technology: New Generations. IEEE, 2013, pp.473–476 examined the impact of packet loss and latency on QoE. Wahab et al. concluded that the effect is game dependent. For example a slow game such as FIFA is less affected by the latency. Meanwhile, Clincy et al concludes that even 1% packet loss makes a game unplayable. In M. Jarschel et al., “An evaluation of qoe in cloud gaming based on subjective tests,” in 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing. IEEE, 2011., the authors identified that downstream loss is the most important feature to determine the QoE. Suznjevic et al found that GFN prioritizes frame rate over resolution when network resources are limited, prioritizing smooth streaming at the cost of quality. Subjective testing revealed that games performed poorly with increased latency but showed resilience to packet loss. Lindstrom et. al, “Cloud gaming: A qoe study of fast-paced singleplayer and multiplayer gaming,” in 2020 IEEE / ACM 13th International Conference on Utility and Cloud Computing (UCC). IEEE, 2020 analyzed network and video related QoS metrics with both perceived QoE and in-game performance. They find that the concept they dub ”frame age” – which is the time elapsed between the creation and display of a frame – is highly correlated to QoE, particularly in faster-paced games. The frame age includes network latency, hence the authors conclude that network latency has a higher impact on QoE than packet loss. Linear regression models were proposed that model the QoE metrics. The results of papers Lindstrom et. al, M. Claypool and D. Finkel, “The effects of latency on player performance in cloud-based games,” in 201413th Annual Workshop on Network and Systems Support for Games. IEEE, 2014, pp.1–6, and S. S. Sabet et al., “Towards applying game adaptation to decrease the impact of delay on quality of experience,” in 2018 IEEE international symposium on multimedia (ISM). IEEE, 2018, pp.114–121 all conclude that perceived QoE and in-game performance are strongly related. In Lindstrom et. al, this claim is supported with correlation analysis. While the influence of network conditions on cloud gaming services is known, there are no methods currently available to quantify the impact of network conditions on end user QoE or to provide predictive models for improving the gaming experience. Predictive models and performance boosting systems in the network require objective QoE metrics at the application side, as well as appropriate transport QoS metrics and radio parameters from the CSP to perform network improvement for the given service type as well as to prove service quality improvement (compare service quality before and after improvements). No such technical solutions currently exist for cloud gaming services. SUMMARY The present disclosure relates to machine learning models and methods of training machine learning models to predict the QoE of a cloud gaming service, defined as in-game performance, based on measurable, QoS metrics in the communication network. During training, an objective, measurable, in-game performance metric is used as a target. QoS metrics covering different degradation scenarios ranging from good network conditions to poor network conditions are collected and used as training data. A selected set of measurable QoS metrics predictive of in-game performance is determined by comparing the performance of a constrained QoS model trained using only selected QoS metrics measurable in the network with the performance of a reference QoS model trained using all significant QoS metrics. In cases where the constrained QoS model provides similar performance to the reference QoS model, the selected set of measurable QoS metrics is validated. The constrained QoS model can be deployed in the network to estimate QoE of the cloud-gaming service and to manage network resources to ensure that QoS guarantees are met. Additionally, the QoS model can be used to provide a performance boosting service to users of the cloud-gaming service. A first aspect of the disclosure comprises methods of estimating QoE of a cloud- gaming service delivered over a communication network using a ML model trained using QoS metrics measurable in the communication network. In one embodiment, the method comprises collecting quality of service (QoS) metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network. The method further comprises estimating the service quality of the cloud gaming service based on the QoS metrics using a machine learning (ML) model trained with sample QoS parameters correlated with sample in-game performance metrics. A second aspect of the disclosure comprises a network node configured to estimate service quality of a cloud gaming service delivered over a communication network. The network node is configured to collect quality of service (QoS) metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network. The network node is further configured to estimate the service quality of the cloud gaming service based on the QoS metrics using a machine learning (ML) model trained with sample QoS parameters correlated with sample in- game performance metrics. A third aspect of the disclosure comprises a network node configured to estimate service quality of a cloud gaming service delivered over a communication network. The network node comprises network interface circuitry configured for communication with other network nodes over a communication network and processing circuitry. The processing circuitry is configured to collect quality of service (QoS) metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network. The processing circuitry is further configured to estimate the service quality of the cloud gaming service based on the QoS metrics using a machine learning (ML) model trained with sample QoS parameters correlated with sample in- game performance metrics. A fourth aspect of the disclosure comprises a computer program for a network node in a communication network. The computer program comprises executable instructions that, when executed by processing circuitry in the network node, causes it to perform the method according to the first aspect. A fifth aspect of the disclosure comprises a carrier containing a computer program according to the fourth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium. A sixth aspect of the disclosure comprises methods of training a service quality model to estimating QoE of a cloud-gaming service delivered based on measurable, QoS metrics in the communication network. The method comprises obtaining training data comprising sample quality of service (QoS) metrics measurable in the communication network correlated with measured in-game performance metrics. The method further comprises training the service quality model to estimate service quality of the cloud gaming service using the training data. A seventh aspect of the disclosure comprises a computing device configured to estimate service quality of a cloud gaming service delivered over a communication network. The computing device is configured to obtain training data comprising sample quality of service (QoS) metrics measurable in the communication network and correlated with measured in-game performance metrics. The computing device is further configured to train the QoE model to estimate service quality of the cloud gaming service using the training data. An eighth aspect of the disclosure comprises a computing device configured to train a service quality model to estimating QoE of a cloud-gaming service delivered based on measurable, QoS metrics in the communication network. The computing device comprises network interface circuitry configured for communication with remote devices over a communication network and processing circuitry. The processing circuitry is configured to obtain training data comprising sample quality of service (QoS) metrics measurable in the communication network and correlated with measured in- game performance metrics. The processing circuitry is further configured to train the QoE model to estimate service quality of the cloud gaming service using the training data. A ninth aspect of the disclosure comprises a computer program for a network node in a communication network. The computer program comprises executable instructions that, when executed by processing circuitry in the network node, causes it to perform the method according to the sixth aspect. A fifteenth aspect of the disclosure comprises a carrier containing a computer program according to the ninth aspect. The carrier is one of an electronic signal, optical signal, radio signal, or a non-transitory computer readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a diagram illustrating the system architecture of a wireless communication network over which a cloud-gaming service is delivered. Figure 2 is a graph showing the distribution of incoming video bitrate for a video game at different resolutions and frame rates. Figure 3 is a graph showing the distribution of normalized Hit Factor scores for different degradation scenarios. Figure 4 is a graph comparing feature performance of a ML model trained using network-related metrics with a ML model using all significant QoS metrics. Figure 5 is a flow chart showing a method of estimating service quality of a cloud- gaming service delivered over a communication network based on measurable, QoS metrics in the communication network. Figure 6 is a flow chart showing a method of training a predictive ML model for estimating service quality of a cloud-gaming service based on measurable, QoS metrics in the communication network. Figure 7 is a functional block diagram of a network node configured to estimate service quality of a cloud gaming service delivered over a communication network based on measurable, QoS metrics in the communication network. Figure 8 is a functional block diagram of a computing device configured to train a service quality model to estimate service quality of a cloud-gaming service delivered over a communication network based on measurable, QoS metrics in the communication network. DETAILED DESCRIPTION The present disclosure relates to Quality of Service (QoS) assurance for a cloud gaming service delivered over a wireless communication network. A machine learning model is trained to predict the Quality of Experience (QoE) of a cloud gaming service, defined as in-game performance, based on measurable, QoS metrics in the communication network. The service quality model can be deployed in the wireless communication network to estimate QoE of the cloud-gaming service using only QoS metrics measurable the network and to manage network resources to ensure that QoS guarantees are met. Additionally, the QoS model can be used to provide a performance boosting service to users of the cloud-gaming service. Figure 1 illustrates the network architecture for a wireless communication network 10 implementing an ML model for quality assurance. The present disclosure is described in the context of a Fifth Generation (5G) wireless communication network but the techniques described herein are more generally applicable to other wireless communications implementing data analytics in the core network or management domain. The wireless communication network 10 generally comprises a 5G RAN 20 including one or more gNodeBs (gNBs) 25, a core network 30, and a network management domain 70. The core network 30 comprises a collection of network functions (NFs) performing different network tasks. Figure 1 illustrates various NFs relevant to this disclosure including the UPF 35, Access and Mobility Management Function (AMF) 40, Session Management Function (SMF) 45, Policy Control Function (PCF) 50, Network Exposure Functions (NEF) 55, and NWDAF 60. The NWDAF 60 implements a ML service quality model as hereinafter described. Alternatively, a Management Domain Application Functions (MDAF) 80 in the management domain may perform data analytics and implement the ML service quality model. The network management domain 70 comprises a Network Slice Management Function (NSMF) 75 and optionally a MDAF 80. The NFs in core network and network management domain comprise logical entities that may be implemented by one or more processors, hardware, firmware, or a combination thereof. In cloud-based networks, the NFs are typically implemented as virtual machines (VMs), or as containers (e.g., Kubernetes). Network 10 may include multiple instances of each NF type. The NFs can also be implemented in stand-alone servers or specialized servers. The UPF 35 supports handling of user plane traffic, including packet inspection, packet routing and forwarding, traffic usage reporting, and QoS handling. The UPF 35 connects with external IP networks and serves as an IP anchor point for user equipment (UEs) served by the UPF 35 so that the UEs 15 are reachable even when moving around in the network 10. The UPF 35 processes data being forwarded. Such processing may include packet inspection, classification, and QoS marking of forwarded packets. The UPF 35 generates traffic usage reports, which the SMF 45 includes in charging reports, and is involved in policy enforcement. The AMF 40 is a network function that manages access to the 5G network and handles mobility-related functions for the UEs. Its role is similar to that of the Mobility Management Entity (MME) in Fourth Generation (4G) networks. When a UE 15 is not in idle mode, the gNB handovers are exposed to the AMF 40. Additionally, the AMF 40 can use other signaling elements to determine which gNB 25 a UE 15 is attached to. The AMF 40 can expose these events by means of a standardized interface as well. The SMF 45 manages Packet Data Unit (PDU) sessions for the UEs 15, which includes the establishment, modification, and release of PDU sessions. The SMF 45 selects the UPF 35 to handle a PDU session for a UE 15 and controls the UPF 35. The SMF 45 receives Policy and Charging Control (PCC) Rules from the PCF 50 and configures the UPF 35 for various data flow tasks, such as shaping, policing to provide bandwidth, and charging functions. The PCF 50 supports a unified policy framework to govern the network behavior. Specifically, the PCF 50 provides Policy and Charging Control (PCC) rules to the Policy and Charging Enforcement Function (PCEF), i.e., the SMF 50 / UPF 35 that enforces policy and charging decisions according to provisioned PCC rules. Per flow analytics is implemented in either the NWDAF 60 or MDAF 80. The NWDAF 60 or MDAF 80 receives signaling related events from core network functions (e.g., AMF 40 and SMF 45) and gNB radio events from the RAN 20 including UL and DL radio measurements. The per-flow analytics system correlates the signaling related events from the core network and radio measurements from the RAN with user plane probe reports provided by the UPF 35 or other transport network probes to generate per flow correlated session records, which include all transport KPIs. The per-flow session records serve as input to ML models implemented by the NWDAF and / or MDAF 80. ML model output is the subjective or objective end-to-end QoS and QoE KPIs. The NSMF 75 is responsible for coordination, management, and orchestration of network slice instances (NSIs) in a cloud-based infrastructure, i.e., NSI life cycle management. In general, the NSMF 75 may use the output of ML models implemented in the NWDAF 60 and / or MDAF 80 to set or modify policy settings in the PCF 50 and improve service quality provided by the network in a closed-loop function. Service quality estimates can also be used for service assurance, e.g., reporting, checking, ensuring SLS target requirements. In the present disclosure, the NSMF 75 plays a role in quality assurance for in- cloud gaming services. The NSMF 75 uses the service quality estimates provided by the NWDAF 60 or MDAF 80 to monitor compliance with service level agreement (SLA) with other application provider. When service quality is degraded and fails to comply with SLAs, the NSMF 75 may trigger corrective action to restore the service quality. Additionally, in some embodiments, a user of the cloud-gaming service can request the network through NEF 55 to increase service quality. In response to a user request to “boost” performance, the NSMF 75, NWDAF 60, or MDAF 80 can implement closed loop functions to increase the quality of experience for the user of the cloud-gaming service. When a performance boost is requested, the NWDAF 60 or MDAF 80 measures QoS parameters before and after the network boost. quantifies the QoS and QoE improvements and reports the QoS and QoE improvements to the cloud gaming provider and / or user. Charges for this service may be agreed in the SLA with the cloud gaming provider. In short, the NWDAF 60 and / or MDAF 80 can use a service quality model to estimate a service quality (e.g., QoS or QoE) of a cloud-gaming service delivered over the communication network. The estimates of the of the service quality can be used in a closed-loop process to provide quality assurance for the cloud-gaming service and / or a performance boosting service. To achieve these objectives, the service quality model needs to accurately estimate the service quality. The following describes an approach to training the service quality model to accurately estimate a QoS or QoE of a cloud gaming service based on QoS metrics measurable in the network. An initial step in the development of the service quality models is understanding the relationship between the quality of experience of cloud gaming service users and the network conditions. To achieve this, the data collected on the client side (i.e. WebRTC logs) are used as explanatory variables. The QoE of the user is defined as in-gaming performance. As an example, the QoE may comprise any an objective score that is a major goal of the game. Test Platform Among the most popular cloud gaming services, is the NVIDIA GeForce NOW gaming platform. This gaming platform is not restricted to a specific game library but allows users to play almost any game and can be run from a web browser. The player can play for up to 60 minutes continuously but needs to wait in a queue to get cloud resources. There is no lower limit for the quality of the stream, but the typical resolution is 540p or 720p and the frame rate is around 60 fps (frames / sec). Priority, the first subscription plan, allows up to 6 hours of gameplay at a maximum resolution of 1080p and a maximum frame rate of 60 fps. The Ultimate package offers up to 4K resolution, up to 120 fps frame rate, 8 hours of maximum continuous play and ray-traced servers. The following exploration is based on the Priority package. GeForce NOW requires at least 15Mbps bandwidth for 720p at 60 fps video stream and 25 Mbps for 1080p at 60 fps. Less than 80 ms latency from an NVIDIA data center is required, however, for the best experience, less than 40ms is recommended. To model the effect of network quality on the user experience, two fast-paced games were tested because they are inherently sensitive to network quality. The selected games were both FPS (first-person shooter) games, Counter-Strike: Global Offensive (CSGO) and Apex Legends (APEX). Community-developed practice maps are available for CSGO, and natively for APEX to practice the various game mechanics that are quintessential to shooting games. For each game, 5 different exercises testing certain aspects of the player’s ability in different ways were selected. These basic aspects are speed, accuracy, reflex, precision target following, and fine motor control. Testing these aspects separately provides insight into how network conditions affect the player’s abilities. In CSGO the community-built Yprac trainer map were used, and in APEX the built-in Firing Range was used. The tasks and which skill set they test are shown in Table I.

[0002] Table 1 Training Exercises For CSGO and APEX Games Data Collection A method and data collection system published in G. Dobreff et al., “Data collection framework for end-to-end radio and transport network quality monitoring,” in 15th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2023 was used for data collection. The data collection system is a Python-based application designed to perform repeatable and automated measurements. The measurements were performed on Ubuntu 22.04 on an AMD FX-8320 processor with 16GB of random access memory (RAM) and a GeForce GTX 1060 OC graphics card running the GeForce Now client in a Chromium browser. The system included an Ethernet interface for Internet access, with a consistent idle download speed of 150 Mbps and a latency of 8 ms. After starting the cloud gaming service and launching data collection scripts are launched, measurements were collected while a user completed the 5 training exercises shown in Table 1 for each game. For each measurement (one complete set of training exercise for one game), a downlink network degradation of some kind was induced on the network interface, which remained unchanged in degree and type until the end of the measurement. Four types of degradations were used: packet loss, bandwidth reduction, latency, jitter. For each type of degradation, five different levels of degradation were tested: • Packet loss: 1%, 2%, 3%, 4%, 5% • Bandwidth reduction: 50Mbps, 30Mbps, 20Mbps, 10Mbps, 5Mbps • Latency: 50ms, 100ms, 150ms, 200ms, 250ms • Jitter: 50ms latency ± 5ms, 10ms, 15ms, 30ms, 50ms In a given measurement, only one degradation configuration was induced in the whole time. Following the principle of soundness, each of the above configurations was repeated 5 times. In total, 200 measurements were performed. In addition, 10 measurements – 5 for each game – were performed where no degradation was induced. The same player was used for all measurements. Upon completion of each measurement, a video recording of the training exercise and Web Real-Time Communication (WebRTC) logs were collected. On the practice maps the achieved score of each training exercise was also recorded by the gaming platform The metrics for in-game performance in the training exercise for CSGO are: the number of normal hits on target (# hits), number of missed shots (# misses) and the time to complete the practice. The same metrics were available in APEX, with the difference that bullseye hits were distinguished and registered in addition to normal hits (denoted as # bullseye). A QoE metric for the cloud gaming service, referred to herein as the Hit Factor (HF) was computed based on the performance metrics collected from the gaming platform. The calculation of the Hit Factor is based on the metric introduced by the United States Practical Shooting Association (USPSA), which aims to encourage balanced shooting performance. Missed shots were penalized, bullseye hits were rewarded. For APEX, the formula is as follows: HFAP EX= (# hits − 0.5 · # misses + 1.5 · # bullseye) / Time Eq.1 A slightly different calculations of the Hit Factor was used for CSGO because bull’s eyes cannot be distinguished from normal hits in CSGO. For CSGO, the Hit Factor is calculated as follows: HFCSGO= (# hits − 0.5 · # misses) / Time Eq.1 The dataset collected consists of video recordings of the measurements, log files of the measurements and logs of the WebRTC sessions. The WebRTC framework establishes direct communication between the GeForce Now web browser client and the NVIDIA data centers in a peer-to-peer manner. The internal logs showing how the WebRTC connection is set up and maintained can be downloaded at chrome: / / webrtc- internals / . The communication is divided into distinct sessions, which are streams or channels used for real-time data transfers. Typically, there are two main types: media sessions and data sessions. Media sessions are responsible for transporting audio and video streams. In a WebRTC internals logs, low-level information related to the codecs used, media statistics (e.g., frame rate, resolution), and network statistics (e.g., latency, packet loss) for each media session can be found. The logs contain these metrics in a time series format. A list of all metrics can be found on the W3C website. The score for each training exercise performed was obtained from the screen capture. Due to the diverse nature of the training exercises, they produced a wide range of Hit Factor values due, for example, to the number of targets, the movement of the targets, the duration of the exercise, etc. Therefore, Min-Max normalization was used to map the HF values of the different exercises in the interval 0 - 100, to enable comparisons of game performance for different training exercises. The equation of used normalization is given by: Xn = (X−Xmin) · 100 / (Xmax−Xmin), Eq.3 where Xn is the value to be normalized and the Xmin and Xmax denote the minimum and maximum score achieved for that given training exercise. The resulting normalized Hit Factor (NHF) is used to describe the user experience and will be the target variable for the machine learning models. Findings The collected data was analyzed to determine the behavior of the cloud gaming service under different network conditions and the impact of different degradation scenarios. Specifically, the relationship between the network degradation and the user experience and QoS metrics was investigated. The tests consistently achieved a frame rate of around 60 fps when using the control settings, i.e., no added network degradation. However, the video quality was not always FullHD, i.e., 1920x1080 pixels, throughout all the measurements. Both the resolution and the frame rate varied. The distribution of video quality across all measurements was as follows: 1920p@60 fps (7.5%), 1920@30 fps (0.1%), 1280p@60 fps (45.3%), 1280p@30 fps (0.5%), 960p@60 fps (23.1%) and 960p@30 fps (23.5%). As it can be seen 1920@30 fps and 1280p@30 fps video quality is rarely observable, possibly only during transient periods. The incoming audio bandwidth remained stable at 196kbps bitrate, but typically dropped when bandwidth was low or packet loss was high. Figure 2 is a bar graph showing the distribution of the bandwidth requirement for different video quality settings. As shown in Figure 2, the video stream bitrate is below the recommended 15Mbps most of the time. These observations are consistent with the behavior reported in earlier works. During the tests, there were a few instances of changes in video and audio stream quality that were not justified by the induced degradation, such as sudden increases in resolution that led to very serious glitches. Note that the values shown in Figure 2 represent the main video stream. It was observed that in every case there is one incoming audio stream and two incoming video streams. The second video stream is a retransmission stream that sends data redundantly to allow for corrections of errors. Its bandwidth ranges from 1.2 to 3.6 Mbps. The distribution of NHF scores on each training exercise is shown in Figure 3, by degradation type and level. NHF is the normalized in-game performance, as defined above. For each type of degradation, the Pearson correlation between the degradation level and the NHF score is as follows: bandwidth 0.12, jitter −0.33*, packet loss −0.31** and latency −0.87***. Note that the statistical significance is indicated by ∗, ∗∗, ∗∗∗ when the p-value is below .05, .01 and .001 respectively. As the induced latency increased, performance in the game decreased heavily accordingly. Smaller degradation, i.e., 50-100ms, mainly affected the controllability of games compared to games run locally or even hosted in the cloud without degradation. However, at higher values, the visual and sound quality has also degraded considerably, which is evident in scores. In the jitter measurements, a constant latency of 50ms was induced and worse results were thus obtained than in nondegraded measurements already at low jitter values. However, at low jitter, we observed multiple times that the scores remained roughly unchanged, and then suddenly worsened with the introduction of the most severe degradation (above ±30ms). This phenomenon, that increasing jitter does not gradually but abruptly reduce the user experience, has also been observed by others in empirical research. The deterioration of the user experience in the bandwidth measurements can be understood from Figure 3. Although the bandwidth was limited to 50-30-20 Mbps during the measurements, the service did not always use the full bandwidth. Thus, in these cases there was no noticeable effect of degradation, only for the 10 and 5 Mbps limitation. In fact, the user experience was negatively affected when the game would suddenly jump to a higher resolution and then back again at high bandwidth, causing the game to freeze for a moment. In terms of packet loss measurements, there was no significant deterioration in the user experience at lower packet loss values. An interesting finding is that there was no consistent relation between the various degradation scenarios and packet loss. Sometimes more sometimes less packet loss was observed in the game. One interpretation of this observation is that cloud gaming services can tolerate some amount of packet loss due to various error correction techniques. The impact of various of types and levels of network degradation on shooting skills was also analyzed. Table 2 below shows the correlation between the normalized Hit Factor (NHF) values and degradation levels for different types of network degradation and shooting skills. To a small extent, bandwidth reduction affects accuracy and control, while jitter affects accuracy, tracking and control. Packet loss influences all skills except speed, while latency influences all skills. Seeing the variation in user experience caused by different levels of network degradation, the relatively low correlation values can be explained by the fact that, except for latency, network quality is not linearly related to performance below some threshold. Table 2: Correlation Between The Network Degradation And In-Game Performance For Different Shooting Skills The correlation between the WebRTC metrics describing internal operations and the NHF value was also investigated. For the correlation analysis, WebRTC metrics were averaged per exercise, and all metrics were measured on a per-second basis, i.e. they are not cumulative. Table 3 shows the extent to which each WebRTC metric correlates with player performance. Redundant metrics have been omitted from the table. As can be seen, and as expected, network latency has the highest correlation with player performance. The table shows that both the audio and video stream metrics capture the phenomena encountered during various network problems. Table 3: Correlation Between The Most Metrics And In-Game Performance. The available bandwidth is described well by several metrics for both audio and video traffic and correlates well with the NHF value. Possible jitter and packet loss refers to the dropping or untimely arrival of video and audio packets, which can lead to video and / or audio freezing and interrupts. These phenomena can be observed, for example, in the jitter buffer, in the multiplication or dropping of audio packets, or in the time taken to decode frames. The properties of these internal mechanics to avoid these errors are described by several WebRTC metrics, for example, the variation of JitterBufferDelay,interruptionCount / s or freezeCount / s. Table 3 shows that these metrics correlate well with NHF values. Interestingly, frame rate measure in frames per second and resolution are not linearly related to in-game performance. These results show that there is a linear correlation – albeit not a strong one – between user experience and WebRTC metrics describing internal operations. It is important to note that the above analysis investigates the linear relationship between two variables. The deeper – i.e., non-linear and multivariate – correlation will be considered more fully below. In the following, machine learning models for estimating user experience are introduced, which have been trained using the collected client-side WebRTC logs and the in-game performance from the training exercises in the games. The preprocessing steps are outlined first, followed by the modeling process and results. To train the models, the per-task normalized value of the in-game performance (NHF) is used as the target variable and the WebRTC metrics are used as the explanatory variables. Only the metrics describing the main audio and video data streams are used as explanatory variables. Because the WebRTC logs were only approximately 1 second granularity and had missing values, resampling was performed and linear interpolation was used to fill in the missing data. Because each explanatory variable describes the state over a particular second, e.g.: number of packets received in a given second, the target variable had to be transformed as well. As there are 5 exercises per measurement, the value of the target variable for a given measurement takes the value of the NHF value associated to a given exercises continuously between the start and the end of a given exercises, and takes no value otherwise. This procedure results in a dataset, where each row describes a specific second of a particular exercise of a measurement. A given record of the dataset contains metrics describing the quality of service of the video and audio stream, as well as the NHF score associated with the exercise played in the given second. For the final data set, only the parts where the target variable is available are kept, i.e. the pre- and post-measurement transition periods and the transitions between exercises are excluded. The resulting data set with a granularity of 1 second used in the training of the ML models consists of 93,993 datapoints, and represents a total of about 26 hours of data. The audio and video streams are described by 44 and 53 metrics, respectively. The ML models were trained using the PyCaret Python package, which is an AutoML framework. AutoML libraries perform the training of ML models in an automated way. Data preprocessing, postprocessing, normalization, selection of explanatory variables and other steps before modeling are performed automatically in a configurable manner. Training and evaluating multiple ML models are also done in an automatized way. This approach can be used to efficiently evaluate multiple machine learning algorithms. The PyCaret library was used to perform z-score normalization on the explanatory variables, and basic feature selection. In many cases, the explanatory variables contain redundant information. Thus variables with a very strong correlation - greater than 0.9 for the two variables - were discarded. GroupKFold cross-validation was used during the training to avoid overfitting. It is a grouped cross-validation technique that reduces the possibility of overfitting by splitting the dataset into K subsets by group, – in this case by measurement – instead of splitting purely randomly. This approach avoids cases where data points from the same measurement could be included in both the training and validation sets. The hyperparameter optimization of each ML model was performed using the Python library scikit-optimize using cross-validation. To evaluate the performance of the trained models on unknown data, a previously isolated test set was used with data points not included in any previous training or validation set. This data set is representative: it includes data from one iteration of each of the four degradation types and their five levels from both games, plus 1-1 degradation-free measurement. Thus, the data set used for training contains data points from 42 measurements, which is one fifth of the total data. The performance of the linear and more complex models were investigated: Extra Trees, Gradient Boosting (GBM), Random Forest, KNN Regression, Elastic Net (LinReg). Furthermore, the results obtained using different input feature sets was investigated. Three input feature sets investigated are: (1) All WebRTC metrics, i.e. all 44 + 53 metrics describing audio and video streams, (2) network-related metrics only, (3) application-level metrics only, i.e. all metrics excluding network-related metrics. Network- related metrics include the round-trip time, the control messages sent and received per second by the client, and bitrate (Kbs) for both audio and video streams, packets received per second for both audio and video streams, jitter for both audio and video streams, and packets lost per second for both audio and video streams. The accuracy of the different ML models was assessed using several different metrics: the Mean Absolute Error (MAE), Root Mean Squared Error (RMSE) and R2 (coefficient of determination), which are calculated as follows: MAE =1^n yi − yˆi Eq. 45 whereyˆiis the predicted and yiis the observed value of the datapoint i , and y i is themean of the observed values. The performance of the best performing linear and complex models trained on the three input feature sets mentioned above is shown in Table 4. The scores in the table are calculated on the hold-out test set. Table 4: Results Of Models Trained On Different Input Feature Sets As the table shows, the Gradient Boosting algorithm using all metrics estimates the NHF value – ranges between 0-100 – with an average absolute error of 8.7. It can be seen that, using the same set of input attributes in all three cases, the more complex GBM model achieves significantly better results than the linear model. This result implies the non-linearity of the problem, which is confirmed by the fact that Models 2, 4 and 6, adding application-level metrics to network-level metrics, does not improve accuracy. It can also be seen that – regardless of whether linear or GBM model results are considered – using only the application-level metrics results in much lower accuracy. When comparing Model 1 and Model 3, we see that using the network metrics only can achieve relatively good results. Shapley Additive Explanations (SHAP) values were used to determine the parameters considered important by the two best models, i.e. Model 1 and 3. Figure 4 shows the absolute mean of these values. As shown in Figure 4, the higher the value, the greater the influence of the parameter in making the prediction. It can be seen that both models considered network latency as the most important parameter and jitter was also considered an important metric. For Model 3 using only network-related metrics, the video stream bitrate (i.e. packet per second) is also an important parameter to monitor. An interesting result is that when analyzing the graph, is can be observed that it is also worth monitoring the control messages sent (packetsSent / s and bytesSent / s). This interpretation can be explained by the fact that, if the client detects bad network conditions, it communicates the conditions to the server in the form of frequent control messages. The audio bitrate and packet loss rate were found to be less significant metrics, which is in line with preliminary expectations. For the model using all metrics, the metrics describing the different application- level impairments improve the prediction. The additional metrics considered include:: audio interruptions, inserted audio samples for deceleration, number of video freezes and the number of PLI (Picture Loss Indicator) packets. Overall, results show that WebRTC metrics are excellent descriptors of user experience. It can be concluded that promising results can be achieved with network- level metrics alone, which is a useful insight for communication service providers (CSPs), as they are able to estimate the experience provided by cloud gaming services by monitoring both upstream and downstream network traffic. Furthermore, our results can be used to target specific aspects of the network for improvement, enabling CSPs to meet SLA requirements more effectively. Additionally, these findings show that it is possible to improve the accuracy of ML models for service quality estimation and / or prediction by monitoring application-level metrics, especially those describing application impairments. However, CSPs generally do not have access to the application level metrics due to encrypted traffic, requiring separate agreements with the service providers. The experimental results indicate that a service quality model can be trained to predict the Quality of Experience (QoE) of a cloud gaming service, defined as in-game performance, based on measurable, QoS metrics in the communication network. The service quality model can be deployed in the wireless communication network to estimate QoE of the cloud-gaming service using only QoS metrics measurable the network and to manage network resources to ensure that QoS guarantees are met. Additionally, the QoS model can be used to provide a performance boosting service to users of the cloud-gaming service. The process for training the service quality model includes: 1) Measure in-game performance. The performance of a user over a test period is measured to use as a target for training the ML model. The in-game performance metrics should be measured over different degradation scenarios, including no degradation. The degradation scenarios should preferably account for different types and levels of network degradations that are likely to occur. The in-game performance metrics are provided by the gaming platform. In one embodiment using the GeoForce Now gaming platform, the in-game performance metrics is the Hit Factor calculated as herein described. 2) Collect WebRTC metrics. WebRTC metrics correlated with the in-game performance metrics are collected at the client side at 1 sec intervals for each measurement being performed As one example, a measurement may comprise a set of training exercises to be completed by the user. The WebRTC metrics includes QoS metrics measurable in the communication network. The collected Web RTC metrics include QoS metrics measurable in the communication network and QoS metrics not measurable in the communication network. 3) Determine QoS metrics predictive of QoE – One or more test models are trained using only WebRTC metrics measurable only in the communication network. Each test model may use a different combination of WebRTC metrics and / or different ML algorithms. A reference service quality model is also trained using WebRTC metrics measurable in the communication network and additional WebRTC metrics not measurable in the communication network. In one embodiment, the reference model can be trained using all WebRTC metrics. Performance of the test models is compared with the performance of the reference model to determine the efficacy of the test models. Based on the comparison, the QoS metrics and ML algorithm are selected for use in the online service quality model. 4) Train the service quality model - The performance data and correlated WebRTC metrics determined in the previous step are used as input to train the service quality model for deployment in the communication network. The performance metric is used as a target variable and the WebRTC metrics are used as explanatory variable. For training the service model, only QoS metrics measurable in the network are used. The set of QoS metrics selected for training can be determined beforehand by testing and experimentation as described above. Once the service quality model is trained, the service quality model can be deployed in the network and used to estimate a QoE for the cloud gaming service delivered over the communication network. The service quality model can be used, for example, to monitor QoS provided by the network for compliance with SLAs with cloud gaming providers. When service quality is degraded or fails to comply with SLAs, the NSMF 75 may trigger corrective action to restore the service quality. Examples of corrective actions includes modifications of PCC Rules and / or instantiation of NSI. Additionally, in some embodiments, a user of the cloud-gaming service can request the network through NEF 55 to increase service quality. In response to a user request to “boost” performance, the NSMF 75, NWDAF 60, or MDAF 80 can implement closed loop functions to increase the quality of experience for the user of the cloud- gaming service. When a performance boost is requested, the NWDAF 60 or MDAF 80 measures QoS parameters before and after the network boost. quantifies the QoS and QoE improvements and reports the QoS and QoE improvements to the cloud gaming provider and / or user. Charges for this service may be agreed in the SLA with the cloud gaming provider. The QoE model as herein described uses a standard and objective metric, i.e., Hit Factor, as end user QoE metrics. This metric is objective and can be precisely measured. Gaming platforms can log information used to calculate the objective metric or provide a direct measure. By using an objective, measurable metric for labeling training data for ML models, the labeling of training data can be automated. Subjective factors can be eliminated and the time consuming process to obtain subjective labels are avoided. Using a QoE ML model trained for different network degradation scenarios, the relation between the WebRTC (buffer, frame and packet) metrics and the end user service quality (Hit Factor) can be established. The impact of the different parameters on the end user service quality is determined. By selecting the most significant WebRTC parameters, i.e., the parameters most predictive of service quality, as input to the model, the service quality model is simplified. Model accuracy can be checked using a reference model to estimate the Hit Factor by using all WebRTC parameters as input and the limits (i.e. the best possible accuracy) of this model can be determined. Using QoS metrics measurable in the communication network as input to the service quality model enables the CSP to use the model network wide operation for estimating the end user service quality real-time. Comparing the performance of this model to the one using the full WebRTC parameter set, indicates whether the end user service quality can be estimated by the network operator and what is the expected accuracy. Estimating the end user service quality can be used for assuring compliance with SLAs. The method herein described can be used to determine a set of SLA parameters that can be used for service assurance. The estimated end user service quality can also be used in a network boost feature / service: By improving the identified significant QoS parameters with a technical (e.g., closed loop) solution, the end user service quality (QoE) is automatically improved. The improvement of the boost can be quantified both in the CSP network and at the end user, namely measuring and comparing the QoS parameters before and after boost and measuring and comparing the QoE parameters before and after boost. This feature can contribute directly to the revenue of the operator as well as to the end user satisfaction. Figure 5 is a flow chart illustrating a method 100 implemented by a data analytics function (e.g., NWDAF 60 or MDAF 80) in a communication network of estimating service quality (e.g., QoE) of a cloud gaming service. A data analytics function collects network quality of service (QoS) metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network (block 110). Based on the collected network QoS metrics, the data analytics function estimates the service quality of the cloud gaming service using a service quality model trained with sample QoS parameters correlated with measured in-game performance metrics (block 120). In some embodiments of method 100, collecting QoS metrics comprises receiving traffic probe reports per traffic flow, and calculating the QoS metrics based on based on the traffic probe reports. In some embodiments of method 100, calculating the QoS metrics based on the traffic probe reports comprises calculating transport key performance indicators (KPIs) based on the traffic probe reports. In some embodiments of method 100, the transport KPIs are aggregated per network function, per cell, per radio access network node, per network slice, per service type, per user group, or a combination thereof. In some embodiments of method 100, the transport KPIs comprise one or more of the following: round trip time, video jitter, audio jitter, delay in uplink, delay in downlink, packets sent per second, packets received per second, bytes sent per second, bytes received per second, video packets lost per second, audio packets lost per second, and audio packets received per second. Some embodiments of method 100 further comprise comparing the estimated service quality with a target service quality, and triggering a corrective action when the estimated service quality does not meet the target service quality. In some embodiments of method 100, estimating the service quality of the cloud gaming service based on the QoS metrics comprises estimating the service quality based on the aggregated transport KPIs. In some embodiments of method 100, estimating the service quality of the cloud gaming service based on the QoS metrics comprises estimating the in-game performance metric based on the QoS metrics. In some embodiments of method 100, the in-game performance metric comprises a score indicative of user performance in a game being played. In some embodiments of method 100, the in-game performance metric comprises a score indicative of shooting performance in a game being played. Some embodiments of method 100 further comprise comparing the estimated service quality to a target service quality, and triggering corrective action when the estimated service quality does not meet the target service quality. In some embodiments of method 100, triggering corrective action comprises one or more of: changing a policy rule, and instantiating additional network slice instances. Some embodiments of method 100 further comprise receiving a request to boost network performance for one or more users of the cloud gaming service specified in the request, responsive to the request, triggering a performance boosting action to boost network performance for the one or more users specified in the request. Some embodiments of method 100 further comprise estimating the improvement in the QoS metric and / or service quality as a result of the performance boosting action, and send a report quantifying the improvement in the QoS metric and / or service quality to a designated recipient. Figure 6 is a flow chart illustrating a method 200 by a computing device of training a service quality model to estimate service quality (e.g., QoE) of a cloud-gaming service delivered over a communication network. The computing device obtains training data comprising sample quality of service (QoS) metrics measurable in the communication network and correlated with measured in-game performance metrics (block 210). The QoS metrics used in this step are those that have previously been determined to be predictive of in game performance. The computing device trains the service quality model to estimate the service quality of the cloud gaming service using the sample QoS metrics for a selected set of QoS parameters predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network for use as training data (block 220). In some embodiments of method 200, obtaining the training data comprises measuring in-game performance over a testing period and collecting, over the testing period, network QoS metrics measurable in the communication network and correlated with measured in-game performance metrics. Some embodiments of method 200 further comprises selecting QoS metrics predictive of the service quality of the cloud gaming service to use as the training data In some embodiments of method 200, selecting QoS metrics predictive of the service quality of the cloud gaming service to use as the training data comprises training a test model using QoS metrics for a selected set of QoS parameters measurable in the communication network as a training set, training a reference model using QoS metrics for a superset of the selected set of QoS parameters, and comparing a performance of test model with a performance of the reference model to determine a predictive capability of the selected set of QoS parameters. In some embodiments of method 200, training the QoE model using sample QoS metrics comprises training the service quality model using QoS metrics for the selected set of QoS parameters. In some embodiments of method 200, the sample QoS metrics comprise one or more sample WebRTC metrics. In some embodiments of method 200, the selected QoS parameters comprise transport key performance indicators (KPIs) measurable in the communication network. In some embodiments of method 200, the QoS metrics comprise transport KPIs aggregated per network function, per cell, per radio access network node, per network slice, per service type, per user group, or a combination thereof. In some embodiments of method 200, the transport KPIs comprise one or more of the following round trip time, video jitter, audio jitter, delay in uplink, delay in downlink, packets sent per second, packets received per second, bytes sent per second, bytes received per second, video packets lost per second, audio packets lost per second, and audio packets received per second. An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and / or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by one or more processors, carries out the techniques described herein. Figure 7 illustrates an exemplary network node 300 in a wireless communication network configured to estimate the service quality of a cloud gaming service delivered over the communication network. The network node 300 generally comprises network interface circuitry 310, processing circuitry 320, and memory 330. The network interface circuitry 310 comprises a network adapter for coupling the network node 300 to a communication network to enable communication with other network nodes. The network adapter may for example, comprise a wired interface (e.g., Ethernet interface), optical interface (e.g., Synchronous Optical Network (SONET)), or wireless interface (e.g., Wireless fidelity (WiFi)). The processing circuitry 320 comprises one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 320 in one embodiment is configured to perform the methods herein described. According to an embodiment, the processing circuitry 320 comprises a collecting unit 322 and an estimating unit 324. The collecting unit 322 is configured to collect network quality of service (QoS) metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network. The estimating unit 324 is configured to estimate the service quality of the cloud gaming service using a service quality model trained with sample QoS parameters correlated with measured in-game performance metrics. The processing circuitry may optionally include a correcting unit 326 and a boosting unit 328. The correcting unit 326 is configured to trigger corrective action when the estimated service quality does not mee that target service quality. The boosting unit 328 is configured to receive a request to boost network performance for one or more users of the cloud gaming service specified in the request, responsive to the request and trigger a performance boosting action to boost network performance for the one or more users specified in the request. Memory 330 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 320 for operation. Memory 330 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 330 stores a computer program 340 comprising executable instructions that configure the processing circuit 230 to implement the perform the methods herein described. A computer program 340 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 340 for configuring the processing circuitry 320 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 340 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium. Figure 8 illustrates an exemplary computing device 400 for training a service quality model to estimate the service quality of a cloud gaming service delivered over a communication network. The computing device 400 may comprise a network node In the communication network or may comprise a device external to the communication network. The computing device 400 generally comprises network interface circuitry 410, processing circuitry 420, and memory 440. The network interface circuitry 410 comprises a network adapter for coupling the computing device 400 to a communication network to enable communication with remote devices coupled to the network. The network adapter may for example, comprise a wired interface (e.g., Ethernet interface), optical interface (e.g., Synchronous Optical Network (SONET)), or wireless interface (e.g., Wireless fidelity (WiFi)). The processing circuitry 420 comprises one or more microprocessors, hardware, firmware, or a combination thereof. The processing circuitry 420 in one embodiment is configured to perform the methods herein described. According to an embodiment, the processing circuitry 420 comprises an obtaining unit 422 and a training unit 424. The obtaining unit 422 is configured to obtain training data comprising sample quality of service (QoS) metrics measurable in the communication network and correlated with measured in-game performance metrics. The training unit 424 is configured to train the QoE model to estimate service quality of the cloud gaming service using the training data. Memory 430 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuit 420 for operation. Memory 430 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 430 stores a computer program 430 comprising executable instructions that configure the processing circuit 240 to implement the perform the methods herein described. A computer program 430 in this regard may comprise one or more code modules corresponding to the means or units described above. In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random access memory (RAM). In some embodiments, computer program 430 for configuring the processing circuitry 420 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 430 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium. Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above. Embodiments further include a carrier containing such a computer program. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium. In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above. Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by a computing device. This computer program product may be stored on a computer readable recording medium.

Claims

CLAIMS What is claimed is:

1. A method implemented by a management system in a communication network of estimating service quality of a cloud-gaming service delivered over a communication network, the method comprising: collecting quality of service (QoS) metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network; and estimating the service quality of the cloud gaming service based on the QoS metrics using a machine learning (ML) model trained with sample QoS parameters correlated with sample in-game performance metrics.

2. The method of claim 1, wherein collecting QoS metrics comprises: receiving traffic probe reports per traffic flow; and calculating the QoS metrics based on based on the traffic probe reports.

3. The method of claim 2, wherein calculating the QoS metrics based on the traffic probe reports comprises calculating transport key performance indicators (KPIs) based on the traffic probe reports.

4. The method of claim 3, wherein the transport KPIs are aggregated per network function, per cell, per radio access network node, per network slice, per service type, per user group, or a combination thereof.

5. The method of claim 3 or 4, wherein the transport KPIs comprise one or more of the following: round trip time; video jitter; audio jitter; delay in uplink; delay in downlink; packets sent per second; packets received per second; bytes sent per second;bytes received per second; video packets lost per second; audio packets lost per second; and audio packets received per second.

6. The method of claim 1, further comprising: comparing the estimated service quality with a service quality target; and triggering a corrective action when the estimated service quality does not meet the service quality target.

7. The method of claim 4 or 5, wherein estimating service quality of the cloud gaming service based on the QoS metrics comprises estimating the service quality based on the aggregated transport KPIs.

8. The method of any one of claims 1 – 7 wherein estimating service quality of the cloud gaming service based on the QoS metrics comprises estimating the in-game performance metric based on the QoS metrics.

9. The method of claim 8, wherein the in-game performance metric comprises a score indicative of user performance in a game being played.

10. The method of claim 9, wherein the in-game performance metric comprises a score indicative of shooting performance in a game being played.

11. The method of any one of claims 1 - 10, further comprising: comparing the estimated service quality to a target service quality; and triggering corrective action when the estimated service quality does not meet the target service quality.

12. The method of claim 11, wherein triggering corrective action comprises one or more of: changing a policy; adding a network slice instance; and adding a network node.

13. The method of any one of claim 1 – 12, further comprising: receiving a request to boost network performance for one or more users of the cloud gaming service specified in the request; and responsive to the request; triggering a performance boosting action to boost network performance for the one or more users specified in the request.

14. The method of claim 13, further comprising: estimating the improvement in the QoS metrics and / or service quality as a result of the performance boosting action; and send a report quantifying the improvement in the QoS metric and / or service quality to a designated recipient.

15. A method of training a service quality model to estimate service quality of a cloud-gaming service delivered over a communication network, the method comprising: obtaining training data comprising sample quality of service (QoS) metrics measurable in the communication network correlated with measured in- game performance metrics; and training the service quality model to estimate service quality of the cloud gaming service using the training data.

16. The method of claim 15, wherein obtaining the training data comprises: measuring in-game performance over a testing period; and collecting, over the testing period, QoS metrics measurable in the communication network and correlated with measured in-game performance metrics.

17. The method of claim 15 or 16, further comprising selecting QoS metrics predictive of the service quality of the cloud gaming service to use as the training data.

18. The method of claim 17, wherein QoS metrics predictive of the service quality of the cloud gaming service to use as the training data comprises: training a test model using QoS metrics for a selected set of QoS parameters measurable in the communication network as a training set;training a reference model using QoS metrics for a superset of the selected set of QoS parameters; and comparing a performance of test model with a performance of the reference model to determine a predictive capability of the selected set of QoS parameters.

19. The method of claim 16, wherein training the service quality model using sample QoS metrics comprises training the service quality model using QoS metrics for the selected set of QoS parameters.

20. The method of any one of claims 15 – 19, wherein the sample QoS metrics comprise one or more sample WebRTC metrics.

21. The method of any one of claims 15 -20, wherein the selected QoS parameters comprise transport key performance indicators (KPIs) measurable in the communication network.

22. The method of claim 21, wherein the QoS metrics comprise transport KPIs aggregated per network function, per cell, per radio access network node, per network slice, per service type, per user group, or a combination thereof.

23. The method of claim 21 or 22, wherein the transport KPIs comprise one or more of the following: round trip time; video jitter; audio jitter; delay in uplink; delay in downlink; packets sent per second; packets received per second; bytes sent per second; bytes received per second; video packets lost per second; audio packets lost per second; andaudio packets received per second.

24. A network node configured to estimating service quality of a cloud gaming service delivered over a communication network, the network node being configured to: collect QoS metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network; and estimate the service quality of the cloud gaming service based on the QoS metrics using a machine learning (ML) model trained with sample QoS parameters correlated with measured in-game performance metrics.

25. The network node of claim 24, further configured to perform the method of any one of claims 2 -14.

26. A network node configured to estimating service quality of a cloud gaming service delivered over a communication network, the network node comprising: network interface circuitry configured for communication with other network nodes over a communication network; and processing circuitry configured to: collect QoS metrics predictive of in-game performance of a user of a cloud gaming service and measurable in the communication network; and estimate quality of experience (QoE) of the cloud gaming service based on the QoS metrics using a machine learning (ML) model trained with sample QoS parameters correlated with measured in-game performance metrics.

27. The network node of claim 26, wherein the processing circuitry is further configured to perform the method of claim any one of claim 2 - 15.

28. A computer program comprising executable instructions that, when executed by processing circuitry in a network node causes it to perform the method of claims 1 - 15.

29. A carrier containing a computer program of claim 28, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

30. A non-transitory computer-readable storage medium containing a computer program comprising executable instructions that, when executed by processing circuitry in a network node causes it to perform the method of any one of claims 1 – 15.

31. A computing device for training a service quality model to estimate service quality of an cloud gaming service, the system being configured to: obtain training data comprising sample quality of service (QoS) metrics measurable in the communication network and correlated with measured in-game performance metrics; and train the QoE model to estimate service quality of the cloud gaming service using the training data.

32. The computing device of claim 29, further configured to perform the method of any one of claims 17 -23.

33. A computing device for training a service quality model to estimate service quality of an cloud gaming service, the system comprising: interface circuitry configured for communication with other network nodes over a communication network; and processing circuitry configured to: obtain training data comprising sample quality of service (QoS) metrics measurable in the communication network and correlated with measured in-game performance metrics; train the QoE model to estimate service quality of the cloud gaming service using the training data.

34. The computing device system of claim 33, wherein the processing circuitry is further configured to perform the method of any one of claims 17 - 23.

35. A computer program comprising executable instructions that, when executed by processing circuitry in a computing device causes it to perform the method of claims 16 - 23.

36. A carrier containing a computer program of claim 35, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

37. A non-transitory computer-readable storage medium containing a computer program comprising executable instructions that, when executed by processing circuitry in a computing device causes it to perform the method of any one of claims 16 – 23.

Images