Cloud gaming monitoring apparatus and process

Abstract

A computer-implemented cloud gaming monitoring process, including: receiving mirrored data packets of general network traffic of a communications network, the data packets including data packets of one or more cloud gaming platforms, and other data packets unrelated to cloud gaming platforms; processing the received data packets to detect a cloud gaming session of a cloud gaming platform and a network client by identifying a subset of the received data packets as constituting network flows of a platform session of the cloud gaming platform and the network client; processing network flows of the network client to detect a gameplay session of the cloud gaming platform by identifying at least one of the network flows as being a gameplay management flow of the cloud gaming platform; determining network client and gaming server IP addresses of the gameplay management flow; using the network client and gaming server IP addresses to identify subsequent network flows as being gameplay session flows of the gameplay session; processing the identified gameplay session flows to determine metadata and generate volumetric metrics of the gameplay session flows; processing the metadata and volumetric metrics of the gameplay session flows to identify one of the gameplay session flows as being a downstream video flow or a gameplay management flow; and processing at least one of the downstream video flow and the gameplay management flow of the gaming session to generate one or more quality-of- experience (QoE) metrics for the game and network user.

Description

[0001] CLOUD GAMING MONITORING APPARATUS AND PROCESS

[0002] TECHNICAL FIELD

[0003] The present invention relates to cloud gaming and network traffic management, and in particular to an apparatus and process for detecting cloud gaming sessions in general network traffic and monitoring the detected cloud gaming sessions.

[0004] BACKGROUND

[0005] Cloud gaming, also known as Game-as-a-Service (GaaS), is a technology that allows users to play graphics-intensive games without having to own expensive hardware such as gaming consoles (e.g., PlayStation and X-Box) and PCs equipped with high-end graphics cards. Instead, the game graphics is rendered on powerful cloud-hosted servers, and the resulting video is streamed to the end-user's device. Though cloud gaming is still nascent, it is estimated to grow by 58.75% each year to become a $22.53 billion industry by 2030. Companies such as Nvidia, Amazon, Sony, and Microsoft, as well as a slew of smaller companies, already have commercial cloud gaming offerings in the market.

[0006] However, a significant barrier to the growth of cloud gaming is the high bandwidth that it demands from the network. Whereas a typical game (e.g., a shooting game such as Call-of-Duty or a sports game such as FIFA) played on a console or PC requires bit rates of only a few hundred Kilobits-per-second (Kbps) from the network, a cloud-based game demands two orders of magnitude more, namely tens of Megabits-per-second (Mbps). If the required bandwidth is not consistently available from the network at low latency, gaming experience becomes frustrating due to the resulting drop in frame rate (frames- per-second, fps), poor resolution, jerky movements, and teleportation. Internet Service Providers (ISPs) often bear the brunt of this frustration, leading to complaints, churn, and reputational damage.

[0007] ISPs have historically dimensioned their networks to aggregate bit-rates, agnostic to the application mix. However, this is ceasing to be cost-effective as access speeds increase, because elastic applications (e.g., large downloads) can consume unconstrained amounts of network bandwidth at any instant, thereby degrading performance for experience-sensitive applications (such as cloud gaming). ISPs are therefore looking for ways to distinguish (and potentially segregate) the latter - as one example, Comcast recently announced that it will support low latency forwarding in its cable broadband network for application streams marked by specific content providers (such as Apple and Nvidia). ISPs are therefore seeking to gain visibility and control into which content is worthy of preferential bandwidth or latency treatment, custom tune their network provisioning by locale in an on-demand manner to retain cost-efficiency, and continuously monitor user experience of sensitive applications to protect brand reputation.

[0008] It is desired to overcome or alleviate one or more difficulties of the prior art, or to at least provide a useful alternative.

[0009] SUMMARY

[0010] In accordance with some embodiments of the present invention, there is provided a computer-implemented cloud gaming monitoring process, including: receiving mirrored data packets of general network traffic of a communications network, the data packets including data packets of one or more cloud gaming platforms, and other data packets unrelated to cloud gaming platforms; processing the received data packets to detect a cloud gaming session of a cloud gaming platform and a network client by identifying a subset of the received data packets as constituting network flows of a platform session of the cloud gaming platform and the network client; processing network flows of the network client to detect a gameplay session of the cloud gaming platform by identifying at least one of the network flows as being a gameplay management flow of the cloud gaming platform; determining network client and gaming server IP addresses of the gameplay management flow; using the network client and gaming server IP addresses to identify subsequent network flows as being gameplay session flows of the gameplay session; processing the identified gameplay session flows to determine metadata and generate volumetric metrics of the gameplay session flows; processing the metadata and volumetric metrics of the gameplay session flows to identify one of the gameplay session flows as being a downstream video flow or a gameplay management flow; and processing at least one of the downstream video flow and the gameplay management flow of the gaming session to generate one or more quality-of-experience (QoE) metrics for the game and network user.

[0011] In some embodiments, the processing of the received data packets to detect a gameplay session includes: generating volumetric metrics of network flows of the received data packets, and comparing the generated volumetric metrics to known volumetric metrics of platform management flows of the cloud gaming platform.

[0012] In some embodiments, the volumetric metrics of network flows of the received data packets include packet sizes of network flows in upstream and downstream directions for known port numbers of the cloud gaming platform.

[0013] In some embodiments, the processing of the received data packets to detect a gameplay session includes: identifying requests from the network client to known service domains of the cloud gaming platform to generate a stateful list of active services of the cloud gaming platform requested by the network client; and comparing the stateful list to a corresponding list of known services of the cloud gaming platform; wherein a platform session of the cloud gaming platform is detected only if all of the known services of the cloud gaming platform are in the stateful list of active services of the cloud gaming platform.

[0014] In some embodiments, the process further includes classifying an access mode of the network user and the cloud gaming platform as one of: a web browser, a smartphone console application, a personal computer console application, a smartphone browser, a personal computer browser, and a gaming hardware console by comparing, for each of the access modes, a corresponding list of known services for the access mode and the cloud gaming platform to active services in the stateful list, wherein the access mode of the network user and the cloud gaming platform is classified as the access mode whose list of known services best matches active services in the stateful list.

[0015] In some embodiments, the processing of the received data packets to detect a gameplay session includes identifying network flows as being gameplay management flows of the cloud gaming platform if a port number and a sequence of packet payload sizes of the network flows in upstream and downstream directions match a corresponding known port number and sequence of packet payload sizes of gameplay management flows of the cloud gaming platform.

[0016] In some embodiments, the volumetric metrics include upstream packet rate, downstream packet rate, and throughput, and the metadata includes a 5-tuple representing source IP address and port number, destination IP address and port number, and protocol.

[0017] In some embodiments, processing the metadata and volumetric metrics of the gameplay session flows to classify each of the gameplay session flows includes classifying each of the gameplay session flows as being one of: a gameplay management flow, an upstream user input flow, an upstream audio flow, a downstream audio flow, a downstream video flow, and a streaming media and user input flow.

[0018] In some embodiments, processing the at least one downstream video flow of the gaming session to generate quality-of-experience (QoE) metrics for the game and network user includes generating a client-platform latency metric from timestamps of TCP packet pairs with matching sequence and acknowledgement numbers of the gameplay management flow.

[0019] In some embodiments, processing the at least one downstream video flow of the gaming session to generate quality-of-experience (QoE) metrics for the game and network user includes generating a gaming video frame rate metric from periodic patterns of packet payload sizes of the at least one downstream video flow or streaming media and user input flow.

[0020] In some embodiments, processing the at least one downstream video flow of the gaming session to generate quality-of-experience (QoE) metrics for the game and network user includes generating a gaming video graphics resolution metric from bandwidth consumption of the at least one downstream video flow or streaming media and user input flow.

[0021] In some embodiments, the cloud gaming monitoring process includes periodically repeating the step of processing the downstream video flow of the gaming session to provide quasi-continuous monitoring of the one or more quality-of-experience (QoE) metrics for the game and network user over time. In some embodiments, step of processing at least one of the downstream video flow and the gameplay management flow of the gaming session includes: processing the downstream video flow to generate video frame rate and graphics resolution QoE metrics; and processing the gameplay management flow to generate a latency QoE metric for the game and network user.

[0022] In some embodiments, the cloud gaming monitoring process further includes aggregating volumetric network traffic statistics of the identified gameplay session flows of all access modes of the network user.

[0023] In some embodiments, the cloud gaming monitoring process further includes:

[0024] (i) monitoring QoE metrics, including client-platform latency, gaming graphic resolution, and gaming video frame rate of detected user gameplay sessions for each network user to determine whether any of the monitored QoE metrics is degraded; and

[0025] (ii) responsive to determining that one or more of the monitored QoE metrics are degraded, reconfiguring network settings to improve the level of performance for the network user.

[0026] In some embodiments, the step of automatically reconfiguring network settings includes using one or more network application programming interfaces (APIs) to change one or more network settings to effect one or more of the following network reconfigurations: amending routing paths, prioritising traffic flows, and mapping traffic to network slices.

[0027] In accordance with some embodiments of the present invention, there is provided a computer-readable storage medium having stored thereon processor-executable instructions that, when executed by at least one processor of a cloud gaming monitoring apparatus, cause the cloud gaming monitoring apparatus to execute the cloud gaming monitoring process of any one of the above processes.

[0028] In accordance with some embodiments of the present invention, there is provided a cloud gaming monitoring apparatus of a communications network, including: a network interface to receive mirrored data packets of general network traffic of a communications network, the data packets including data packets of one or more cloud gaming platforms, and other data packets unrelated to cloud gaming platforms; a platform session detector to process the received data packets to detect a cloud gaming session of a cloud gaming platform and a network client by identifying a subset of the received data packets as constituting network flows of a platform session of the cloud gaming platform and the network client; a gameplay session detector to process network flows of the network client to detect a gameplay session of the cloud gaming platform by identifying at least one of the network flows as being a gameplay management flow of the cloud gaming platform, and to determine network client and gaming server IP addresses from the gameplay management flow; a gameplay session flow detector to identify subsequent network flows as being gameplay session flows of the gameplay session based on the network client and gaming server IP addresses; a gaming video flow detector to process the identified gameplay session flows to determine corresponding metadata and generate volumetric metrics of the gameplay session flows, and to process the metadata and volumetric metrics to identify one of the gameplay session flows as being a downstream video flow or as being a gameplay management flow; and a QoE monitor to process at least one of the downstream video flow and the gameplay management flow of the gaming session to generate one or more quality- of-experience (QoE) metrics for the game and network user.

[0029] In some embodiments, the cloud gaming monitoring apparatus includes a network reconfiguration component configured to use one or more network application programming interfaces (APIs) to change one or more network settings to effect one or more of the following network reconfigurations: amending routing paths, prioritising traffic flows, and mapping traffic to network slices.

[0030] In some embodiments, the platform session detector is configured to: generate volumetric metrics of network flows of the received data packets; and compare the generated volumetric metrics to known volumetric metrics of platform management flows of the cloud gaming platform; wherein the subset of the received data packets is identified as constituting network flows of a platform session of the cloud gaming platform and the network client if the generated volumetric metrics match the known volumetric metrics of platform management flows of the cloud gaming platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Some embodiments of the present invention are hereinafter described, by way of example only, with reference to the accompanying drawings, wherein:

[0032] Figure 1 is a schematic diagram of a testbed network with a measurement server for determining network traffic attributes characteristic of a cloud gaming platform to configure a cloud gaming monitoring apparatus in accordance with embodiments of the present invention;

[0033] Figure 2 is a schematic diagram illustrating the various network flows of cloud gaming sessions, for the example of Nvidia's GeForce NOW cloud gaming platform ("GFN");

[0034] Figure 3 is a chart illustrating the time evolution of platform and gameplay sessions and their network flows during a GFN cloud gaming session accessed via its PC console application;

[0035] Figures 4 and 5 are charts showing the time evolution of different types of network flows of GFN cloud gaming sessions accessed via web browser and mobile smart phone console applications, respectively;

[0036] Figure 6 is a chart showing the time evolution of gameplay session flows of a GFN cloud gaming session;

[0037] Figures 7 to 10 are charts showing the time evolution of packet payload sizes during representative downstream video flows of gameplay sessions, respectively for: a macOS console application, a Windows console application, multiple packets containing data of one video frame, and a web browser executing on a Windows personal computer;

[0038] Figure 11 is a flow diagram of a cloud gaming monitoring process in accordance with some embodiments of the present invention;

[0039] Figure 12 is a schematic flow diagram illustrating the detection of a GFN platform session and the identification of the access mode of the platform session;

[0040] Figure 13 is a schematic diagram illustrating the classification of gameplay session flows using the flow 5-tuple and the volumetric attributes, and showing parameter values determined for GFN;

[0041] Figures 14 to 16 are graphs showing the video frame rate as a function of time, illustrating inferred deviations of nominal frame rates during gameplay, respectively for: a selected frame rate of 60 fps, a selected frame rate of 30 fps, and a selected frame rate of 60 fps but during commencement of a gameplay session;

[0042] Figures 17 to 19 are bar graphs illustrating GFN cloud gaming usage patterns measured by the cloud gaming monitoring apparatus for the three access modes of personal computer console application, mobile smart phone console application, and web browser, respectively showing counts of gaming sessions, the duration of each gaming session, and the bandwidth as a function of time during gameplay sessions;

[0043] Figures 20 to 22 are bar charts illustrating respective QoE metrics for GFN cloud gaming, respectively: frame rate, graphics resolution, and wired connection latency;

[0044] Figure 23 is a block diagram of a cloud gaming monitoring apparatus in accordance with an embodiment of the present invention;

[0045] Figure 24 is a block diagram showing cloud gaming monitoring components of the cloud gaming monitoring apparatus of Figure 23;

[0046] Figures 25, 26, and 27 are charts showing the time evolution of different types of network flows of cloud gaming sessions using Microsoft's Xbox Cloud Gaming platform accessed via Xbox hardware console, PC web browser, and mobile smart phone console application, respectively;

[0047] Figure 28 is a schematic flow diagram illustrating the detection of a platform session and the identification of the access mode of the platform session for Xbox Cloud Gaming platform;

[0048] Figure 29 is a schematic diagram illustrating the classification of gameplay session flows using the flow 5-tuple and the volumetric attributes, and showing parameter values determined for Xbox Cloud Gaming platform;

[0049] Figure 30 is a generalised schematic flow diagram illustrating the detection of a platform session and the identification of the access mode of the platform session that is applicable to cloud gaming platform regardless of its provider; and Figure 31 is a generalised schematic diagram illustrating the classification of gameplay session flows using the flow 5-tuple and the volumetric attributes, the criteria or threshold of which are obtained from training process on ground-truth packet traces for a cloud gaming platform regardless of its provider. DETAILED DESCRIPTION

[0050] The inventors have identified a need to give network operators such as Internet Service Providers (ISPs) the means to measure the prevalence of, and user experience of, cloud gaming over their network infrastructures so that they can dynamically take remedial action in response to poor user experience measures, and / or to pre-emptively provision their networks to meet expected demands. However, it was not at all clear whether it would be technically feasible to identify cloud gaming sessions from other network traffic, and even if that could be done, what attributes of cloud gaming sessions could be gleaned, and what if any quantitative metrics could be generated. Prior studies of cloud gaming have been limited to considering video processing delays on cloud servers, energy consumption on mobile devices, and monitoring connection quality at a cloud gaming streaming server operated by a cloud gaming provider ( / .e., where cloud gaming traffic is the only network traffic and is known in advance) to adjust rendering of gameplay video, which are not relevant to ISPs.

[0051] In view of the above, the inventors performed extensive research to investigate whether it would be technically feasible to detect cloud gaming sessions within general network traffic, and if so, what quantitative metrics and other attributes could be determined.

[0052] Based on this research, and as described herein, the inventors have developed practical methods that can be deployed by network operators such as ISPs to enable fine-grained visibility into cloud gaming behaviours and experience, so that they can be actively involved in, rather than be cut out of, managing service delivery quality.

[0053] In particular, embodiments of the present invention include a cloud gaming monitoring apparatus and process that address the technical limitations of the prior art by detecting and identifying cloud gaming sessions from general network traffic, and optionally also classifying user activities of those cloud gaming sessions. The cloud gaming apparatus and process described herein enable telecommunications network operators (including, but not limited to, ISPs) to monitor and quantify cloud gaming network usage and user experience of their network users so that they can quantify and improve the performance of their networks for cloud gaming applications, both dynamically in response to indications of poor user experience, and also to make better planning and provisioning decisions. In particular, the cloud gaming monitoring apparatus can use network application programming interfaces (APIs) to automatically change network settings in response to the quantified cloud gaming network usage and / or user experience of their network users, as described below.

[0054] Table 1 below lists the popular cloud gaming platforms operated by the dominant "tech giants", together with their supported operating systems (OS) and software terminals. For convenience, the different available combinations of OS and software terminal are also referred to herein as "user terminals", or simply "terminals". The table shows that all of the popular cloud gaming platforms are accessible on all of the major operating systems, and via both dedicated bespoke applications (also referred to herein as 'console' applications) and web browser applications, except that XBox Cloud Gaming does not support macOS, and is only available via its console application.

[0055] Table 1: Popular cloud gaming platforms currently on the market and their specifications.

[0056] Embodiments of the present invention are described herein in the context of their application to a representative cloud gaming platform, namely Nvidia's cloud gaming platform, GeForce NOW (GFN). This particular platform was chosen for the following reasons:

[0057] (a) it is currently recognized as the leader in the global cloud gaming market;

[0058] (b) it supports a rich collection of multiplayer games from all major game publishers, including Steam, Ubisoft, and Epic Games;

[0059] (c) it has local servers hosted in the inventors' geographic region;

[0060] (d) the inventors had access to the personnel operating the GFN servers for the region, making it easier to seek clarifications and corroborate findings; and (e) it is accessible broadly, both via the browser and a bespoke app, on desktops ( / .e., macOS and Windows) and mobile devices ( / .e., android and iOS), thereby providing a rich set of conditions under which it can be studied.

[0061] To identify a specific cloud gaming platform, the apparatus first needs to be trained to recognise the unique traffic characteristics of that platform. This generally involves capturing and analysing the isolated network traffic of the cloud gaming platform to identify the establishment of various flows, and benchmark their volumetric patterns associated with gameplay management, user input, and audio / video streaming, while identifying the differences between the user playing on a browser versus a dedicated app. It also involves identifying attributes in the traffic that aid in user experience estimation, such as sequence numbers for measuring latency, and marker packets or stochastic patterns in packet payload sizes for detecting frame boundaries.

[0062] Once these network traffic characteristics of the specific platform have been quantified, the cloud gaming monitoring apparatus and process can be applied to detect cloud gaming sessions of the platform in general network traffic, and measure gameplay experience of that platform. The described embodiments use a stateful mapping mechanism that tracks service domains accessed by active user flows so as to detect the start of a cloud gaming session, and identify the user's operating system and terminal type. The gameplay flows are then categorised as gaming management, user input, or video streaming, based on volumetric attributes. Finally, user experience measures are determined, including client-platform latency (between the gamer device and cloud platform), video frame rate, and video graphic resolution.

[0063] In order to illustrate how embodiments of the present invention can be applied to an arbitrary cloud gaming platform, the following describes, by way of example, how the network traffic of the GeForce NOW platform was captured and characterised to generate corresponding network traffic attributes for use by the cloud gaming monitoring apparatus and process to distinguish network traffic of the platform from other network traffic, and automatically detect sessions of the platform, and for each session, to automatically detect the user access method (OS and software terminal), and quantify the user's experience of cloud gaming on the platform.

[0064] Figure 1 is a schematic diagram of a testbed network for determining network traffic attributes of cloud gaming. A network traffic measurement server 102 is arranged to receive mirrored network traffic flowing 104 between a network access gateway 106 and the Internet 108. The access gateway 106 provides access to the Internet 108 from the testbed network, which includes wired 110 and wirelessly connected 112 user devices capable of cloud gaming, such as desktop computers, laptop computers, mobile phones, and tablet computers. As described in detail below, the network traffic measurement server 102 processes the network traffic 104 between the user devices 110, 112 and one or more remote servers 114 of a cloud gaming platform to generate corresponding network traffic attributes characteristic of the cloud gaming platform. The resulting attributes can be used by the cloud gaming monitoring apparatus and process to automatically identify sessions of the cloud gaming platform from general network traffic.

[0065] Overview of Cloud Gaming Sessions

[0066] Cloud gaming platforms serve as intermediaries that connect user devices and individual game servers. These platforms receive input from gamers, including mouse movement, keyboard input, and upstream audio, and perform graphic and gaming computations on their behalf. Real-time gameplay scenes are streamed back to users via video and audio services. Therefore, a typical cloud gaming 'session' actually includes two types of session, for platform administration and actual gameplay, respectively.

[0067] The following describes an example gameplay of the popular first-person shooting game "CS:GO" on the GeForce NOW platform. After logging into the cloud gaming platform via either a console application or a web browser, the gamer is directed to a "platform session", during which the gamer can browse available games, and select one to play. The platform session also provides access to graphics settings such as resolution and frame rate. Once a game has been selected and gameplay is about to commence, the optimal cloud server for this gameplay is automatically selected on the basis of a set of network measurements for key performance metrics, such as latency and throughput between the gamer device and candidate gaming clusters.

[0068] After the initial setup, the "gameplay session" begins, and the gamer plays the selected game, in this example being the CS:GO shooting game. As discussed further below, the gameplay session places significant demands on network throughput and latency, which directly impact the quality of experience ("QoE") for cloud gaming. After exiting a gameplay session, the gamer enters a further platform session to either select / start another gameplay session, or to finish the platform session and thus leave the cloud gaming platform.

[0069] It should be clear from the description above that in this specification the term "cloud gaming session" refers to a composite session in which a user logs into a cloud gaming platform, plays at least one game for a period of time, and then logs out of the cloud gaming platform. As described above, this involves at least three distinct sessions involving different servers and network flows: an initial "platform session" (users logs in to the cloud gaming platform, selects a game to play, etc), a gameplay session (user plays the selected game), and a further platform session (user returns to the cloud gaming platform after leaving the game, can select another game if desired, in which case a further gameplay session would commence, or otherwise exits the platform session to log out of the cloud gaming platform).

[0070] Cloud gaming network traffic characterization

[0071] Dataset

[0072] The GeForce NOW cloud gaming platform can be accessed via dedicated 'console' applications and web browser applications on PCs (macOS and Windows) and mobile devices ( / .e.ziOS and android). The measurement testbed of Figure 1 was used to capture packet trace (PCAP) files during cloud gaming sessions of a real-time shooting game (CS:GO), and a massively multiplayer role-play game (Path of Exile) for all of the above modes of access, which are categorized as either desktop console application, mobile console application, or browser. Because the testbed network is isolated and private, it was possible to ensure that all of the captured packet traces represent only cloud gaming network traffic, and not any other type of network traffic.

[0073] The following description primarily focuses on the network traffic characteristics of gameplays on the console application installed in a macOS desktop, while also describing the differences to other modes of access ( / .e.zChrome browser in macOS, console application in Windows, Chrome browser in Windows, console application in android, and Safari browser in iOS). To validate the network traffic characteristics, traffic traces of 20 cloud gaming sessions for each type of user access mode (terminal) were further collected. The 20 cloud gaming sessions for each access mode cover three commonly available graphic resolutions, namely: full high-definition (FHD), high definition (HD), and standard definition (SD), with video frame rates of either 30fps or 60fps.

[0074] Anatomy of Service Flows

[0075] Figure 2 is a schematic diagram illustrating the different types of network communications between a gamer device / terminal 202 and the cloud gaming platform 204, as determined from the captured network traffic. A cloud gaming session involves three types of network traffic flow, respectively providing platform administration (shown as dotted lines in Figure 2), platform management (dashed lines), and gameplay (solid lines).

[0076] Specifically, once a user opens the cloud gaming console application or browser 202, entering a platform session, a series of administration flows (labelled ©in Figure 2) are initiated for administrative support, such as content management systems (CMS), API utilities, login portals, and account management. After the user selects a game to play, a number of platform management flows (@in Figure 2) are initiated to perform network diagnosis and cloud server selection. These flows provide critical roles in the successful initiation of the subsequent gameplay. During actual gameplay sessions, three types of flow are initiated, each serving a specific purpose. Firstly, there are gaming management flows ((f)) responsible for controlling the gameplay session (e.g., measuring run-time latency) and exchanging metadata (e.g., coordinating interactions between the client and the platform). In addition, there are specific gameplay flows dedicated to user input (from mice, keyboard and microphone) and streaming media contents ( / .e., downstream video and audio), respectively labelled as ® and ®. After a gameplay session, platform administration flows @ return the user to the platform session.

[0077] Figure 3 is a time-series chart illustrating the various service flows as a function of time as two games were played via the macOS console application. The user activity can be separated into six stages, as indicated by respective labels 1 to 6 across the top of the chart, as follows. The user: logged into the cloud game platform and selected a first game (CS:GO) to play in stage 1; played the CS:GO game in stage 2; returned to the platform in stage 3; entered the login and character selection page of a second game (Path of Exile) in stage 4; entered the world of Path of Exile in stage 5, and finished cloud gaming play in stage 6. Each of these six activity stages correspond to either a platform session or a gameplay session, as indicated by respective shadings across the top of the chart. The median throughput of each flow is indicated by the corresponding line thickness.

[0078] The labels on the vertical axis of the chart provide, for each of a representative subset of network flows, a corresponding simplified service prefix (extracted from SNI or DNS records) and identifiable port number. The labels for other flows are not included for readability.

[0079] Platform Administration Flows

[0080] Platform administration flows are shown with solid fill in Figure 3. They are all sent to the service port TCP\443, and are all HTTPS flows toward the provider root domains ( / .e.znividia.com, geforcenow.com, and gefore.com) for administrative services, as indicated in their subdomain prefixes, such as content management system (CMS) (cms) and frontend APIs (gx-target-experiments-frontend-api). Depending on their service purposes, some of the flows (e.g., login and userstore) are only active during the platform session, whereas others (e.g., cms and events) remain active during the subsequent gameplay sessions. As for their volumetric profiles, all of them have only a small (or even negligible) amount of volume usage, less than several Megabytes.

[0081] Upon comparing the platform administration flows across different user terminals ( / .e., different combinations of OS and software terminal), it was observed that sessions via the Chrome browser (an example is shown in Figure 4) have most of the service flows seen in this example, except for cms and als which are related to high-performance graphics. Also, unlike personal computer sessions, cloud gaming sessions via the android mobile console application utilise only a limited number of platform administrative flows (an example is given in Figure 5) that cover only essential services such as login, event, and userstore.

[0082] Platform Management Flows

[0083] As shown by the shaded horizontal bars in Figure 3, the platform management flows involve a relatively low amount of traffic compared to the platform administration flows. The platform management flows are all HTTPS flows sent to service port TCP\443 of the GeForce NOW cloud server clusters, which are all associated with the cloud cluster domain nvidiagrid.net.

[0084] Unlike the administration flows that support user administration on the platform, such as login, events, and user store, the platform management flows service critical tasks relating to the delivery of subsequent gameplay sessions. In stage 1 of Figure 3, the first management (shaded) flow is directed towards the subdomain prefix gfnpc.api. entitlement-prod, which grants the client access to the GeForce NOW cloud servers. This is followed by multiple flows toward subdomain prefixes in the format of server_[location]_pnt, which serve the purpose of cloud server selection by measuring network performance metrics between the user terminal and various available cloud servers As indicated by the service prefixes of the labels in Figure 3 from top to bottom, the selection process progresses from a regional node to a city node, and then to individual cloud servers.

[0085] Despite the indispensable roles of platform management flows, i.e., system access and server selection, no significant differences were observed across different user access modes, except that four flows toward the service prefix img of the production system domain nvidiagrid.net are seen in platform sessions on android mobile but not on iOS mobile and personal computers.

[0086] Gameplay Session Flows

[0087] Once a suitable cloud server is successfully selected via the platform management flows, the actual gameplay session is started. As depicted by the crosshatched bars in stage 2 of Figure 3, five gameplay session flows are initiated in parallel for CS:GO gameplay. This remains consistent for all gameplays via console applications on both mobile and PC devices. Similarly, as shown in stages 4 and 5 of Figure 3, the same combination of five gameplay session flows is observed for Path of Exile gameplay.

[0088] Specifically, the first gameplay session flow is always directed to destination service port TCP\322 on the cloud server. This is followed by four RTP (Real-Time Transport Protocol) flows originating from client ports UDP\49003, UDP\49004, UDP\49005, and UDP\49006 toward dynamically selected service ports on the cloud server. According to Nvidia Support, the four client ports are assigned for downstream audio, upstream audio, downstream video, and user input, respectively. The dynamic selection of service ports is performed by the first TCP flow for gaming management, as illustrated in Figure 2. Notably, the flows originating from UDP\49005, responsible for downstream video, consistently consume larger amounts of bandwidth than all other gameplay session flows. In the 10-minute CS:GO gameplay (stage 1), this resulted in downstream video data transfers of 1422MB, while in the Path of Exile gameplay (stages 4 and 5), it reached 1961MB. The flows for downstream audio, upstream audio, and upstream user input consumed several MB, several tens of MB, and several tens of MB data transfers, respectively.

[0089] Similar insights are observed for gameplay session flows across console applications on both mobile and personal computers. However, in browser-based gameplay, their gaming management flows is delivered using the WebRTC protocol toward the service port TCP\49100, as shown in Figure 4. Additionally, gameplays on browsers utilize a single UDP flow to carry both downstream media content and upstream user input, in contrast to the separate flows seen for console applications.

[0090] Given that user gaming experience on cloud platforms is predominantly influenced by the gameplay session flows, their flow-level volumetric profiles and packet statistics are described below.

[0091] Profile of Gameplay Session Flows

[0092] Gameplay session flows (represented by the crosshatched bars in Figure 3) are critical to the cloud gaming experience. As described below, for flows that are active during a gameplay session, their volumetric profiles are analyzed; in particular, their bandwidth consumptions are benchmarked across different levels of graphic quality, including frame rate and resolution; and video frame rates are identified by packet size patterns of flows carrying downstream video data.

[0093] Flow Volumetric Profile

[0094] Figure 6 is a time-series chart showing the inbound and outbound packet rates of the five gameplay session flows for the example gameplay described above, respectively providing gameplay management, upstream user input, upstream audio, downstream audio, and downstream video. First, the gameplay management flow, represented by the inbound and outbound packets towards or from the service port TCP\322, exhibits a low packet rate of only one pair of packets every two seconds ( / .e.z0.5pps), and can serve as a reliable metric of network latencies between the user and the cloud platform by tracking the sequence and acknowledgement numbers in TCP packet headers.

[0095] Second, the upstream user input flow for keyboard and mouse actions exhibits packet rates ranging from 11 to 267 pps, depending on the user's activity level. As shown in Figure 6, the baseline packet rate (of llpps) occurs when the user is actively moving the mouse or typing on the keyboard, such as during the matching phase of a shooting game (beginning of stage 2 in Figure 6), or the opening cinematic of a role-play game (beginning of stage 4). It is observed that the outbound and inbound packet rates of this flow are often equal. Regarding the throughput of the user input flow, the outbound direction exhibits minimum and maximum values of 1Kbps and 82Kbps, respectively, which are approximately ten times larger than those observed in the inbound direction.

[0096] Third, the two flows for upstream and downstream audio both have constant packet rates and throughputs in their respective directions. Specifically, the upstream audio flow exhibits a packet rate of lOOpps with a throughput of 15Kbps, while the downstream audio flow has a packet rate of 300pps with a throughput of 37Kbps. These values remain consistent regardless of the status of the input / output voice, such as being muted, at low volume, or at high volume. In their opposite directions (e.g.zthe inbound direction for upstream audio flow), both of them constantly have a 2pps packet rate with 156bps throughput.

[0097] The most bandwidth consuming flow type, downstream video flows, exhibits a packet rate that ranges from approximately 300pps during less active scenes (as shown in Figure 6) to 3000pps. The corresponding bandwidth consumptions for these packet rates are 3Mbps and 34Mbps, respectively. During active gameplay periods, such as stage 5 in Figure 6, the bandwidth consumption mostly remains at the upper bound level. On the other hand, the low packet rate is observed during inactive periods, which often coincide with the inactivity of the user input flows. However, there are exceptions during static scenes with frequent user inputs, such as the login scene of a gameplay.

[0098] Similar patterns in terms of packet rates and bandwidth consumption are observed for gameplay session flows across both desktop and mobile console applications. However, for gameplay sessions on browsers, there is only one gameplay session flow that combines all four flows for user input and media. In this case, the volumetric pattern of the flow is primarily dominated by the downstream video content.

[0099] The numerical results described above are specific to a video configuration with a frame rate of 60fps, and a resolution of 1920x1080 (FHD). However, users have the flexibility to choose from a wide range of frame rates and resolutions, either statically or dynamically adjusted based on the network conditions during gameplay. The bandwidth consumption of downstream video flows for different graphic configurations are described below.

[0100] Bandwidths of Downstream Video Flows across Video Configurations

[0101] Frame rate and graphics resolution are configurable parameters for cloud gameplay. To determine the effect of these parameters on bandwidth consumption for downstream video flows, different frame rates (30 or 60fps) and graphic resolutions (ranging from 1920x1200 to 1024x768) were selected. For browser sessions where only one gameplay flow is present, the bandwidth consumption was also measured because it is primarily influenced by downstream video content. As discussed above for Figure 6, the packet rate and throughput for downstream video flows are capped at a peak level during active gameplay scenarios. Table 2 below shows the observed peak bandwidth consumptions of those downstream video flows for different graphics configurations.

[0102] Table 2: Representative Peak bandwidth consumption of downstream video flows for frame rates of 30 fps and 60 fps during active gameplay, and for full high-definition (FHD), high- definition (HD), and standard-definition (SD) graphics resolutions, respectively.

[0103] The three columns in Table 2 correspond to different screen resolutions, namely: full high-definition (FHD), high-definition (HD), and standard definition (SD). FHD is the best graphic quality, while HD and SD are less optimal graphic configurations, and thus may provide a subpar gaming experience. As expected, lower frame rates and coarser- grained graphic resolutions result in lower bandwidth consumption, both in active and less active scenarios. Consequently, by examining the peak bandwidth consumption of video flows and the current frame rate (which can be inferred from packet size patterns as discussed below), it is possible to infer the current graphic resolution of an active cloud gameplay.

[0104] Packet Size Patterns in Downstream Video Flows across Frame Rates

[0105] The current frame rate can be determined from the number of packets with a frame marker set in the RTP header of downstream video flows. In RTP flows, the frame marker indicates the completion of the currently transmitted video frame. Accordingly, a count of packets with the frame marker in the downstream video flow can be used to accurately determine the frame rate of a cloud gameplay. However, in a high-speed network, this method requires decoding RTP packets, which could introduce non- negligible overheads (e.g., via multiple sequential packet parsers decoding respective packet layers). Moreover, frame marks might not always be set correctly, or might be encrypted, preventing decoding of RTP headers. For a lightweight and robust method, the inventors observed characteristic patterns in the packet sizes of downstream video flows to determine the frame rate without decoding the Ethernet / IP / UDP / RTP headers. This approach ensures resilience even when frame marks are not available.

[0106] Some example downstream video flows are shown in Figure 7. In general, a video frame is carried by two types of RTP packets. The first type carries all or the majority of video frame data, and has a fixed large packet size (e.g., 1466 bytes in console applications). The second type consists of smaller-sized packets (e.g., 216 bytes) that carry frame markers, indicating the completion of a frame transmission. During gameplay sessions, the number of data packets required to transmit a video frame is dynamically adjusted, based on the amount of video data in the frame. This can range from one packet to multiple packets, as visually depicted in Figure 9. Importantly, there is always one small-sized marker packet indicating the end of each frame, as shown in Figure 7. These frame marker packets can have larger sizes (but still smaller than the size of data packets) to accommodate any remaining video data from the previous packets, as shown in Figure 7. The downstream video flows exhibit a consistent pattern where the number of packet groups (comprising one or more data packets followed by one marker packet) aligns perfectly with the current frame rate. An example of this pattern can be seen in Figure 8, with 30 packet groups within one second for a frame rate of 30fps. Thus the size patterns of these packet groups accurately identify the frame rate being received by the user's terminal in real-time.

[0107] In contrast to gameplay via console applications, browsers are slightly different as there is only one RTP flow responsible for carrying downstream video, audio, and user input data. In Figure 10, each vertically aligned group of packets consists of several data packets corresponding to a video frame, followed by a frame marker packet and three additional packets for downstream audio, upstream audio, and user input. Despite this difference, the consistent pattern observed in each packet group via browsers still aligns with the delivery of video frames.

[0108] Cloud Gaming Detection and Monitoring

[0109] Having identified network traffic attributes of the cloud gaming platform as described above, these attributes can subsequently be used by the cloud gaming monitoring apparatus and process to automatically detect cloud gaming sessions in general network traffic, and, for each detected cloud gameplay session, to identify the corresponding user access mode and continuously generate quality-of-experience (QoE) metrics during gameplay, as summarised in Figure 11.

[0110] Detecting Cloud Gaming Sessions

[0111] As described above and shown in Figures 2 and 3, cloud gameplay typically involves two types of sessions: platform sessions (for platform administration, game browsing, and cloud server selection) and gameplay sessions. The platform sessions exhibit identical usage of service flows, regardless of access mode. The flows of each gameplay session can be used to directly determine the corresponding user's cloud gaming experience.

[0112] As described below, the flows directed towards specific service domains can identified from the Server Name Indication (SNI) field in the SSL headers, when available. Alternatively, if the SNI field is inaccessible, for example if encrypted-SNI (ESNI) is adopted in the future, then an encryption-agnostic approach can be used to overcome this challenge. Platform Sessions and User Terminal Identification

[0113] As described above, a user enters platform sessions in the initial phase after launching the cloud game platform (e.g., stage 1 in Figure 3) and also in a subsequent phase after finishing each gameplay session (e.g., stages 3 and 6 in Figure 3). As discussed above, when a platform session begins, a series of HTTPS flows are initiated towards different service prefixes of the cloud gaming domains. These HTTPS flows contain service names in the server name indication (SNI) fields of their SSL handshakes, which precede the encrypted application data communication. Accordingly, the cloud gaming monitoring apparatus and process detects platform sessions by monitoring flows toward each service domain, extracted from their respective SNI records (if available).

[0114] Cloud gaming traffic flows of platform sessions are categorised as representing either core services or terminal-specific services. Core service flows are always initiated during the starting phase of platform sessions, regardless of the user access mode, in contrast to terminal-specific service flows, which are specific to user access modes. For example, flows directed toward login. nvidia are always active, while flows toward play. nvidia only occur in platform sessions via browsers. Additionally, the majority of core service flows follow a sequential order, starting from user login and progressing to server selection. In contrast, terminal-specific service flows often exhibit randomness in their sequence.

[0115] As shown in Figure 12, for each known cloud gaming platform, the cloud gaming monitoring apparatus and process uses wildcard matching of a corresponding stateful list of active service domains to a corresponding table of known service domain names for the cloud gaming platform to detect the start of cloud gaming play (through core services) and identify the user access mode ( / .e.zdesktop console application, mobile console application, or browser). For example, as illustrated by the top portion of Figure 12, the exact match of flows in the core service table constitutes the successful detection of a cloud gaming session. Additionally, a confident match in the desktop console application table ( / .e., wherein the number of matching services in that table is greater than the number of matching services in the tables for the other user access modes) determines that the cloud game session is being played on a desktop console.

[0116] Gameplay Sessions and Gameplay Flow Classification

[0117] As described above, after starting a gameplay on the cloud platform, one SSL-encrypted TCP gaming management flow and one or more (one for browser and four for console applications) UDP flows are initiated for media and user input. Cloud gaming sessions from console applications have their TCP gaming management flows directed to service port number TCP\322, while browsers use port TCP\49100. Irrespective of the user access mode, all gaming management flows have their service name (extracted from SNI if available) following a consistent pattern of a-b-c-d. pnt.nvidiagrid.net, where the a-b-c-d in this pattern represents the IP address a.b.c.d of a cloud server assigned to a gameplay session, which is also the server IP address for the subsequent gameplay UDP flows.

[0118] The cloud gaming monitoring apparatus and process identify gameplay session flows based on their service names for active users detected in the platform sessions. Subsequently, the gameplay session flows are classified by purpose (as gameplay management, upstream user input, upstream audio, downstream audio, or downstream video) from their flow metadata ( / .e., a five-tuple representing the corresponding source IP address and port number, destination IP address and port number, and protocol) and volumetric profiles (packet rate and volume) .

[0119] As shown in Figure 13, the first step 1302 of the process is to classify gameplay management flows. A candidate TCP flow (identified by its service name) 1304 is sent from an active user to port TCP\322 (console application) or TCP\49100 (browser). The client and server IP addresses of the gameplay management flow are stored in a runtime database 1306 for detection of subsequent gameplay UDP flows that are initiated shortly afterward (e.g., within 0.5s).

[0120] The second step 1308 is to classify gameplay media and user input flows. In the case of browser-based sessions, a gameplay UDP flow can either carry media and user input data or provide support for the STUN WebRTC service, which can be identified based on its packet rate. For gameplay UDP flows in console application-based sessions, they can serve different purposes, including downstream video, downstream audio, upstream audio, or user input. Classification of these flows is based on their volumetric profiles, specifically the packet rate and throughput, in both inbound and outbound directions, as depicted in Figure 13. For console applications, it could be argued that the functionality of a gameplay UDP flow can be easily determined by its client port number as discussed above. However, the presence of network address translation (NAT) in ISP networks typically obfuscates the client port numbers of users, rendering them an unreliable classification criterion in comparison to volumetric profiles. The following discusses the continuous monitoring of the downstream video flows via console applications, the flows carrying media with user input via browsers, and gaming management flows as an effective approach to infer user experience per cloud gameplay session.

[0121] Encryption-Agnostic Service Flow Identification

[0122] Recent proposals for SNI encryption techniques have sparked discussions in the industry, and could potentially be adopted in the near future. If that happens, current network analytics relying on SNI signatures will immediately become ineffective. To address this anticipated difficulty, the cloud gaming monitoring apparatus and process can include an encryption-agnostic classification process as an alternative to reliance on service names extracted from the SNI field. This alternative process can identify flows with their service names associated with platform and gameplay sessions from packet payload sizes, without the need to inspect TLS headers.

[0123] To detect platform and gameplay management flows without inspecting SNI, the cloud gaming monitoring apparatus and process determines the sequence of payload sizes in the first few packets of a TCP flow, which contain predefined service requests after the usual TCP three-way handshake. As determined from the training process described above, representative upstream and downstream packet payload sizes in gameplay management flows with their destination port numbers across different operating systems are provided in Table 3 below. Similarly, representative examples of signatures for platform flows are shown in Table 4 below. The inventors find that the precise specification of signatures for platform flows enables 100% accuracy in detecting cloud gaming sessions. Table 3: Packet payload sizes (in bytes) after three-way handshakes of gameplay TCP management flows for different operating systems (OS) observed in both upstream t and downstream / directions.

[0124] It is important to note that this encryption-agnostic detection is for platform HTTPS flows and gameplay management TCP flows. The detection of gameplay UDP flows does not rely on service name signatures in any case, and is already encryption-agnostic, meaning it will not be affected by future SNI encryption.

[0125] Table 4: Packet payload sizes (in bytes) after three-way handshakes of platform service flows observed in the upstream t direction

[0126] Measuring Gameplay User Experience

[0127] A user's experience in a cloud gameplay session is primarily determined by three factors: the synchronization speed of the user's mouse / keyboard input with the cloud platform (measured by client-platform latency), the smoothness of the streamed gaming scene (measured by video frame rate), and the clarity of the gaming graphics (indicated by graphic resolution).

[0128] The following describes metrics to monitor these three key performance indicators, derived from real-time volumetric statistics of gameplay session flows. The metrics are computed from transport-layer headers and packet sizes, and therefore are agnostic to the encryption of application-layer headers and payloads.

[0129] Client-Platform Latency

[0130] The first metric is client-platform latency, which represents the response time from the moment a gamer inputs commands with their keyboard or mouse until those commands are executed by the cloud platform. As discussed above, a single gaming management flow over TCP remains active throughout the entire gameplay session. This flow is detected volumetrically, as described above.

[0131] As described above, the gaming management flow exhibits a constant packet rate of one pair of TCP packets every two seconds, where each pair consists of an upstream and downstream packet with matching sequence and acknowledge numbers in their TCP headers.

[0132] To measure the real-time latency experienced by a cloud gamer, the cloud gaming monitoring apparatus and process continuously monitor the arrival timestamps of each packet pair (identified by their sequence and acknowledgement numbers) within the gaming management flow. Specifically, the timestamps tup and tdown are tracked, and used to calculate the latency At between tdown and tup .

[0133] Gaming Video Frame Rate

[0134] The second metric for user experience is the video frame rate being streamed to the cloud gamer. As discussed above, a higher frame rate (e.g., 60fps) imposes stricter network requirements in terms of higher bandwidth and lower packet loss, but provides a smoother gaming video experience.

[0135] To monitor this QoE metric, the cloud gaming monitoring apparatus and process leverage the periodic patterns of packet payload sizes observed in downstream video flows for both console applications and browsers during each gameplay session. These periodic patterns serve as a direct measure of the video frame rate, as explained above and shown in Figures 7 to 10. The process that measures frame rate from downstream video flows is shown as pseudocode below.

[0136] The process processes downstream packets in a gameplay video flow, using a configurable interval AT (set to a value of 1 second in the described embodiments) that determines the frequency of measurements, and a configurable payload size margin lsize (fine-tuned using ground-truth sessions to 1 byte) that allows for variations in the payload size of full-size video packets in the corresponding flow.

[0137] To begin, the process initializes four local variables (lines #1 to #4). Within the loop that processes the packet streams, the measured frame rate per interval is reported and reset from line #6 to line #9. Additionally, the process determines the full payload size of video packets in the monitored flow (lines #10 to #13).

[0138] Pseudocode implementation of a process for measuring the video frame rate of cloud gameplay using packet size patterns of downstream video flows:

[0139] Require: packets in gaming video flow; measurement interval AT ; payload size margin l size

[0140] Ensure: measured frame_rate

[0141] 1 : frame_rate <- 0

[0142] 2: tstan <- packets[Q . arrival :imestamp

[0143] 3: sizemax <- packets[Q].payload_size

[0144] 4: flagmax < — FALSE

[0145] 5: for p in packets do

[0146] 6: if p. arrival _timestamp > tstan + AT then

[0147] 7 : report frame_rate / AT

[0148] 8: frame_rate <- 0

[0149] 9: end if

[0150] 10: if p.payload_size > sizemax then

[0151] 11 : sizemax <- p.payload_size

[0152] 12: continue

[0153] 13: end if

[0154] 14: if p.payload_size < sizemax - ! size then 15: if flagmax is TRUE then

[0155] 16: frame_rate <- frame_rate + 1

[0156] 17: flag max FALSE

[0157] 18: end if

[0158] 19: continue

[0159] 20: end if

[0160] 21 : flagmax ^ TRUE

[0161] 22: end for

[0162] From line #14 to line #21, the process captures the stochastic periodical pattern observed in the packet payload sizes of a video flow. This pattern includes sequences of full-sized video packets followed by smaller-sized ones carrying frame markers and / or remaining data.

[0163] Considering that the GeForce NOW platform offers frame rate options of 30fps and 60fps, the observed count of periodical patterns per second is expected to closely align with one of these two values.

[0164] Gaming Graphic Resolution

[0165] The third QoE metric is graphics resolution, which represents the visual quality of the graphics being streamed to the gamer by the cloud gaming platform. The graphics resolution and the video frame rate determine the bandwidth consumption of a downstream video flow, as discussed above. As just described, the current video frame rate is directly determined. Therefore, the real-time graphic resolution is determined by measuring the current bandwidth consumption of the video flow and using the mapping provided in Table 2 accordingly. This mapping determines the current graphics resolution based on the given throughput and frame rate.

[0166] In the described embodiment, the cloud gaming monitoring apparatus can be configured to use network APIs to automatically change network settings in response to the quantified cloud gaming network usage and / or user experience of their network users to improve the performance of their networks for cloud gaming applications, and to improve the QoE of cloud gaming users. For example, the network APIs can be used to amend routing paths, prioritise traffic flows, and / or map traffic to network slices in order to effect improvements in one or more of at least the following network parameters: network bandwidth, network slices, latency, and jitter.

[0167] One or more of these network APIs can be selectively invoked when only certain operator defined conditions are satisfied; for example, a network operator may configure the apparatus to only use the APIs to change network settings when cloud gaming gameplay flows of one or more specified user access modes of one or more specified games are detected by the apparatus, or when a specified QoE metric is degraded below a specified level.

[0168] Although the cloud gaming monitoring apparatus and process have been described above in the context of their application to detect and monitor sessions of the GeForceNow cloud gaming platform, the network traffic attributes described above are equally applicable to detect and monitor sessions of other cloud gaming platforms, the only differences being the specific values of the various parameters described above ( / .e., domain names, port numbers, packet payload sizes, and other network traffic attribute values as described herein). These parameter values can be determined for an arbitrary cloud gaming platform by analysing network traffic attributes of the platform in an isolated (testbed) network, as described above, to identify those attributes that are characteristic of the platform. The cloud gaming monitoring apparatus can then be configured with those characteristic network traffic attributes, and used to identify and monitor sessions of the new platform.

[0169] For example, Figures 25, 26, and 27 are charts showing the time evolution of service flows of representative Xbox Cloud Gaming user sessions accessed from the testbed network via Xbox hardware console, PC browser, and mobile smart phone applications, respectively. As described above for the GeForce NOW platform, there are platform administrative flows once the user launches the platform page, platform management flows selecting appropriate cloud servers for a subsequent gameplay session, and gameplay flows synchronising user inputs and streaming gameplay media via the RTP protocol. Platform sessions accessed via different user access modes have shared and unique service names. Notably, in gameplay sessions of Xbox Cloud Gaming, similar to GeForce NOW accessed by browser, the streaming media and user input are always carried by one single RTP flow, instead of four different flows for GeForce NOW accessed via its console application. The resulting processes of user platform detection and access mode identification for the Xbox Cloud Gaming platform are shown in Figure 28, and gameplay flow classification in Figure 29. These two processes for Xbox Cloud Gaming share the same input attribute format, underlying inference mechanism, and output format, the differences being the criteria or thresholds for each attribute type and output types (e.g., category of user access modes and gameplay flow types).

[0170] Figures 30 and 31 show the generic processes that are applicable to any cloud gaming platforms that share the same networking methodology for cloud gaming services. The generalised models take standard input attributes, make inferences on pre-trained criteria or threshold on the attributes, and generate standard output results from a predefined category. Accordingly, the inventors have developed training processes to automatically generate the relevant parameter values for an arbitrary cloud gaming platform from ground-truth PCAP traces generated for that platform in a testbed network. The resulting parameter values are then used to configure the cloud gaming monitoring apparatus and process to automatically detect and monitor sessions of that platform, as described herein.

[0171] The training processes implement two key inference models, each of which processes the ground-truth PCAP files, as described herein. A session and access mode model processes the ground-truth PCAP files to generate active service names (of flows) that are used to configure the platform session and user access mode detector 2410 so that it can subsequently detect cloud gaming sessions and determine user access modes, using the process shown in Figure 30. The training process extracts the service names of each TLS flow toward pre-defined cloud gaming service domains (e.g., nvidiagrid, geforce, and geforcenow for the GFN platform) from their SNI fields in the client hello handshake messages. For each cloud gaming platform, each PCAP file is associated with its corresponding user access mode. After processing all ground-truth PCAP files of a cloud gaming platform, the training process generates a list of unique service names that are accessed across all user access modes, and lists of unique service names that are only present in PCAPs of respective access modes (e.g., a mobile smart phone console application). Those automatically generated lists are ready to be used by the detector 2410, as shown in Figure 30.

[0172] Similarly, a gameplay session flow model processes ground-truth PCAP files to generate flow metadata and volumetric attributes of gameplay sessions of known cloud gaming platforms, these being used to configure the gameplay session detection and gameplay flow classification module 2412 so that it can subsequently classify candidate flows from their flow metadata and volumetric attributes, using the process illustrated in Figure 31.

[0173] First, the training process extracts patterns in the flow metadata (5-tuple) of gameplay management flows from the ground-truth PCAPs. The gameplay management flows are identified by their SNI fields that are mapped to a pre-defined pattern for the platform; for example, "*-*-*-*. pnt.geforcenow.nvidiagrid.net" for the GeForce NOW platform. Consistently identified values in the metadata are then output as criteria for gameplay management flow classification. For example, for GFN sessions via browser, transportlayer protocol "TCP" and destination port number "49100" are always identified in ground-truth PCAPs, and consequently are output as platform detection criteria.

[0174] Second, the training process generates numerical thresholds for volumetric attributes, including upstream / downstream packet rate and throughput. For each type of gameplay session flow identified by UDP source port number that is mapped to a pre-defined list, their volumetric attributes are computed periodically throughout their active period and saved to respective numerical arrays. After processing all PCAPs, the upper bound and lower bound thresholds for each metric, and the difference between paired upstream / downstream metrics are determined by taking the T% percentile of the respective numerical array. In the described embodiment, the training process uses a 1% margin for the percentile (e.g., 99% for the upper bound threshold, and 1% for the lower bound threshold), rounded to the nearest integer value. The resulting numerical thresholds for each volumetric attribute are ready to be used by the gameplay flow classification module 2412 shown in Figure 31, although the criteria can be simplified by removing redundant attributes and thresholds.

[0175] In the described embodiments, the cloud gaming monitoring apparatus 2300 is a computer system such as an Intel IA-64 based computer system, as shown in Figure 23, and the cloud gaming monitoring processes executed by the cloud gaming monitoring apparatus 2300 are implemented as executable instructions of one or more software modules 2400 stored on non-volatile (e.g., hard disk or solid-state drive) storage 2304 associated with the computer system, as shown in Figure 23. However, it will be apparent that at least parts of the cloud gaming monitoring process 2400 could alternatively be implemented in other forms, such as configuration data of one or more field programmable gate arrays (FPGAs), or as one or more dedicated hardware components, such as application-specific integrated circuits (ASICs), or as any combination of these various forms, for example.

[0176] In the described embodiments, the cloud gaming monitoring apparatus 2300 includes standard computer components, including random access memory (RAM) 2306, at least one processor 2308, and external interfaces 2310, 2312, 2314, all interconnected by a bus 2316. The external interfaces include universal serial bus (USB) interfaces 2310, at least one of which is connected to a keyboard 2318 and a pointing device such as a mouse 2319, a network interface connector (NIC) 2312 which connects the cloud gaming monitoring apparatus 2300 to a communications network such as the Internet 2320, and a display adapter 2314, which is connected to a display device such as an LCD panel display 2322.

[0177] The cloud gaming monitoring apparatus 2300 also includes a number of standard software modules 2326 to 2330, including an operating system 2324 such as Linux or Microsoft Windows, and structured query language (SQL) support 2330 such as MySQL, available from http: / / www.mysql.com, which allows data to be stored in and retrieved from an SQL database 2332.

[0178] In the described embodiments, the software modules of the apparatus 2400 include a packet processing module 2402 written in Golang, built over the open-source Virtual Network Functions Framework NFF-GO 2404, available from https : / / github.com / area m / nff-go. In turn the NFF-GO framework utilises the DPDK (Data Plane Development Kit) 2406, available from https: / / www.dpdk.org / , as its underlying packet processing functions. The packet processing module 2402 maintains states for network flows. The apparatus also includes a cloud gaming flow detector 2408, a platform session and user access mode detector 2410, a gameplay session detection and gameplay flow classification module 2412, a gameplay session user experience analyser 2414, a cloud gaming usage pattern analyser 2416 to store and summarise cloud gaming usage data representing cloud gaming usage for each cloud gaming platform and network user to provide insights into cloud gaming usage for network operators such as ISPs, and network APIs 2418 for changing network setting to improve cloud gaming network performance and cloud gaming user quality-of-experience. The network APIs 2418 are those defined by the CAMARA project, as described at https: / / camaraproject.org. EXAMPLE

[0179] The cloud gaming monitoring apparatus and process described above were applied to a large University campus with tens of thousands of students, including several hundred who reside in the University dormitories. The apparatus received as input a raw feed of all traffic to / from the campus, obtained via optical taps on the fibres connecting the campus to the Internet ( / .e., similar to the mirroring arrangement of the testbed network shown in Figure 1). The accuracy of the apparatus was first assessed by playing cloud games from the inventors' laboratory on campus, and comparing the resulting outputs of the apparatus to the known ground truth. Subsequently, the apparatus was applied to network traffic in the wild over a 1-month period to showcase insights of interest to network operators.

[0180] The insights include both individual gameplay session flow statistics with user experience metrics, and aggregated volumetric network traffic statistics of identified gameplay session flows with their user experience metrics of access modes of network users. With these statistics, network operators can reconfigure their networks automatically to guarantee the service quality of cloud gaming service flows that may otherwise suffer degraded user experience, such as via amending routing paths of gameplay flows, prioritising traffic flows that exhibit degraded user experience, and mapping cloud gaming traffic to high-performant network slices.

[0181] Lab Evaluation with Known Ground-Truth

[0182] Once the models were trained and deployed using the training process described above, student volunteers played a total of 90 sessions of GeForce NOW gaming from the inventors' lab, and the recorded ground truth was compared against the outputs reported from the apparatus monitoring campus-wide traffic for the specific IP address of the lab devices. The sessions encompassed various different configurations, including user access modes (PC app, mobile app, and browser), frame rates (60fps and 30fps), and video resolution bands (FHD, HD, and SD).

[0183] The apparatus correctly reported detection of all the cloud gaming sessions immediately upon commencement of actual gameplay, and the user access mode (PC app, mobile app or browser play) was also identified with 100% accuracy. The client-side latency reported by the apparatus (based on tracking TCP sequence numbers), the frame rates (measured from stochastic payload size patterns as described above), and the video resolution (inferred from the real-time bandwidth usage and corresponding frame rate) were all corroborated accurately (with less than a 2 fps deviation for frame rates, and with over 95% accuracy for video resolution) with the ground-truth collected on the client side, where traffic traces were recorded and analysed.

[0184] An interesting observation is that although the GeForce NOW platform dynamically adjusts the frame rate to either 60fps or 30fps, in practice the received frame rate can vary slightly (e.g., within a range of 5fps) around the set rate. This variation is shown in Figures 14 and 15 for the 60fps and 30fps settings, respectively. Indeed, there can be significant variations in the frame rate, particularly during the beginning of a gameplay session (as shown in Figure 16), or in situations where the network conditions are unstable.

[0185] Field Deployment Insights

[0186] Having confirmed the accuracy of the apparatus outputs, the apparatus was applied in the wild over the entire month of May 2023 in the University campus network, yielding some insights of interest to ISPs, including how user settings in GeForce NOW cloud gaming impact network bandwidth demand and end-user experience.

[0187] User Settings and Bandwidth Demand

[0188] During the month of May, the apparatus detected 2533 cloud game sessions in the wild ( / .e.zexcluding the ones played by the volunteers), corresponding to 362 hours of playtime and 3.8 TB of data volume consumption. The majority of gameplay sessions (53%) and gameplay time (51%) were via the PC app, as shown in Figures 17 and 18, respectively, followed by browsers (29% of sessions and 36% of total playtime), and finally the mobile app (17% of sessions and 14% of playtime). This is of relevance to ISPs because the bandwidth demanded by the cloud gaming streams vary across these access modes, shown in terms of bandwidth distribution (measured over 1-second intervals) in Figure 19. The browser almost never exceeds 30 Mbps, whereas the PC console app and the mobile app have significant durations of bandwidth demand in the range of 30-75 Mbps. While the underlying reason for this becomes clear when video frame rates and resolutions are examined, visibility into the app / browser mix better equips ISPs in planning and provisioning their network capacity as cloud gaming grows. User Settings and Gaming experience

[0189] As described above, frame rate can vary significantly, so for ease of depiction it is grouped into the following three bands: Low (<40 fps), Medium (40-50 fps) and High (>50 fps). Video resolution has already been banded into FHD, HD, and SD, as shown in Table 2. Figures 20 and 21 are bar charts showing the percentage of time that the cloud game video operates at: the three frame rate bands, and the three resolution bands, respectively, for the three user access modes (PC app, mobile app, and browser).

[0190] It is interesting to note that browser gaming almost always has high fps, but never goes to Full-HD resolution, whereas the PC app has FHD resolution 42% of the time but drops fps to medium or low about 10% of the time. The mobile app provides the best mix of fps and resolution overall, due to the advantage that it only has to work on a smaller screen. Visibility into these trade-offs on frame rates and resolution equips ISPs to better troubleshoot and support their cloud gaming customers on their specific platforms.

[0191] Another interesting observation relates to the in-game latency across user platforms. To keep the comparison fair and not be influenced by wireless characteristics, it is limited to wired end-hosts (which are identifiable by their IP address being on a different subnet to the WiFi network). Figure 22 shows that latency (averaged over 1-second intervals) stays under 20ms (the recommended threshold provided by Nvidia) 90% of the time when played via the app, compared to 70% when played via the browser. Presumably, this is because the app is better optimized for gaming than the browser - indeed, it was shown above that the app maintains separate flows for user input and media, whereas the browser mixes all traffic into the one flow, which can lead to degraded jitter performance. ISPs can use such information to encourage users to move from browser-based to app-based cloud gaming.

[0192] As described above, the cloud gaming monitoring apparatus and process provide visibility into cloud gaming sessions of cloud gaming platforms over communications networks. For an arbitrary cloud gaming platform, network traffic attributes characteristic of the platform can be determined by analysing network traffic patterns in a testbed network during gameplay. In particular, benchmarks can be determined for critical service flows that exhibit unique patterns for different user access modes and gameplay experiences. Cloud gaming sessions can then be detected across the different user access modes by stateful matching of service flows, critical gameplay flows classified from volumetric attributes, and experience metrics (including client-platform latency, frame rate, and graphic resolution) dynamically monitored for those flows.

[0193] Using the example of Nvidia's GeForce NOW platform, it has been demonstrated that the cloud gaming monitoring apparatus and process can be configured (using network traffic attributes determined from ground-truth sessions) to accurately identify sessions of cloud gaming platforms from other (general) network traffic in a large university network, determine the access mode for each detected session, and dynamically monitor quality-of-experience (QoE) metrics for each gaming session. The outputs of the apparatus have been shown to provide operational insights from real-world cloud game sessions. Moreover, the apparatus can be configured to automatically reconfigure their networks to improve QoE for cloud gaming users.

[0194] Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention.

Claims

CLAIMS:

1. A computer-implemented cloud gaming monitoring process, including: receiving mirrored data packets of general network traffic of a communications network, the data packets including data packets of one or more cloud gaming platforms, and other data packets unrelated to cloud gaming platforms; processing the received data packets to detect a cloud gaming session of a cloud gaming platform and a network client by identifying a subset of the received data packets as constituting network flows of a platform session of the cloud gaming platform and the network client; processing network flows of the network client to detect a gameplay session of the cloud gaming platform by identifying at least one of the network flows as being a gameplay management flow of the cloud gaming platform; determining network client and gaming server IP addresses of the gameplay management flow; using the network client and gaming server IP addresses to identify subsequent network flows as being gameplay session flows of the gameplay session; processing the identified gameplay session flows to determine metadata and generate volumetric metrics of the gameplay session flows; processing the metadata and volumetric metrics of the gameplay session flows to identify one of the gameplay session flows as being a downstream video flow or a gameplay management flow; and processing at least one of the downstream video flow and the gameplay management flow of the gaming session to generate one or more quality-of-experience (QoE) metrics for the game and network user.

2. The cloud gaming monitoring process of claim 1, wherein the processing of the received data packets to detect a gameplay session includes: generating volumetric metrics of network flows of the received data packets, and comparing the generated volumetric metrics to known volumetric metrics of platform management flows of the cloud gaming platform.

3. The cloud gaming monitoring process of claim 2, wherein the volumetric metrics of network flows of the received data packets include packet sizes of network flows in upstream and downstream directions for known port numbers of the cloud gaming platform.

4. The cloud gaming monitoring process of any one of claims 1 to 3, wherein the processing of the received data packets to detect a gameplay session includes: identifying requests from the network client to known service domains of the cloud gaming platform to generate a stateful list of active services of the cloud gaming platform requested by the network client; and comparing the stateful list to a corresponding list of known services of the cloud gaming platform; wherein a platform session of the cloud gaming platform is detected only if all of the known services of the cloud gaming platform are in the stateful list of active services of the cloud gaming platform.

5. The cloud gaming monitoring process of claim 4, further including classifying an access mode of the network user and the cloud gaming platform as one of: a web browser, a smartphone console application, a personal computer console application, a smartphone browser, a personal computer browser, and a gaming hardware console by comparing, for each of the access modes, a corresponding list of known services for the access mode and the cloud gaming platform to active services in the stateful list, wherein the access mode of the network user and the cloud gaming platform is classified as the access mode whose list of known services best matches active services in the stateful list.

6. The cloud gaming monitoring process of any one of claims 1 to 3, wherein the processing of the received data packets to detect a gameplay session includes identifying network flows as being gameplay management flows of the cloud gaming platform if a port number and a sequence of packet payload sizes of the network flows in upstream and downstream directions match a corresponding known port number andsequence of packet payload sizes of gameplay management flows of the cloud gaming platform.

7. The cloud gaming monitoring process of any one of claims 1 to 6, wherein the volumetric metrics include upstream packet rate, downstream packet rate, and throughput, and the metadata includes a 5-tuple representing source IP address and port number, destination IP address and port number, and protocol.

8. The cloud gaming monitoring process of any one of claims 1 to 7, wherein processing the metadata and volumetric metrics of the gameplay session flows to classify each of the gameplay session flows includes classifying each of the gameplay session flows as being one of: a gameplay management flow, an upstream user input flow, an upstream audio flow, a downstream audio flow, a downstream video flow, and a streaming media and user input flow.

9. The cloud gaming monitoring process of any one of claims 1 to 8, wherein processing the at least one downstream video flow of the gaming session to generate quality-of-experience (QoE) metrics for the game and network user includes generating a client-platform latency metric from timestamps of TCP packet pairs with matching sequence and acknowledgement numbers of the gameplay management flow.

10. The cloud gaming monitoring process of any one of claims 1 to 9, wherein processing the at least one downstream video flow of the gaming session to generate quality-of-experience (QoE) metrics for the game and network user includes generating a gaming video frame rate metric from periodic patterns of packet payload sizes of the at least one downstream video flow or streaming media and user input flow.

11. The cloud gaming monitoring process of any one of claims 1 to 10, wherein processing the at least one downstream video flow of the gaming session to generate quality-of-experience (QoE) metrics for the game and network user includes generating a gaming video graphics resolution metric from bandwidth consumption of the at least one downstream video flow or streaming media and user input flow.

12. The cloud gaming monitoring process of any one of claims 1 to 11, including periodically repeating the step of processing the downstream video flow of the gaming session to provide quasi-continuous monitoring of the one or more quality-of- experience (QoE) metrics for the game and network user over time.

13. The cloud gaming monitoring process of any one of claims 1 to 12, wherein the step of processing at least one of the downstream video flow and the gameplay management flow of the gaming session includes: processing the downstream video flow to generate video frame rate and graphics resolution QoE metrics; and processing the gameplay management flow to generate a latency QoE metric for the game and network user.

14. The cloud gaming monitoring process of claim 13, further including aggregating volumetric network traffic statistics of the identified gameplay session flows of all access modes of the network user.

15. The cloud gaming monitoring process of any one of claims 1 to 14, further including:(i) monitoring QoE metrics, including client-platform latency, gaming graphic resolution, and gaming video frame rate of detected user gameplay sessions for each network user to determine whether any of the monitored QoE metrics is degraded; and(ii) responsive to determining that one or more of the monitored QoE metrics are degraded, reconfiguring network settings to improve the level of performance for the network user.

16. The cloud gaming monitoring process of claim 15, wherein the step of automatically reconfiguring network settings includes using one or more network application programming interfaces (APIs) to change one or more network settings toeffect one or more of the following network reconfigurations: amending routing paths, prioritising traffic flows, and mapping traffic to network slices.

17. A computer-readable storage medium having stored thereon processorexecutable instructions that, when executed by at least one processor of a cloud gaming monitoring apparatus, cause the cloud gaming monitoring apparatus to execute the cloud gaming monitoring process of any one of claims 1 to 16.

18. A cloud gaming monitoring apparatus of a communications network, including: a network interface to receive mirrored data packets of general network traffic of a communications network, the data packets including data packets of one or more cloud gaming platforms, and other data packets unrelated to cloud gaming platforms; a platform session detector to process the received data packets to detect a cloud gaming session of a cloud gaming platform and a network client by identifying a subset of the received data packets as constituting network flows of a platform session of the cloud gaming platform and the network client; a gameplay session detector to process network flows of the network client to detect a gameplay session of the cloud gaming platform by identifying at least one of the network flows as being a gameplay management flow of the cloud gaming platform, and to determine network client and gaming server IP addresses from the gameplay management flow; a gameplay session flow detector to identify subsequent network flows as being gameplay session flows of the gameplay session based on the network client and gaming server IP addresses; a gaming video flow detector to process the identified gameplay session flows to determine corresponding metadata and generate volumetric metrics of the gameplay session flows, and to process the metadata and volumetric metrics to identify one of the gameplay session flows as being a downstream video flow or as being a gameplay management flow; anda QoE monitor to process at least one of the downstream video flow and the gameplay management flow of the gaming session to generate one or more quality- of-experience (QoE) metrics for the game and network user.

19. The cloud gaming monitoring apparatus of claim 18, including a network reconfiguration component configured to use one or more network application programming interfaces (APIs) to change one or more network settings to effect one or more of the following network reconfigurations: amending routing paths, prioritising traffic flows, and mapping traffic to network slices.

20. The cloud gaming monitoring apparatus of claim 18 or 19, wherein the platform session detector is configured to: generate volumetric metrics of network flows of the received data packets; and compare the generated volumetric metrics to known volumetric metrics of platform management flows of the cloud gaming platform; wherein the subset of the received data packets is identified as constituting network flows of a platform session of the cloud gaming platform and the network client if the generated volumetric metrics match the known volumetric metrics of platform management flows of the cloud gaming platform.

Images