Methods, apparatus, and systems for improving perceived quality in cloud gaming.

Abstract

Methods, apparatus, systems, etc. relating to QoE improvement in cloud gaming are disclosed herein. In one embodiment, a method for improving QoE of video content (e.g., a game, etc.) may be implemented in a client device. For example, the client device may receive a number of packets carrying video frames of the video content from a game server. For example, the client device may apply an initial (e.g., Secure Reliable Transport (SRT)) latency value to the received packets before decoding and displaying. For example, the client device may send a request message to a server indicating a new latency value based on frame pace variation to be applied to subsequent received packets before decoding and displaying.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the benefits of European Patent Application No. 21305349.9, filed on 22 March 2021, which is incorporated herein by reference in its entirety.

[0002] (Field of Invention) This disclosure relates to the cloud gaming domain, also known as game streaming, and more specifically to the quality of experience (QoE) in cloud gaming. [Background technology]

[0003] In cloud gaming, end-user devices do not need to run game executable programs that may run on a server (e.g., an instance). Server instances may be located within a data center (operated, for example, by a cloud provider or any type of operator). In cloud gaming, the user experience of a game may vary depending on different factors such as network latency, server load, and game complexity. This disclosure is designed with the above in mind. [Overview of the Initiative]

[0004] Methods, apparatus, and systems for improving the QoE in cloud gaming are disclosed herein. In one embodiment, a method for improving the QoE of video content (e.g., a game) may be implemented in a client device. For example, the client device may receive a number of packets from a server that carry video frames of the video content. For example, the client device may apply an initial (e.g., secure reliable transport (SRT)) latency value to the received packets before decoding and displaying them. For example, the client device may send a request message to the server. For example, the request message may include first information indicating a new latency value based on frame pace variation. For example, the client device may receive a response message from the server, which includes second information indicating a new latency value. For example, the client device may apply a new SRT latency value to the received subsequent packets before decoding and displaying them (e.g., subsequent packets).

[0005] This specification describes and / or claims various embodiments of apparatus, systems, devices, etc., and / or any elements thereof, configured to perform operations, processes, algorithms, functions, etc., and / or any part thereof. However, it should be understood that any embodiment described and / or claimed herein assumes that any apparatus, system, device, etc., and / or any elements thereof perform any operation, process, algorithm, function, etc., and / or any part thereof (and vice versa). [Brief explanation of the drawing]

[0006] A more detailed understanding can be obtained from the following detailed description, given by way of example in conjunction with the accompanying drawings. The diagrams of such drawings, like the detailed description, are examples. Therefore, the drawings and the detailed description should not be regarded as limiting, and other equally effective examples are possible and likely. Also, like reference numbers in the drawings indicate like elements. [Figure 1] It is a system diagram showing an example of a high-level architecture of game streaming. [Figure 2] It is a system diagram showing an example of video streaming in a cloud gaming architecture. [Figure 3] It is a diagram showing an example of latency window operation in an SRT latency buffer. [Figure 4-01] It is a diagram showing an example of an acknowledgment operation in SRT. [Figure 4-02] It is a diagram showing an example of an acknowledgment operation in SRT. [Figure 5] It is a diagram showing an example of buffer latency of a receiver and a transmitter after an SRT extended handshake procedure. [Figure 6A] It is a diagram showing an example of an inter-frame delay variation metric. [Figure 6B] It is a diagram showing an example of an inter-frame delay variation metric. [Figure 7] It is a system diagram showing an example of an SRT-based cloud gaming system. [Figure 8A] It is a diagram showing two examples of an extended handshake packet format. [Figure 8B] It is a diagram showing an example of a handshake extension message flag. [Figure 9A] It is a diagram showing an example of extended handshake message exchange to demonstrate the ability to support latency dynamic change. [Figure 9B] It is a diagram showing an example of extended handshake message exchange to demonstrate the ability to support latency dynamic change. [Figure 10]This figure shows an example of the format for an SRT message requesting a new SRT latency value. [Figure 11A-01] This figure shows examples of message exchanges for dynamic latency change procedures initiated by the client device and the server, respectively. [Figure 11A-02] This figure shows examples of message exchanges for dynamic latency change procedures initiated by the client device and the server, respectively. [Figure 11B-01] This figure shows examples of message exchanges for dynamic latency change procedures initiated by the client device and the server, respectively. [Figure 11B-02] This figure shows examples of message exchanges for dynamic latency change procedures initiated by the client device and the server, respectively. [Figure 12A] This figure shows an example of a client processing device 12A for improving the QoE of a game. [Figure 12C] This figure shows an example of a server processing device 12C for improving the QoE of a game. [Figure 12B] Figure 12A shows an example of the architecture of either the client processing device or the server processing device shown in Figure 12C. [Figure 13] This diagram shows examples of methods for improving the quality of execution (QoE) of a game. [Figure 14] For example, this figure shows a first example of a method implemented on a client device to improve the QoE of video content. [Figure 15] For example, this figure shows a second example of a method implemented on a server to improve the QoE of video content. [Figure 16] For example, this figure shows a third example of a method implemented on a client device to improve the QoE of video content. [Figure 17] For example, this figure shows a fourth example of a method implemented on a server to improve the QoE of video content.

[0007] Please understand that the (multiple) drawings are intended to illustrate the concepts of this disclosure and are not necessarily the only possible configuration to illustrate this disclosure. [Modes for carrying out the invention]

[0008] It should be understood that the elements shown in the figure can be implemented in various forms of hardware, software, or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices and may include a processor, memory, and input / output interfaces. In this specification, the term “interconnected” means directly connected to one or more intermediate components, or indirectly connected through one or more intermediate components. Such intermediate components may include both hardware-based and software-based components. The term “interconnected” is not limited to wired interconnections but also includes wireless interconnections.

[0009] All examples and conditional language listed herein are intended for educational purposes to help readers understand the principles of this disclosure and the concepts to which the inventors have contributed to advance the art, and should be construed as not being limited to such specifically listed examples and conditions.

[0010] Furthermore, it will be understood by those skilled in the art that, for example, the block diagrams presented herein represent conceptual diagrams of exemplary circuits embodying the principles of this disclosure. Similarly, it will be understood that any flowcharts, flow diagrams, state transition diagrams, pseudocode, etc., substantially represented in a computer-readable medium, represent various processes that can be performed by such computer or processor, whether or not such computer or processor is explicitly indicated.

[0011] The functions of the various elements shown in the diagram may be provided through the use of dedicated hardware, as well as hardware capable of running software in conjunction with appropriate software. Where provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Furthermore, the explicit use of the terms “processor” or “controller” should not be interpreted as exclusively referring to hardware capable of running software, but implicitly including, but not limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), and non-volatile storage devices for storing software.

[0012] Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the diagram are merely conceptual. Their functions may be performed through program logic operation, through dedicated logic operation, through program control and interaction with dedicated logic, or even manually, and specific techniques are selectable by the implementer, as they are better understood in their context.

[0013] In the claims of this specification, any element represented as a means for performing a particular function is intended to encompass any means of performing that function, including, for example, a) a combination of circuit elements for performing that function, or b) any form of software, including firmware, microcode, etc., combined with appropriate circuitry for running the software to perform that function. The disclosure as defined by such claims lies in the fact that the functionalities provided by various enumerated means are combined and assembled in the manner required by the claims. Accordingly, any means capable of providing those functionalities are deemed equivalent to those shown herein.

[0014] For example, in the cases of "A / B", "A and / or B", and "at least one of A and B", it should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to encompass the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). In further embodiments, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to encompass the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to the number of listed items, as will be apparent to those skilled in the art in this and related fields.

[0015] The embodiments described herein relate to cloud gaming, which may also be called game streaming. Cloud gaming can be considered a concept in which a game is run on a (for example, remote) server (hereinafter referred to as either a “game server” or a “server,” collectively as a “server”) and the results are sent as a video stream to an end-user device, which may be referred to herein as a “cloud gaming client,” a “thin client device,” a “client game device,” collectively as a “client device.” The game server and client devices may be interconnected via any type of network. For example, the server may be located in the cloud, and the game may be run on any instance (for example, of any type) of the server.

[0016] For simplicity, embodiments are described herein by reference to games (e.g., streaming) as an example of video content (e.g., streaming). Embodiments described herein are not limited to game streaming and may be applicable to any type of video content. Throughout the embodiments described herein, the terms “game” and “video content” may be used interchangeably.

[0017] For simplicity, embodiments described herein are described by reference to the Secure Reliable Transport (SRT) protocol and its associated SRT latency. The SRT protocol (e.g., and SRT latency) is merely an example of a protocol (e.g., and latency) that may be applicable to the embodiments described herein. Any transport protocol and any type of latency (e.g., management system) that can reproduce in a client device a rate (e.g., time sequence) of packets (e.g., data, video frames) that may be similar to (e.g., can represent) the rate (e.g., time sequence) of packets (e.g., data, video frames) that may be generated in a server may be applicable to the embodiments described herein. The terms “SRT (e.g., request, response, handshake) message” and “(e.g., request, response, handshake) message” may be used interchangeably throughout the embodiments described herein. The terms “SRT latency” and “latency” may be used interchangeably herein.

[0018] Figure 1 is a system diagram illustrating an example of a high-level architecture for game streaming. The client device 11 does not have to run a game executable file that can be run on the server 12 (e.g., a dedicated instance thereof). The server (e.g., an instance) may be located in a data center (e.g., operated by either a cloud provider or an operator). The server 12 may include (e.g., high-performance) hardware such as a graphics processing unit (GPU). For example, the server 12 may acquire (e.g., capture) an image 122 (e.g., of a game scene) rendered by the GPU at (e.g., a given capture) rate (e.g., 60 frames per second (FPS)). The acquired (e.g., captured) image may result in a flow of video frames (121) that can be encoded into a (e.g., live) video stream that can be sent (e.g., transmitted) to the client device 11. The video stream may include multiple packets carrying (e.g., encoded) video frames of the game. On client device 11, the thin game client application may receive (e.g., listen to) this video stream, decode frames when frames are received, and present (e.g., display) them on the screen of client device 11.

[0019] The client device 11 may be lightweight (may involve limited processing resources) by performing some of these operations (e.g., only those). The term "video stream" is used to specify a set of game video (e.g., encoded) frames that can be transmitted to the client device by the game server as multiple packets, each containing either transport layer overhead and (e.g., corresponding) metadata.

[0020] In cloud gaming, perceived quality of experience (QoE) can depend on either objective or subjective factors. For example, QoE can depend on the response to user commands (e.g., response time, latency) and the presence or absence of video stuttering, which can be called video smoothness.

[0021] Video smoothness can be a significant subjective factor affecting QoE, and it is considered to be as important as response time or visual quality, depending on whether the user can tolerate or adapt to it. Video stuttering (e.g., lack of smoothness) can occur, for example, if any number of frames are dropped or duplicated on the client device (which can be visible on the client screen). Frame duplication (e.g., rendering the same frame twice) can occur, for example, if the next frame to be displayed cannot be received, decoded, and displayed in time (e.g., due to increased latency). Similarly, frame dropping (e.g., skipping, not decoding, not displaying a frame) can occur, for example, if the client device's receive buffer is full (e.g., due to the arrival of burst data), and some frames may be dropped. Frames that are dropped (e.g., due to a full client buffer) may no longer be received, for example, if server retransmission of that frame is no longer possible.

[0022] In cloud gaming, the rate regularity (e.g., invariance) of the rate at which game video frames can be received by a client device (e.g., delivered to the client device) can be negatively affected by any number of (e.g., changing) conditions (e.g., factors), such as (e.g., changing) network constraints, encoding variability, the availability of server resources (e.g., disk I / O access, CPU, GPU, and encoder function sharing) linked to the number of game executables that can run on a server instance, and the complexity of the scene at a given time. For example, some video stuttering may occur on the client device screen, for example, due to frame overflows (each underflow) (in the client device's memory buffer), which may cause the client device to drop (each duplicate) some of them and change (e.g., degrade) the pace (e.g., rate regularity) of the rendered video.

[0023] In cloud gaming, client devices may include a memory buffer to buffer video upon reception for, for example, several hundred milliseconds, before the video can be rendered on the screen. The video stuttering problem can be mitigated (e.g., eliminated) by increasing the size (e.g., time) of the memory buffer, which can increase the overall cloud gaming latency, a factor that can be considered important for QoE. There may be a compromise (e.g., a balance) that should be estimated to maintain both a low (e.g., minimum) value of cloud gaming latency and a low level (e.g., none) of video stuttering.

[0024] Since (e.g., live, real) conditions can continuously evolve during a game session, there may be high values ​​of either sporadic or long-term fluctuations in jitter on the client device. Embodiments described herein may allow for the adjustment (e.g., periodically, repeatedly) of a balance (e.g., between increasing latency and reducing video stuttering) that enables the server and client devices to remain (e.g., properly) synchronized.

[0025] Examples of cloud gaming latency In cloud gaming, high responsiveness can enable streamed content to adapt to user actions (e.g., quickly). The term "cloud gaming latency" may be used to refer to the delay between a user's actions in a game and their effect on the display. Cloud gaming latency may include: - Round-trip time delay (RTT), which can be seen as the network transport delay between the first time a packet can be sent by a client device and the second time a response packet can be received from the server. RTT can change over time depending on, for example, the current network conditions, congestion issues, etc. - Processing delay, which can be seen as the time it takes for the server to encode the frame. This time may depend on the server's resource management. - Playout delay, which can be considered the time it takes for the client device to decode the frame and present it on the screen (e.g., display it). - Queuing delay, which can be seen as the time spent in an internal buffer within the receiving protocol stack before a received packet can be delivered to the application for playout. Buffering dimensions may be configuration, and these configurations may be either pre-configured at startup or selected (e.g., determined). For example, buffering dimensions may be negotiated between the server and the client device.

[0026] Cloud gaming latency can be kept low to enable a fun user experience in gaming. For example, cloud gaming latency, which can remain at 100-150 milliseconds, can make it possible to provide a fun user experience for games such as first-person shooters.

[0027] Examples of video streaming in cloud gaming Figure 2 is a system diagram illustrating an example of video streaming in a cloud gaming architecture. On server 21, a video capture and encoding processing module 211 may be configured to capture (multiple) frames rendered by the game at a target rate (e.g., 60Hz). Newly encoded frames may be encoded by the processing module 211 and output at the same target rate. Each encoded frame may be transmitted to the client device 22 using a trusted streaming protocol that is higher than either the user datagram protocol (UDP) or the transport control protocol (TCP). For example, examples of trusted streaming protocols may include, for example, the web real-time communication protocol (WEBRTC) for Stadia and, for example, the real-time protocol (RTP) over UDP for GeForce Now.

[0028] For example, in a live video system, transmission can be affected by congestion, packet loss, reordering, and jitter. Both packet delay and packet loss can result in video stuttering, video freezing, and visual glitches. The protocol stack (e.g., the network distribution system) can compensate for these effects (e.g., attenuation) and improve QoE by one of the following techniques: - The packet buffer 222 can be used, for example, to absorb network fluctuations (e.g., time distribution). - Information redundancy, such as forward error correction (FEC), may be used to improve transmission reliability. - For example, a remediation mechanism can be used by requesting the server to retransmit packets that may have been detected as corrupted or missing by the receiver (such as the Secure Reliable Transport Protocol (SRT)).

[0029] Some video streaming techniques may use significant buffering, measured in seconds, which can greatly increase queuing delays and global cloud latency. These techniques may not be applicable to cloud gaming and its corresponding (e.g., very low) latency (<150ms) expectations, where video buffering can be reduced.

[0030] According to the embodiment, the client device 22 can receive packets (e.g., carrying video frames) (e.g., through the network protocol stack), and the packets are queues that can be buffered (e.g., stored in the receiver buffer 221) before they can be delivered to the application for decoding 222 and presentation (e.g., display) 223. For example, packets can be stored in the buffer 221 for a (e.g., maximum) time limit. For example, the decoding processing module 222 may leverage (e.g., any) hardware acceleration available on the client device to decode (multiple) frames. For example, GeForce Now and Stadia can use DirectX video acceleration (DXVA2). The decoded video frames are, for example, periodic video synchronization signals (V() at 60Hz)))))))))) that can be delivered to the application for decoding 222 and presentation 223))) that can be delivered to the application for decoding 222 and presentation 223)) that can be delivered to the application for decoding 223) that can be delivered to the application for decoding 222 and presentation 223)) that can be delivered to the application for decoding 223) that can be delivered to the application for decoding 222 and presentation 223)) that can be delivered to the application for decoding 222 and presentation 223)) that can be delivered to the application for decoding 222 and presentation 223)) Sync It may be synchronized with (which may be called) and presented on the screen (for example, displayed on the screen).

[0031] For example, due to some reason (such as packet jitter, packet errors, and repairs), video frames may be decoded (for example, quite) slowly. Sync The presentation module 223 may become available some time after the deadline (e.g., a few milliseconds). In such cases, the previous video frame will appear on the screen as two V Sync The last video frame that has been presented (e.g., displayed) over a period of time and has been decoded is the next V SyncThe delay can be extended to a certain period (e.g., 16.66 ms). If this process is repeated for the occurrence of (e.g., some, all) missing (e.g., slow) frames, the delay between the video frames presented on the screen (e.g., displayed) and the video frames generated (e.g., encoded) by the server can accumulate and shift over time. For example, (e.g., at some point in time), a client device may resynchronize with the video source (e.g., encoding rate) by skipping any number of (e.g., fairly) slow decoded video frames and present (e.g., display) the last received frame. For example, the sequence of displayed video frames may not be exactly the same as the sequence of captured video frames because they may contain either repeated or skipped frames, which can cause video stuttering.

[0032] For example, V (which may cause tearing risk) Sync Disabling this feature may not prevent video stuttering. For example, if the delay between consecutive decoded frames is not constant, video frames may not be displayed for the same amount of time, which can cause some video stuttering.

[0033] Example of the SRT protocol The Secure Reliable Transport Protocol (SRT) is an example of a streaming protocol that can be used for cloud gaming.

[0034] SRT was initially developed internally by the video streaming company Haivision to address video streaming applications. SRT is supported by the SRT Alliance, established by Haivision and Wowza. SRT may also be available as the Internet Engineering Task Force (IETF) draft "The SRT protocol draft-sharabayko-mops-srt-01," published on September 9, 2020.

[0035] Based on the UDP-based Data Transfer (UDT) protocol, an SRT packet may include an SRT header that may contain information such as a packet number, message number, timestamp, or destination socket identifier. SRT packets can be of different types, such as control type (for control packets) and data type (for data packets). Control packets may be used for purposes such as session management or acknowledgment. Data packets may be used to deliver stream data (e.g., packets carrying video / audio frames).

[0036] According to the embodiment, the SRT may enable timely delivery of packets to applications. For example, the SRT may enable sending packets to applications at a timing corresponding to when the packets were captured by the SRT.

[0037] According to the embodiment, SRT may enable the avoidance of retransmitting slow packets. For example, SRT can identify packets that may be too slow to be delivered to an application without problems and in a timely manner, and stop retransmitting them.

[0038] According to the embodiment, SRT may enable the estimation of RTT (e.g., periodically, repeatedly) using ACK and ACKACK mechanisms.

[0039] Example of SRT buffer latency Figure 3 shows an example of latency window operation in an SRT latency buffer. For example, a (e.g., game) server may include a transmitter Tx buffer 31 and a transmitter Rx buffer 33. For example, a client device may include a receiver Rx buffer 32 and a receiver Tx buffer 34. The sizes of the server device buffer 31 and the client device buffer 32 may correspond to windows 310 and 320 within buffers 31 and 32, which may be referred to herein as latency windows. For example, latency windows 310 and 320 of the same length for the server transmitter Tx buffer 31 and the client device receiver Rx buffer 32 may be negotiated between the server and the client device during the SRT extended handshake, for example, during session establishment, as described herein.

[0040] On the server side, a (e.g., circular) latency buffer 31 may be used to store SRT packets obtained from the fragmentation of encoded video frames acquired from the video encoder. For example, SRT packets may be held in the (e.g., circular) latency buffer 31 for retransmission for a time period associated with an SRT latency value. These packets may be time-stamped relative to the creation time of the SRT session (e.g., the time at which the SRT session would have been established). For example, the time value may be initialized at the time of SRT session creation and may be incremented based on a (e.g., system) clock. The (e.g., system) clock may be a steady-state clock based on a measure of the number of CPU cycles elapsed since the start of the SRT session. For example, each packet may contain time-stamp information indicating the time associated with the storage of the packet in the server transmitter Tx buffer 31. For example, timestamp information may indicate the packet transmission time (e.g., when the packet was transmitted to the client device (e.g., for the first time)). In another example, timestamp information may indicate the packet origin time (e.g., when the packet was created and stored (e.g., inserted) in the sender Tx buffer 31 (e.g., before, after, and (e.g., the first) transmission time)). Timestamp information corresponding to the time value when different packets were acquired from the encoder (e.g., stored in the server transmitter Tx buffer 31) may be inserted into the packet. Different packets (e.g., having different times indicated by the timestamp information) may remain in the buffer for a (e.g., limited) amount of time. For example, a packet may remain in the buffer until an acknowledgment (ACK) can be received for that packet. For example, a packet may remain in the buffer until the end of the sender latency window (SLW) (e.g., the corresponding time). If the reception of a packet has not been acknowledged by the client device (e.g., within a timeout) and there is still time remaining in the SLW indicated by the packet's timestamp information, the packet may be retransmitted by the transmitter.Otherwise (for example, if the packet's timestamp information indicates that the packet is buffered beyond the SLW), the packet may be dropped, and the server may stop retransmitting the packet. Stopping the retransmission of slow packets can help avoid unnecessary transmission, as packets that are no longer within the client-side receiver latency window (RLW) may not be decoded in time to be displayed.

[0041] On the client device side, the receiver latency buffer 32 may be used to store (e.g., hold) received packets until time can be allocated to deliver them to the application (for decoding and rendering). For example, a packet may be extracted from the receiver latency buffer 32 for decoding based on a comparison between the time indicated by the timestamp information contained in the packet and (e.g., local, system) time. For example, a packet may be extracted for decoding after a (e.g., constant) amount of time that may have elapsed between the time the packet may have been inserted into the server's transmitter latency buffer 31 (e.g., transmitted at a first time) and the time the packet may be extracted from the client device's receiver latency buffer 32 after successful reception by the client device. The (e.g., constant) amount of time may be constant for at least a consecutive (e.g., successive) set of packets. The (e.g., constant) amount of time may be associated with an SRT latency value. For example, a certain amount of time may be equivalent to at least the sum of the SRT latency value and an initial value representing RTT (e.g., half of the initial RTT, which may be measured during the SRT handshake exchange). Packet timestamps may allow the receiver to reflect the pace (e.g., timing) at which the packet may have been generated on the transmitter side (e.g., by an encoder). Reflecting the transmitter pace on the client device may allow the elimination of jitter about the application. This may allow the application to have small (e.g., limited) buffering. For example, an SRT receiver may sort packets based on packet numbering. For example, an SRT receiver may detect missing packets based on holes (e.g., discontinuities) in the sequence of packet numbers. For example, any number of missing packets may be reported by transmitting non-acknowledgment packets (which may be referred to herein as NAKs) to the server, which may trigger the retransmission of the missing packets.After the (e.g., sorted) packets have been delivered to the application, the client device may send an acknowledgment packet (which may be referred to herein as an ACK). For example, both ACKs and NAKs may be sent at regular and configurable intervals (e.g., 10 ms).

[0042] Figure 4 shows an example of acknowledgment operation in SRT. An ACK message 41 may be transmitted by the SRT receiver to trigger the transmission of an ACKACK message 42 by the SRT transmitter, enabling the SRT receiver to calculate the RTT (and any RTT variance). For example, if the number of transmitted packets is large within the ACK period 40, a write ACK message 43 may be transmitted to acknowledge the packets without triggering any ACKACK transmissions or RTT measurements.

[0043] According to the embodiment, if a packet is not received in time to deliver it to an application (for example, for decoding and display), the server does not have to make any further attempts to (re)transmit the packet. For example, a packet may be considered not received in time if it is received after its expected extraction time (for example, a certain amount of time after the time indicated in the timestamp information, where the certain amount of time is associated with the SRT latency value). If a packet is not received in time, a client device (e.g., a receiver stack) may advance its receiver latency window in the same way as if the packet had been received. Similarly, the server may advance its sender latency window as if the packet had been transmitted without issue.

[0044] According to the embodiment, the latency window (for example, of a receiver or transmitter) may enable the provision of (for example, predictable) finite latency. For example, reduced latency may be achieved at the risk of losing more packets, depending on network conditions.

[0045] Example of SRT Extended Handshake The SRT may include an extended SRT handshake process (which may be referred to herein as HSv5). The extended SRT handshake process may be included in a second part of the SRT caller-listener handshake. The extended SRT handshake may enable the determination of at least an SRT latency value in the transmit direction (which may be referred to herein as either transmitter SRT latency or SndTsbPdDelay) and an SRT latency value in the receive direction (which may be referred to herein as either receiver SRT latency or RcvTsbPdDelay). Both the transmitter and receiver SRT latency information (e.g., fields) may refer to latency. Since communication can be bidirectional, there may be one latency value per direction in the extended SRT handshake message. For example, both an extended handshake request (HSREQ) message and an extended handshake response (HSRSP) message may include transmitter and receiver SRT latency information (e.g., SndTsbPdDelay and RcvTsbPdDelay).

[0046] For example, transmitter SRT latency may correspond to the (e.g., minimum) latency that the SRT transmitter can expect the SRT receiver to use. For example, receiver SRT latency may correspond to the (e.g., minimum) latency value that the receiver can apply to the received stream.

[0047] For example, a client device may send an Enhanced Handshake Request (HSREQ) message to the server. The Enhanced HSREQ message may include receiver SRT latency information indicating (e.g., the requested) receiver SRT latency value. For example, the server may send an Enhanced Handshake Response (HSRSP) message to the client device. The Enhanced HSRSP message may include transmitter SRT latency information indicating (e.g., the responded) transmitter SRT latency value. According to the embodiment, the server and the client device may each set their respective SRT latency values ​​in the transmit buffer and receive buffer to the same and highest value among two values ​​(e.g., the requested receiver latency value and the responded transmitter latency value).

[0048] Figure 5 shows an example of receiver and transmitter buffer latency after an extended SRT handshake procedure. The extended SRT handshake may allow for the determination of SRT latency values ​​for the transmitter and receiver in both directions (transmit and receive). For example, a cola may send an extended HSREQ message containing first information indicating the SRT latency values ​​on its side (e.g., Rx, Tx). A listener may calculate the highest value between the SRT latency values ​​contained in the received extended HSREQ and its own. The listener may send an extended HSRSP message containing second information indicating the SRT latency values ​​to be used (e.g., on both sides). In other words, SRT (e.g., buffer) latency may be constructed through the exchange of latency information during the extended handshake process between the initiator and responder (which may be either a server or a client device). The handshake extended message may also include timestamp-based packet delivery (TSBPD) delay information indicating SRT latency values ​​in milliseconds from both the SRT receiver and the SRT transmitter. The SRT latency for a connection can be established (for example, set to the highest value) of the SRT latency transmitted by the initiator and responder. For example, the SRT latency value may be set relative to the duration of the connection.

[0049] Example of Frame - to - Frame Delay Variation According to an embodiment, in the context of cloud gaming, video stuttering may occur in a client device, resulting in an unpleasant user experience for the user. For example, a software probe can be used to track calls to the decoder function of a client device (such as the DXVA2 API, etc.) in order to monitor the delivery time of video frames (e.g., packets carrying video frames) in the output of the protocol stack. For example, the number of encoded frames received per second may be equal to the number of frames generated per second on the server. For example, the average number of frames per second (FPS) at reception may be the same as the FPS on the server, but some variance may be measured, for example, with respect to the frame - to - frame delay.

[0050] According to an embodiment, a measure (e.g., metric) of frame - pace variation (such as variation in frame - to - frame delay, etc.) is referred to as jitter in this specification. ローカル It may be called jitter. Jitter ローカル can be obtained by dividing the jitter metric by the average period. For example, jitter local can be given as follows:

[0051]

Equation

[0052] The jitter metric may be defined as the sum of the differences between consecutive interval durations, where the interval duration may represent the duration between a frame time and the previous frame time. For example, the jitter metric can be given as follows:

[0053]

Equation

[0054] For example, T iThis can represent the duration of the i-th interval between the first time associated with the i-th frame and the second time associated with the preceding (i-1)-th frame. For example, the duration can be included in the jittermetric, provided that it falls within the range of 1 to 32 milliseconds (to exclude outliers, for instance).

[0055] Figures 6A and 6B show the inter-frame delay variation metric (jitter) measured on the client device. ローカル Figure 6A shows two examples of the inter-frame delay variation metric (jitter) obtained over a 13-minute game session of a specific game (a first-person shooter, e.g., Destiny 2) using a fiber optic network (e.g., 200 Mbps). ローカル Figure 6B shows the inter-frame delay variation metric (jitter) obtained for the same game over a 50-second period 61. ローカル ) indicates.

[0056] Examples of large inter-frame delay variability For example, a service provider may operate a cloud gaming solution with high cloud gaming latency (e.g., around 120ms) and may try to keep that latency as low as possible. During a gaming session, low latency values ​​can improve responsiveness to player input. (e.g., global) QoE (considering latency and video smoothness) may be scaled down due to video stuttering issues, which could, for example, cause users to suffer from system malfunctions.

[0057] As shown in Figures 6A and 6B, the measured jitter ローカル This can indicate high levels of inter-frame delay variation due to several phases exceeding 0.4 with an average value of 0.2. For example, Figure 6B shows jitter exceeding 0.5 over 4 seconds. ローカル (For example, shows the sequence of frames #10500 to #10750.) 0.5 jitter ローカルThe value can represent a variation of ±50% of the average inter-frame delay. For example, with an average delay of 16.66ms (which may represent an inter-frame delay of, for example, 60 frames per second), the jitter variation could fall within the range of [8ms~24ms]. For example, such variation, and jitter exceeding a given threshold (for example, 0.2 in this example). ローカル The value can be correlated with the presence of video stuttering. ローカル The value may correspond to the frequency of video stuttering, for example, jitter. ローカル A higher value indicates a higher frequency of stuttering in the video.

[0058] Inter-frame delay variation (e.g., jitter) ローカル ) may increase for any of the following reasons: -The first reason may be related to fluctuations in network conditions (e.g., fluctuations in RTT, the occurrence of congestion, and network errors that can delay the availability of (multiple) video frames by the amount of time associated with error (e.g., loss) recovery). -The second reason may relate to variations in server conditions that can affect the capture rate, encoding rate, and delivery of video frames (e.g., packets carrying video frames). Encoding parameters (e.g., compression by zone of interest), image type (e.g., I vs P vs intra-refresh images), and the amount of motion in the image can all affect the encoding duration, the size of the encoded image, and then the transmission time. For example, jitter that may be observed at frame time arrival may correlate with (e.g., correspond to) the jitter of the encoding. -Other reasons may relate to resource availability on the server, such as the current state of the game being played (and the complexity of the corresponding images), the presence of simultaneous streaming flows on the server, and the GPU power available to render frames at (e.g., a constant) FPS.

[0059] Examples of sporadic and long-lasting inter-frame delay variations For example, constant ad-hoc buffering may be included on the client device side to absorb jitter and obtain a smoother decoded frame rate, so that decoded video frames can be available for display at their presentation time.

[0060] As shown in Figure 6A, inter-frame delay fluctuations do not have to be constant over time and may fluctuate significantly during a game session. For example, they may have sporadic high values ​​over a long period of time at different points in time. Such situations (e.g., the jitter of frames #8000 to #11000 shown in Figure 6B) are examples of this. ローカル In terms of value, significant ad hoc buffering can prevent stuttering and allow all cases to be covered, which can significantly increase game latency and decrease QoE. For example, adding significant ad hoc buffering can affect the entire game session, given that this type of parameter can be set over the duration of the session (e.g., at the start of the session).

[0061] Figure 7 is a system diagram showing an example of a cloud gaming system based on SRT. According to the embodiment, the cloud gaming system may enable an improvement in the QoE of a game. A game can be run on a server instance 71. Images of the game (e.g., running) can be captured by an encoder 711 and encoded, for example, at a given encoding frame rate. The encoded frames can be encapsulated into multiple packets by an encapsulation processing module 712. The packets can be transmitted to a client device, for example, based on the SRT protocol. For example, before transmission, packets may be stored in a transmitter SRT buffer 714 so that packets (e.g., unacknowledged) can be retransmitted via a repair module 713. Packets may be timestamped with timestamp information associated with the time the packet was generated, transmitted, and stored in the transmitter SRT buffer (e.g., may include a timestamp). The server instance 71 may include a latency manager module 715 which can be configured to adjust the size of the transmitter SRT buffer 714 based on the exchange of SRT messages with a client device 72.

[0062] According to the embodiment, the client device 72 can receive, for example, multiple packets carrying video frames of the game via the SRT protocol. The received packets are stored in the receiver SRT buffer 724 and can be extracted based on the SRT buffer latency and the timestamp information contained in the packets, decapsulated by the decapsulation module 722, and decoded by the decoding module 727. The decoding module 727 can generate video frames that can be stored (e.g., push notifications) in the decoded frame queue 728. The presentation module 729 then displays the V sync It is possible to search for (for example, pop) a video frame to display based on the signal.

[0063] According to the embodiment, the client device 72 may include a stuttering monitoring module 726 which can be configured to monitor (e.g., detect, predict) the level of stuttering. The client device 72 may also include a latency manager module 725 which can be configured to adjust the size of the receiver SRT buffer 724 based on the exchange of SRT messages with the server.

[0064] According to one embodiment, the SRT protocol and its latency buffer can reduce video stuttering by making video frames available at the output of the client device protocol stack 720 at the same pace as the server-side production output 710.

[0065] Embodiments described herein may enable the exchange of latency information between a client device and a server (e.g., dynamically) during a game session (e.g., in addition to the start of a game session) in order to dynamically adapt (e.g., adjust) the SRT buffer latency to improve QoE.

[0066] For example, a client device can monitor several metrics (e.g., periodically) to detect and predict the occurrence (multiple) instances (e.g., and levels) of video stuttering.

[0067] According to the embodiment, either the client device or the server may determine a new SRT latency value to be set in the SRT buffer based on, for example, frame pace fluctuations (e.g., monitored stuttering (e.g., level, predicted level)).

[0068] According to one embodiment, a client device may exchange SRT control messages to synchronize the server to a new SRT buffer latency that can be determined (e.g., selected) by the client device (e.g., its latency manager module). The server may include a latency manager module for interacting with the client device.

[0069] According to one embodiment, the synchronization of the client device's presentation module 729 may be adjusted (for example, dynamically) based on changes in the SRT latency value to take into account a new SRT buffer latency value in order to reduce video frame skipping.

[0070] Example of stuttering detection For example, the stuttering monitoring module may obtain (e.g., monitor) the frame arrival time (e.g., the time corresponding to the time when the packet carrying the video frame was extracted from the receiver SRT buffer 724) in either of the outputs of the protocol stack and in the output of the decoding module 723 (e.g., either before or after decoding).

[0071] For example, a stuttering monitoring module can monitor the arrival time of either the first or last packet in a set of packets carrying a single frame. For instance, a packet may include signaling information indicating whether it corresponds to the first or last packet of a video frame.

[0072] For example, a metric (representing either video stuttering level or frame pace variation) can be obtained by applying a distribution function to any monitored arrival time (e.g., either frame or packet) at any point within the client device. The distribution function is applied to the inter-frame delay variation (jitter) of the window (e.g., sliding window). ローカルIt can be based on any of the following range functions: mean, median, standard deviation, variance, mean absolute deviation (MAD), interquartile range, and any other type of range function (e.g., beyond the range [25% to 75%]).

[0073] For example, a stuttering monitoring module measures the number of delayed frames per second (e.g., V Sync The stuttering monitoring module can obtain (e.g., monitor) either the frames that may have been missed or the number of dropped frames per second (e.g., in the case of an overflow of decoded frame queue 728). For example, the stuttering monitoring module may obtain the ratio of a first number to a second number of frames (over the same period), where the first number corresponds to the frames that appear in time (e.g., according to their presentation times) and the second number of frames corresponds to the total number of frames that appear (e.g., including duplicate frames).

[0074] For example, if the reception rate is lower than the capture rate (for example, due to errors in transmission), the stuttering monitoring module may obtain (e.g., monitor) the number of frames received (e.g., decoded) per second, the number of frame errors after decoding, and any statistics from the protocol stack (e.g., packet errors, out-of-order packets, packet drops on either the server or client side).

[0075] For example, any metric in any example described herein may be processed by latency manager modules 715, 725 which may be located in either the client device 72 or the server 71. Latency manager modules 715, 725 may be configured to obtain metrics representing, for example, the level of video stuttering and frame pace variation, based on an application (e.g., any distribution function) to any monitored metric described herein.

[0076] In the first example, the latency manager module 725 may be included in a client device, and the client device may acquire metrics representing, for example, the level of video stuttering and frame pace variation. In the second example, the latency manager module 755 may be included in a server, and the client device may transmit any monitored metrics to the server as described herein. In both the first and second examples, the new SRT latency value may be determined (e.g., requested) based on metrics according to any embodiment described herein.

[0077] The metric representing the level of video stuttering, obtained by latency manager modules 715 and 725, is expressed herein as Y = stuttering value (t i It may also be called a window, and in the formula, t i can also represent the time of receiving (decoding, respectively) the i-th frame, and the window is the time interval (e.g., [t) over which the metric can be processed (combined and calculated). i - window, t i ]) may represent metric Y, for example, each new incoming frame F i It can be obtained against.

[0078] According to the embodiment, latency manager modules 715, 725 calculate the (e.g., predicted) value of metric Y (Y in this specification) based on the history of past values. 予測 It is possible to obtain a new SRT latency value (which may be called SRT buffer latency in this specification). For example, a new SRT latency value (which may be called SRT buffer latency in this specification) can be obtained. 目標 (which may be called) is, for example, any of Y's history (for example, Y 予測) and, for example, frame pace fluctuations over different preceding time periods may be obtained. The new SRT buffer latency value may be set in the SRT client and may correspond to a (e.g., minimum) latency value that is large enough to reduce the (e.g., predicted) stuttering value. For example, the new SRT latency value may be increased (e.g., compared to the initial SRT latency value), while remaining limited to a limit that does not increase the latency beyond a certain value. For example, the new SRT latency value may be obtained based on hysteresis. For example, the latency may be increased (e.g., by the first value) if the level of stuttering exceeds the first level (e.g., the maximum threshold), and decreased (e.g., by the second value) if the level of stuttering falls below the second level (e.g., the threshold). More generally, the new (e.g., requested) latency value may be determined based on frame pace fluctuations (e.g., by latency manager modules 715, 725 which may be located in either the client device or the server). For example, if the frame pace variation (e.g., observed, measured, or predicted) increases, the new (e.g., requested) latency value may be determined to increase (compared to the initial latency value), and if the frame pace variation (e.g., observed, measured, or predicted) decreases, the new (e.g., requested) latency value may be determined to decrease (compared to the initial latency value).

[0079] SRT latency may be adjusted (e.g., updated) in the transmitter and client devices by exchanging SRT messages, as disclosed in embodiments described herein. For example, SRT buffering latency updates may be initiated (e.g., triggered) by either the client device or the server (depending on the location of the latency manager module, for example).

[0080] For example, though not limited to, a new SRT latency value can be described by the following formula, where the arrival time T is...i Inter-frame delay variation (jitter) applied in between ローカル It may be obtained based on the function.

[0081]

number

[0082] Example of SRT session latency settings For example, latency (e.g., through setting the SRT buffer latency value) may be configured during the extended handshake phase and may remain at the same value for the game session.

[0083] According to the embodiment, the SRT protocol may include either a live transmission mode or a cloud gaming transmission mode. The cloud gaming transmission mode may be associated with a function to dynamically update (e.g., adjust) the SRT latency value after the creation of the SRT session (e.g., at any point in time).

[0084] Figure 8A shows two examples of the extended handshake packet format. In the first example, the HSv4 extended handshake packet 81 may include capability information 810 indicating the ability to support dynamic SRT latency updates. In the second example, the HSv5 extended handshake packet 82 may include capability information 820 indicating the ability to support dynamic SRT latency updates.

[0085] For example, the ability to support dynamic SRT latency updates may be indicated by the seventh bit of the fields of SRT flags 810, 820, which may be referred to herein as SRT_OPT_DYNTSBPD (e.g., Dynamic Timestamp Buffer Packet Delivery). An SRT message having the SRT_OPT_DYNTSBPD flag set to 1 may indicate the ability of the SRT message transmitter to support dynamic SRT latency updates. Indicating the ability to support dynamic SRT latency updates through any other bits of the fields of SRT flags 810, 820 is applicable to the embodiments described herein.

[0086] Figure 8B shows an example of a handshake extension message flag. For example, the bitmask for SRT_OPT_DYNTSBPD could be 0x00000100.

[0087] The embodiments described herein are not limited to demonstrating the ability to support dynamic SRT latency updates via a single-bit flag. For example, any value of a field in an SRT extended handshake packet that demonstrates the ability to support dynamic SRT latency updates may be applicable to the embodiments described herein.

[0088] According to the embodiment, any entity (e.g., either an SRT listener or an SRT cola) may indicate whether a dynamic SRT latency update can be activated for a game session. Other entities may accept or reject the activation of the dynamic SRT latency update.

[0089] Figure 9A shows an example of an extended handshake message exchange to demonstrate the ability to support latency dynamic changes. For example, the message exchange may be initiated by the server.

[0090] In the first example 90A, the client device may receive an SRT extended handshake request message 901A containing information indicating the server's ability to support dynamic SRT latency updates. For example, the client device may accept performing dynamic SRT latency updates during the game session by transmitting an SRT extended handshake response message 902A containing information indicating the client's ability to support dynamic SRT latency updates.

[0091] In the second example 91A, the client device may receive a first SRT extended handshake request message 911A from the server that includes information indicating that it does not have support for dynamic SRT latency updates. For example, the client device may transmit a first SRT extended handshake response message 912A that includes information indicating the client's ability to support dynamic SRT latency updates in order to request the server to perform dynamic SRT latency updates. For example, the client device may receive a second SRT extended handshake request message 913A from the server that includes information confirming that it does not have support for dynamic SRT latency updates. For example, the client device may transmit a second SRT extended handshake response message 914A that includes information confirming that dynamic SRT latency updates may not be performed in a game session.

[0092] In the third example 92A, the client device may receive a first SRT extended handshake request message 921A from the server that includes information indicating that it does not have support for dynamic SRT latency updates. For example, the client device may transmit a first SRT extended handshake response message 922A that includes information indicating the client's ability to support dynamic SRT latency updates in order to request the server to perform dynamic SRT latency updates. For example, the client device may receive a second SRT extended handshake request message 923A that includes information indicating the server's ability to support dynamic SRT latency updates and the server's consent to perform dynamic SRT latency updates during the game session. For example, the client device may transmit a second SRT extended handshake response message 924A that includes information confirming that dynamic SRT latency updates may be performed during the game session.

[0093] Figure 9B shows another example of an extended handshake message exchange to demonstrate the ability to support latency dynamic changes. For example, the message exchange may be initiated by the client device.

[0094] In Example 1 90B, the client device may send an SRT Extended Handshake Request Message 901B containing information indicating the client device's ability to support dynamic SRT latency updates in order to request the server to dynamically update the SRT latency during the game session. The client device may receive an SRT Extended Handshake Response Message 902B containing information indicating the server's ability to support dynamic SRT latency updates, which may indicate the server's consent to perform dynamic SRT latency updates during the game session.

[0095] In the second example 91B, the client device may send a first SRT extended handshake request message 911B that includes information indicating that the client device does not support dynamic SRT latency updates. For example, the client device may receive a first SRT extended handshake response message 912B that includes information indicating the server's ability to support dynamic SRT latency updates in order to request the client device to perform dynamic SRT latency updates. For example, the client device may send a second SRT extended handshake request message 913B that includes information confirming that the client device does not support dynamic SRT latency updates in order to reject the server request. For example, the client device may receive a second SRT extended handshake response message 914B that includes information confirming that dynamic SRT latency updates may not be performed in the game session.

[0096] In the third example 92B, the client device may send a first SRT extended handshake request message 921B that includes information indicating that it does not support dynamic SRT latency updates. For example, the client device may receive a first SRT extended handshake response message 922B that includes information indicating the server's ability to support dynamic SRT latency updates in order to request the client device to perform dynamic SRT latency updates. For example, the client device may send a second SRT extended handshake request message 923B that includes information indicating the client's ability to support dynamic SRT latency updates in order to consent to perform dynamic SRT latency updates during a game session. For example, the client device may receive a second SRT extended handshake response message 924B that includes information confirming that dynamic SRT latency updates may be performed during a game session.

[0097] Example of changes in SRT session latency For example, after a dynamic latency update may be operable in an SRT session (for example, based on the exchange of SRT Extended Handshake functions), the receiver may invoke a latency change procedure (for example, a dynamic SRT latency update) based on, for example, the detection of video stuttering. Throughout the embodiments described herein, the expressions “dynamic SRT latency update” and “latency change procedure” can be used interchangeably to refer to the exchange of SRT messages for updating SRT latency values ​​in the server and client devices.

[0098] For example, a new SRT latency value may be obtained according to any embodiment described herein. As further described herein, the latency modification procedure may be initiated by either a client device or a server.

[0099] According to the embodiment, an initiator (e.g., either a server or a client device) may send an SRT request message, which may be referred to herein as a dynamic timestamp buffer packet delivery (DYNTSBPD) request message, indicating a new SRT latency value.

[0100] Figure 10 shows an example of the format of an SRT message for requesting a new SRT latency value. For example, an SRT Dynamic Timestamp Buffer Packet Delivery (DYNTSBPD) request message 1010 may include first information indicating the new SRT latency value requested by the initiator of the SRT latency change procedure (e.g., either a server or a client device). For example, an SRT DYNTSBPD response message 1020 may include second information indicating the responded SRT latency value in response to the new SRT latency value requested by the initiator (e.g., either a server or a client device). The SRT DYNTSBPD response message 1020 may be sent by another party (e.g., either a server or a client device) in response to the SRT DYNTSBPD request message 1010 sent by the initiator.

[0101] Figures 11A and 11B are two diagrams illustrating two examples of message exchange for a dynamic latency change procedure initiated by a client device and a server, respectively.

[0102] Referring to Figures 10 and 11A, in step 1110A, the client device can receive packets carrying the game's video frames and respond to them in accordance with the SRT protocol. The client device may obtain either the metric and a new SRT latency value in accordance with any embodiment described herein.

[0103] For example, in step 1120A, SRT DYNTSBPD request messages 1010, 1121A may be sent by the client device to the game server. The new (e.g., requested) SRT latency value may be indicated in first information (e.g., the TsbPd Rcv field) 1011 which may be included in the SRT DYNTSBPD request messages 1010, 1121A. For example, the SRT DYNTSBPD request messages 1010, 1121A may include a TsbPd Snd field 1012 which indicates a transmitter latency value that may indicate the initial SRT latency. The initial SRT latency may indicate, for example, the value before the SRT latency update (e.g., the current SRT latency value). The initial SRT latency may be a value such as one agreed upon during the (e.g., initial) SRT handshake.

[0104] For example, in step 1130A, the server may accept updating the SRT latency value to a new (e.g., requested) value. To indicate acceptance, the server may transmit SRT DYNTSBPD response messages 1020, 1131A to the client device indicating the new SRT latency value. The SRT DYNTSBPD response messages 1020, 1131A may include second information (e.g., the TsbPd Snd field) 1022 that can be set to the new (e.g., requested) SRT latency value. For example, the SRT DYNTSBPD response messages 1020, 1131A may include the TsbPd Snd field 1021 that can be set to the new SRT (e.g., requested) latency value. In step 1132A, the client device may update the SRT receiver latency value to the new SRT latency value after receiving SRT DYNTSBPD response messages 1020 and 1131A indicating acceptance of the new SRT latency value.

[0105] For example, in step 1140A, the server may reject (e.g., refuse) the change. To indicate refusal, the server may transmit SRT DYNTSBPD response messages 1020, 1141A to the client device indicating the current (e.g., initial) SRT value. SRT DYNTSBPD response messages 1020, 1141A may include second information (e.g., the TsbPd Snd field) 1022 which can be set to the current (e.g., initial) SRT latency value. In step 1142A, after receiving SRT DYNTSBPD response messages 1020, 1141A which include second information indicating refusal of the new SRT latency value, the client device may maintain the SRT receiver latency value at the current (e.g., initial) SRT latency value.

[0106] For example, in step 1150A, the server may accept updating the SRT latency value with an alternative value (which is different from the new value). For example, the alternative value may be any SRT latency value that is strictly between the current (e.g., initial) SRT latency value and the new (e.g., requested) SRT latency value.

[0107] To indicate partial acceptance (e.g., and alternative proposal), the server may transmit SRT DYNTSBPD response messages 1020, 1151A indicating an alternative SRT value to the client device. The SRT DYNTSBPD response messages 1020, 1151A may include second information (e.g., the TsbPd Snd field) 1022 which can be set to the alternative SRT latency value. In step 1152A, after receiving the SRT DYNTSBPD response messages 1020, 1151A which include the second information indicating the alternative SRT latency value, the client device may update the SRT receiver latency value to the alternative SRT latency value.

[0108] Referring to Figures 10 and 11B, in step 1110B, the client device can receive packets carrying the game's video frames and respond to them in accordance with the SRT protocol. The client device can obtain either the metric and the new SRT latency value in accordance with any embodiment described herein.

[0109] For example, in step 1111B, the client device may send a message containing information representing, for example, video frame rate fluctuations. For example, the information may describe (e.g., indicate) any value of any metric obtained based on monitoring of any arrival time in the client device according to any embodiment described herein.

[0110] For example, in step 1120B, SRT DYNTSBPD request messages 1010, 1121B may be sent by the server to the client. The new (e.g., requested) SRT latency value may be indicated in first information (e.g., the TsbPd Snd field) 1012 which may be included in the SRT DYNTSBPD request messages 1010, 1121B. For example, the SRT DYNTSBPD request messages 1010, 1121B may include a TsbPd Rcv field 1011 which indicates a receiver latency value that may represent the current (e.g., initial) SRT latency.

[0111] For example, in step 1130B, the client device may accept updating the SRT latency value to a new value. For example, the client device may update the SRT receiver buffer latency according to the new value. To indicate acceptance, the client device may transmit SRT DYNTSBPD response messages 1020, 1131B to the server indicating the new SRT latency value. The SRT DYNTSBPD response messages 1020, 1131B may include second information (e.g., the TsbPd Rcv field) 1021 that can be set to the new (e.g., requested) SRT latency value. For example, the SRT DYNTSBPD response messages 1020, 1131B may include the TsbPd Snd field 1022 that can be set to the new SRT (e.g., requested) latency value. In step 1132B, after receiving SRT DYNTSBPD response messages 1020 and 1131B indicating acceptance of the new SRT latency value, the server may update the SRT transmitter latency value (SndTsbPdDelay) to the new SRT latency value.

[0112] For example, in step 1140B, the client device may reject the change (e.g., refuse) and maintain the SRT receiver latency value at the current (e.g., initial) SRT latency value. To indicate rejection, the client may send SRT DYNTSBPD response messages 1020, 1141B to the server indicating the current (e.g., initial) SRT value. The SRT DYNTSBPD response messages 1020, 1141B may include second information (e.g., the TsbPd Rcv field) 1021 that can be set to the current (e.g., initial) SRT latency value. In step 1142B, after receiving the SRT DYNTSBPD response messages 1020, 1131B indicating rejection of the new SRT latency value, the server may maintain the SRT transmitter latency value at the current (e.g., initial) SRT latency value.

[0113] For example, in step 1150A, the client device may accept updating the SRT latency value with an alternative value (e.g., different from the new value). For example, the new value (e.g., requested by the server) may be incompatible with the client device's buffering capacity (the buffer may be too small). In another example, the client device may determine that the new SRT latency value requested by the server may be too small (e.g., if the new latency is less than three times the current RTT value) to allow the retransmission technique to recover lost packets (e.g., efficiently). For example, the alternative value may be any SRT latency value that is strictly between the current (e.g., initial) SRT latency value and the new (e.g., requested) SRT latency value. To indicate partial acceptance (e.g., and alternative proposal), the client device may transmit SRT DYNTSBPD response messages 1020, 1151B indicating the alternative SRT value to the server. The SRT DYNTSBPD response messages 1020, 1151B may include second information (e.g., the TsbPd Snd field) 1021 which can be set to an alternative SRT latency value. In step 1152B, after receiving the SRT DYNTSBPD response messages 1020, 1151B indicating the alternative SRT latency value, the server may update the SRT transmitter latency value to the alternative SRT latency value.

[0114] The embodiments described herein are not limited to the SRT message formats (e.g., fields) described herein, and any other SRT message format, more generally, any message conforming to any other protocol for updating buffer sizes between a server and a client device, may be applicable to the embodiments described herein.

[0115] Example of presentation module processing for protocol latency updates An example of processing by a client device (e.g., a client device presentation processing module) is described herein with reference to, for example, Figure 7.

[0116] In the first example, the processing of the client device (e.g., the client device's presentation processing module 729) may be independent (e.g., immutable) of any SRT latency (e.g., dynamic) updates. For example, the presentation processing module 729 may retrieve (e.g., listen to) decoded frames in the decoded frame queue 728. Sync In each (e.g.) period of the signal, one available frame may be popped (e.g., extracted) from queue 728 and displayed on the screen. Sync If there are no available frames in the queue during the signaling period, previously presented (e.g., displayed) frames may be presented again (e.g., displayed). If the decoded frame queue 728 overflows, any number of frames may be dropped according to any criteria. The criteria may be independent of the frame presentation time, for example.

[0117] In the second example, the client device (its presentation processing module 729) can process the presentation time associated with each frame. For example, if the SRT latency value is updated with a new value, the presentation processing module 729 may notify (e.g., instruct) this change (e.g., according to any mechanism within the client device). For example, an update to the SRT latency may be represented (e.g., indicated) in the form of an additional signed delay (e.g., positive in the case of a latency increase, and negative otherwise). For example, an additional signed delay corresponding to the difference between the previous protocol latency and the current protocol latency may represent additional signed time that the frame may have been buffered in the protocol stack before being delivered to the application (e.g., for decoding).

[0118] For example, the presentation processing module 729 processes the first frame (F0) as V SyncA clock reference time (T0) can be obtained (e.g., maintained) that corresponds to the time that can be presented (e.g., displayed) on the screen during the period. For simplicity and without loss of generality, frames can be delivered sequentially at the capture rate. For example, frames can be obtained by the sequence index of the frames (e.g., the i-th frame F). i is, T i The calculated presentation time T can be obtained from (which can be presented in the capture period = T0 + ix capture period) i It may be presented (e.g., displayed) based on the following. If the SRT latency is updated, an additional signed delay may be used to shift the clock reference time to positive or negative, for example, T0 = T0 + delay.

[0119] For example, if the SRT latency increases, the last displayed frame may be displayed again a number of times corresponding to the latency increase before the operation of popping a frame from the decoded frame queue 728 for display is restored (e.g., a frame may not be popped from the decoded frame queue 728 for display). The last displayed frame (which may be duplicated) during an SRT latency increase may correspond to the last frame that could have been displayed before the SRT latency could be updated. For example, (e.g., frame F i Presentation time T (after it is displayed) i If a 33-millisecond SRT latency increase occurs (for example, corresponding to two periods on a 60Hz screen), the presentation module may display this same frame for two subsequent (e.g., capture) periods before it can pop any frame from the queue. For example, one frame may be frozen over three periods, while all subsequent frames may be displayed at regular intervals without stuttering. Duplicating frames at a point in time (e.g., a single point in time) corresponding to the latency buffer increase can allow any stuttering issues to be concentrated during the latency change time, contributing to an overall QoE improvement.

[0120] For example, adjusting the clock reference time based on an additional signed delay could also make it possible to determine which (e.g., slower) frames may be dropped (e.g., not displayed). For example, frame F i If it is too late to decrypt and present it in time, and the next frame F i+1 If available in the decoded frame queue 728, the presentation processing module determines which frame is V based on the clock reference time. Sync In the next period of the signal (for example, time T) sync_period It is possible to determine whether it can be presented (e.g., displayed) in T0. For example, using a clock reference time, the presentation time T for the i-th and (i+1)th frames can be determined based on T0. i and T i+1 It is possible to determine V. Sync The frame displayed in the next cycle of the signal is the one with a presentation time of T sync_period It is most likely to be (for example, |T sync_period -T i |, |T sync_period -T i+1 (Minimum value between |)(F i and F i+1 It may be a frame (within F i and F i+1 Other frames (within this case) may be dropped.

[0121] Example of server processing related to protocol latency updates According to one embodiment, the SRT buffer latency is used on the server side to hold transmitted packets (for example, for a time period corresponding to the SRT buffer latency) and wait for acknowledgments of those packets by the receiver (for example, for possible retransmissions). According to one embodiment, the server can adapt the SRT transmitter buffer to any changes in buffer latency values ​​that may occur during an SRT session.

[0122] In the first example, SRT buffer latency (e.g., RcvTsbPdDelay) can be reduced by the client device (e.g., receiver). The server (e.g., transmitter) may decide to send some retransmitted packets (e.g., due to updated shorter client deadlines (e.g., lower SRT buffer latency)). To anticipate high jitter differences (e.g., and future increases in SRT buffer latency due to frequent buffer latency changes), the server may hold packets that have just expired for an extra duration so that they can be retransmitted if the buffer increases in the near future.

[0123] In the second example, if the receiver increases the SRT buffer latency to buffer more packets, the transmitter can use the extra time to send more retransmitted packets.

[0124] Figure 12A shows an example of a client (e.g., processing) device 12A for improving the QoE of video content, such as games. For example, the processing device 12A may include a cloud gaming (e.g., thin) client that can interact with a content (e.g., game) server. For example, the processing device 12A may have an input interface 1260 configured to receive commands (e.g., user input) from any number of input devices (e.g., joysticks, mice, keyboards, remote controls, etc.). The commands may be directed to the content (e.g., game) server. For example, the processing device 12A may have a network interface 1200 for connecting to a network. The network interface 1200 may be configured to receive multiple packets from a server. The packets may contain (e.g., carry) video frames of video content (e.g., games) that are decoded and displayed by the client (e.g., processing) device 12A (e.g., packets may contain a portion of a video frame, and the video frame may be carried through a set of packets). For example, the network interface 1200 may be configured to send packets to the server. The packets may include, for example, (user) input commands. For example, the network interface 1200 may be configured to exchange any type of control message (e.g., SRT messages) with the server. According to the embodiment, the network interface 1200 may be any of the following: - Wireless local area network interfaces such as Bluetooth, any type of Wi-Fi, or any type of wireless interface in the IEEE 802 series of network interfaces, -Wired LAN interfaces such as Ethernet, IEEE 802.3, or any wired interface in the IEEE 802 series of network interfaces. -Wired bus interfaces such as USB, FireWire, or any type of wired bus technology, - Broadband cellular wireless network interfaces such as 2G / 3G / 4G / 5G cellular wireless network interfaces that comply with 3GPP specifications in any of the releases. - Wide-area network interfaces such as xDSL, FFTx, or WiMAX interfaces.

[0125] More generally, any network interface that enables sending any of the packets (e.g., user input commands and control packets) and receiving multiple packets carrying video frames may be applicable to the embodiments described herein.

[0126] According to one embodiment, the network interface 1200 may be coupled to a processing module 1220 which can be configured to apply an initial latency value to received packets before decoding and display. For example, the processing module 1220 may be configured to obtain a metric value representing either the video stuttering level or the variation in the pace at which video frames may arrive. For example, the processing module 1220 may be configured to exchange messages with a server via the network interface 1200 to obtain a new latency value based on the metric value. In a first example of message exchange, the processing module 1220 may be configured to send a request message to the server containing first information indicating a requested latency value determined based on the frame pace variation, and the processing module 1220 may be configured to receive a response message from the server containing second information indicating a new latency value. In a second example of message exchange, the processing module 1220 may be configured to send information to the server indicating a metric representing frame pace variation, the processing module 1220 may be configured to receive a request message from the server containing first information indicating a requested latency value, and the processing module 1220 may be configured to send a response message to the server containing second information indicating a new latency value based on the requested latency value. For example, the processing module 1220 may be configured to apply a new SRT latency value to received subsequent packets before decoding and displaying the subsequent packets. For example, the processing module 1220 may be configured to decode video frames and send video frames for display on a video output unit 1240, such as a display means. According to the embodiment, the internal or external display means may be a personal computer screen, a TV screen, a tablet screen, or a smartphone screen. More generally, any display means that enables the display of video content, such as a game, may be applicable to the embodiments described herein.

[0127] Figure 12C shows an example of a server (e.g., processing) device 12C for improving the QoE of video content, such as a game. For example, the processing device 12C may interact with a client device. For example, the processing device 12C may include a network interface 1210 for connecting to a network. The network interface 1210 may be configured to send a plurality of packets to the client device. The packets may contain (e.g., carry) video frames of video content (e.g., a game) to be decoded and displayed by the client device. For example, a packet may contain a portion of a video frame, and the video frame may be carried through a set of packets. For example, the network interface 1210 may be configured to receive packets from the client device. The packets may contain (e.g., user) input commands. For example, the network interface 1210 may be configured to exchange any kind of control messages (e.g., SRT messages) with the client device. According to the embodiment, the network interface 1210 may be any of the following: - Wireless local area network interfaces such as Bluetooth, any type of Wi-Fi, or any type of wireless interface in the IEEE 802 series of network interfaces, -Wired LAN interfaces such as Ethernet, IEEE 802.3, or any wired interface in the IEEE 802 series of network interfaces. -Wired bus interfaces such as USB, FireWire, or any type of wired bus technology, - Broadband cellular wireless network interfaces such as 2G / 3G / 4G / 5G cellular wireless network interfaces that comply with 3GPP specifications in any of the releases. - Wide-area network interfaces such as xDSL, FFTx, or WiMAX interfaces.

[0128] More generally, any network interface that enables receiving any of the packets (e.g., user input commands and control packets) and transmitting multiple packets carrying video frames may be applicable to the embodiments described herein.

[0129] According to one embodiment, the network interface 1210 may be coupled to a processing module 1230 which can be configured to send a plurality of packets carrying video frames of video content to a client device. For example, the processing module 1230 may be configured to hold packets in a buffer for retransmission for a first amount of time associated with an initial latency value. For example, the processing module 1230 may be configured to exchange messages with a server via the network interface 1210 to obtain a new latency value. In a first example of message exchange, the processing module 1230 may be configured to receive a request message from the client device containing first information indicating a requested latency value, and the processing module 1230 may be configured to send a response message to the client device containing second information indicating a new latency value (which may be set to, for example, the requested latency value). In a second example of message exchange, the processing module 1230 may be configured to receive information indicating a metric representing a variation in frame pace, the processing module 1230 may be configured to send a request message to a client device, the request message including first information indicating a requested latency value determined based on the indicated metric, and the processing module 1230 may be configured to receive a response message from the client device, the response message including second information indicating a new latency value.

[0130] For example, the processing module 1230 may be configured to run a game instance. For example, the processing module 1230 may include a GPU (not shown) which can be configured to render (e.g., subsequent) video frames of the game. For example, the processing module 1230 may be configured to encode the rendered (e.g., subsequent) video frames and encapsulate the encoded (e.g., subsequent) video frames in a plurality of (e.g., subsequent) packets to be sent to a client device. For example, the processing module 1230 may be configured to send a plurality of subsequent packets carrying subsequent video frames of video content to a client device. For example, the processing module 1230 may be configured to hold the subsequent packets in a buffer for retransmission for a second amount of time associated with a new latency value.

[0131] Figure 12B shows an example of one of the client and server (e.g., processing) devices 12A, 12C described herein. Processing devices 12A, 12C may comprise one or more processors 1210, which may be, for example, a CPU, GPU, or DSP (an acronym for Digital Signal Processor), along with internal memory 1250 (e.g., any of RAM, ROM, or EPROM). Processing devices 12A, 12C may comprise any number of input / output interfaces 1230 adapted to transmit output information and / or allow a user to input commands and / or data (e.g., any of a keyboard, mouse, touchpad, webcam, or display), and / or transmit / receive data via a network interface, and a power supply 1270 which may be external to the processing devices 12A, 12C.

[0132] According to one embodiment, the processing devices 12A and 12C may further include a computer program stored in the memory 1250. The computer program may include instructions that cause the processing devices 12A and 12C to perform the processing methods described with reference to any of Figures 13, 14, 15, 16, and 17, when executed by the processing devices 12A and 12C, particularly by the (multiple) processors 1210. According to one variant, the computer program may be stored outside the processing devices 12A and 12C on non-temporary digital data support, for example, on an external storage medium such as an SD card, HDD, CD-ROM, DVD, read-only and / or DVD drive, or DVD read / write drive, all known in the art. The processing devices 12A and 12C may be provided with an interface for reading the computer program. Furthermore, the processing devices 12A and 12C may access any number of Universal Serial Bus (USB) type storage devices (e.g., "Memory Sticks") via a corresponding USB port (not shown).

[0133] According to the embodiment, the processing device 12A may be any of the following: a game device, a set-top box device, a TV set, a digital media player (e.g., a renderer) device, an internet gateway, a mobile device, a communication device, a tablet (or tablet computer), a smartphone, a laptop computer, or a desktop computer. According to the embodiment, the processing device 12C may be any of the following: a desktop computer, a server, or an instance of a server.

[0134] Figure 13 shows an example of a method for improving the QoE of a game. For example, the method may be implemented in a client device (e.g., cloud gaming). According to one embodiment, in step 1310, a number of packets carrying video frames of the game may be received by the client device (e.g., cloud gaming) from a server. According to one embodiment, in step 1320, an initial SRT latency value may be applied to the received packets before decoding and display. According to one embodiment, in step 1330, a metric value representing the video stuttering level may be obtained based on video frame pace fluctuations. According to one embodiment, in step 1340, SRT messages may be exchanged between the client device (e.g., cloud gaming) and the server to obtain a new SRT latency value based on the metric value. According to one embodiment, in step 1350, the QoE of the game may be improved by applying the new SRT latency value to subsequent packets received before decoding and display.

[0135] For example, received packets and subsequent received packets may be stored in an SRT receiver buffer, and the SRT latency value may be applied to the packets by extracting them from the SRT receiver buffer based on the SRT latency value and timestamp information contained in the packets. For example, the timestamp information may indicate the respective time associated with the storage of the packets in the server's SRT transmitter buffer. For example, the SRT latency value may be either an initial SRT latency value or a new SRT latency value.

[0136] For example, a timestamp included in a packet may indicate the first time the packet may have been stored in the SRT transmitter buffer. For example, a packet may be extracted at a second time corresponding to a time quantity after a first time, the time quantity may be constant for consecutive packets, and may be associated with an SRT latency value.

[0137] For example, video frame pace variation may be variation in the pace at which video frames are received, extracted from the SRT receiver buffer, and decoded.

[0138] For example, the new SRT latency value may be obtained based on a history of metric values ​​acquired over different preceding periods.

[0139] For example, metric values ​​can be obtained based on one of the following distribution functions: frame arrival time, packet arrival time, frame decoding time, and frame display time.

[0140] For example, the variance function may be the mean, median, standard deviation, variance, mean absolute deviation, or quartiles.

[0141] For example, the metric value may be based on either the first number of delayed frames or the second number of dropped frames.

[0142] For example, an SRT request message may be sent to a server by a client device (e.g., a cloud gaming device). The SRT request message may include first information indicating the requested SRT latency value. The requested SRT latency value may correspond to a new SRT latency value that can be determined by the client device (e.g., a cloud gaming device).

[0143] For example, an SRT response message may be received by a client device from a server (e.g., in cloud gaming). The SRT response message may include second information indicating the transmitter SRT latency value in response to the requested latency value. For example, the new SRT latency value applied to subsequent received packets may be set to the transmitter SRT latency value, provided that the transmitter SRT latency value is equal to the requested SRT latency value, or falls strictly between the requested SRT latency value and the initial SRT latency value.

[0144] For example, metric values ​​may be sent to the server by a client device (e.g., a cloud gaming device).

[0145] For example, an SRT request message may be received by a client device from a server (e.g., cloud gaming). The SRT request message may include first information indicating a new SRT latency value.

[0146] For example, an SRT response message may be sent to the server by a client device (e.g., a cloud gaming device). The SRT response message may include second information indicating a new SRT latency value for acknowledging the new SRT latency value.

[0147] For example, an SRT Enhanced Handshake Request message may be sent by a client device (e.g., cloud gaming) to a server before the packets carrying video frames can be received. The SRT Enhanced Handshake Request message can indicate the client's ability to support dynamic latency operation.

[0148] For example, an SRT enhanced handshake response message may be received by a client device (e.g., cloud gaming) from a server in response to an SRT enhanced handshake request message. The client device (e.g., cloud gaming) may perform dynamic latency operation on the condition that the SRT enhanced handshake response message indicates the server's ability to support dynamic latency operation.

[0149] For example, before packets carrying video frames can be received, an SRT enhanced handshake request message may be received by a client device (e.g., in cloud gaming) from a server. If the SRT enhanced handshake request message indicates the server's ability to support dynamic latency operation, an SRT enhanced handshake response message indicating the client's ability to support dynamic latency operation may be sent by the client device (e.g., in cloud gaming) to the server.

[0150] For example, an SRT enhanced handshake request message may be received by a client device (e.g., cloud gaming) from a server before packets carrying video frames can be received, and an SRT enhanced handshake response message may be sent by the client device (e.g., cloud gaming) to a server indicating the client's ability to support dynamic latency operation, provided that the SRT enhanced handshake request message does not include an indication of the server's ability to support dynamic latency operation.

[0151] For example, another SRT extended handshake request message may be received by a client device (e.g., cloud gaming), and dynamic latency operation may be performed on the condition that the other SRT extended handshake request message indicates the server's ability to support dynamic latency operation.

[0152] Figure 14 shows a first example of a method implemented in a client device to improve the QoE of video content. For example, in step 1410, the client device may receive multiple packets from the server carrying video frames of video content. For example, in step 1420, the client device may apply an initial latency value to the received packets before decoding and displaying them. For example, in step 1430, the client device may send a request message to the server that may include first information indicating a requested latency value which may be determined based on frame pace fluctuations. For example, in step 1440, the client device may receive a response message from the server that may include second information indicating a new latency value. For example, in step 1450, the client device may apply the new latency value to subsequent packets received before decoding and displaying them.

[0153] For example, the initial latency value and the new latency value can be applied to the packet and the subsequent packet respectively by extracting the packet and the subsequent packet from the receiver buffer based on (i) the initial latency value and the new latency value, respectively, and (ii) the timestamp information contained in the packet and the subsequent packet. The timestamp information may, for example, indicate the respective time associated with the storage of the packet and the subsequent packet in the server's transmitter buffer.

[0154] For example, timestamp information contained in either a packet or a subsequent packet may indicate a first time when the packet may have been stored, for example, in a transmitter buffer. For example, the packet may be extracted at a second time corresponding to a time amount after the first time, where the time amount is constant for consecutive packets and is associated with either an initial latency value or a new latency value.

[0155] For example, frame pace variation may be a variation in the pace at which a video frame is received, extracted from the receiver buffer, or decoded.

[0156] For example, a new latency value may be obtained based on a history of frame rate fluctuations over different preceding periods.

[0157] For example, the new latency value may be based on either the number of delayed frames or the number of dropped frames.

[0158] For example, a new latency value may be applied to subsequent packets, provided that the new latency value is equal to the requested latency value, or falls strictly between the requested latency value and the initial latency value.

[0159] For example, before a packet carrying a video frame can be received, a client device may send a handshake request message to the server. For example, the handshake request message may contain third information indicating the client's ability to support dynamic latency operation.

[0160] For example, a client device may receive a handshake response message from a server in response to a handshake request message. For example, a client device may perform dynamic latency operation by sending a request message, provided that the handshake response message contains fourth information indicating the server's ability to support dynamic latency operation.

[0161] For example, a client device may receive a handshake request message from the server before a packet carrying a video frame can be received. Provided the handshake request message includes a third piece of information indicating the server's ability to support dynamic latency operation, the client device may send a handshake response message to the server, which may include a fourth piece of information indicating the client's ability to support dynamic latency operation.

[0162] For example, the initial latency value and the new latency value may be the Secure Reliable Transport (SRT) latency value associated with the SRT protocol.

[0163] Figure 15 shows a second example of a method implemented on a server to improve the QoE of video content. For example, in step 1510, the server may send multiple packets carrying video frames of the video content to the client device. For example, in step 1520, the server may hold packets in a buffer for a first amount of time associated with an initial latency value, for example, for retransmission. For example, in step 1530, the server may receive a request message from the client device, which may include first information indicating a requested latency value. For example, in step 1540, the server may send a response message to the client device, which may include second information indicating a new latency value. For example, the new latency value may be set to the requested latency value to indicate the server's acceptance of the requested latency value. For example, in step 1550, the server may send multiple subsequent packets carrying subsequent video frames of the video content to the client device. For example, in step 1560, the server may hold subsequent packets for retransmission in a buffer, for example, for a second amount of time associated with the new latency value.

[0164] Figure 16 shows a third example of a method implemented in a client device to improve the QoE of video content. For example, in step 1610, the client device may receive multiple packets from the server carrying video frames of video content. For example, in step 1620, the client device may apply an initial latency value to the received packets before decoding and displaying them. For example, in step 1630, the client device may send information to the server indicating a metric representing frame pace variation. For example, in step 1640, the client device may receive a request message from the server, which may include first information indicating the requested latency. For example, in step 1650, the client device may send a response message to the server, which may include second information indicating a new latency value based on the requested latency value. For example, the new latency value may be set to the requested latency value to indicate acceptance of the requested latency value by the client device. For example, in step 1660, the client device may apply the new latency value to subsequent packets received before decoding and displaying them.

[0165] For example, the metric can be obtained based on one of the following distribution functions: frame arrival time, packet arrival time, frame decoding time, and frame display time.

[0166] For example, the variance function can include the mean, median, standard deviation, variance, mean absolute deviation, and quartiles.

[0167] For example, the metric may be based on either the number of delayed frames or the number of dropped frames.

[0168] Figure 17 shows a fourth example of a method implemented on a server to improve the QoE of video content. For example, in step 1710, the server may send multiple packets carrying video frames of video content to a client device. For example, in step 1720, the server may hold packets in a buffer for retransmission for a first amount of time associated with an initial latency value. For example, in step 1730, the server may receive information indicating a metric representing frame pace variation. For example, in step 1740, the server may send a request message to the client device that may include first information indicating a requested latency value. For example, the server may determine the requested latency value based on the indicated metric. For example, in step 1750, the server may receive a response message from the client device that may include second information indicating a new latency value. For example, the new latency value may be set to the requested latency value to indicate acceptance of the requested latency value by the client device. For example, in step 1760, the server may send multiple subsequent packets carrying subsequent video frames of the video content to the client device. For example, in step 1770, the server may hold the subsequent packets for retransmission in a buffer, for example, for a second amount of time associated with the new latency value.

[0169] conclusion While features and elements are described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein can be implemented in computer programs, software, or firmware embedded in computer-readable media for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with software can be used to implement a radio frequency transceiver for use in a WTRU102, UE, terminal, base station, RNC, or any host computer.

[0170] Although not explicitly stated, the embodiments described herein may be used in any combination or partial combination. For example, the principles described herein are not limited to the modifications described herein, and any modifications and arrangements of the embodiments may be used. Furthermore, the principles described herein are not limited to the channel access methods described herein, and any other type of channel access method may be compatible with these principles.

[0171] Any features, modifications, or embodiments described in the Method are compatible with apparatus devices including means for processing the disclosed Method, compatible with devices comprising a processor configured to process the disclosed Method, compatible with computer program products including program code instructions, and compatible with non-temporary computer-readable storage media storing program instructions.

[0172] Furthermore, the embodiments described above include other devices, including processing platforms, computing systems, controllers, and processors. These devices may include at least one central processing unit ("CPU") and memory. According to the convention of those skilled in the art in the field of computer programming, references to operations and symbolic representations of arithmetic or instructions may be performed by various CPUs and memories. Such operations and arithmetic or instructions may be referred to as "executed," "executed by the computer," or "executed by the CPU."

[0173] Those with ordinary art in the relevant field will understand that operations and symbolically represented arithmetic or instructions involve the manipulation of electrical signals by the CPU. The electrical system represents data bits that can cause a resulting transformation or reduction of electrical signals, and maintains these data bits in memory locations in the memory system, thereby reconfiguring or otherwise modifying the CPU's operations and processing of other signals. The memory locations where the data bits are maintained are physical locations having specific electrical, magnetic, optical, or organic properties that correspond to or represent the data bits. It should be understood that typical embodiments are not limited to the platforms or CPUs described above, and other platforms and CPUs may support the methods provided.

[0174] Data bits may also be maintained on computer-readable media, including magnetic disks, optical disks, and any other volatile (e.g., Random Access Memory ("RAM")) or CPU-readable non-volatile (e.g., Read-Only Memory ("ROM")) mass storage systems. The computer-readable media may include cooperative or interconnected computer-readable media distributed among multiple interconnected processing systems, which may reside exclusively on a processing system or be local or remote to the processing system. Typical embodiments are not limited to the memory described above, and it is understood that other platforms and memories may support the methods described.

[0175] In exemplary embodiments, any of the operations, processes, etc., described herein may be implemented as computer-readable instructions stored on a computer-readable medium. These computer-readable instructions may be executed by processors in mobile devices, network elements, and / or any other computing devices.

[0176] There is little distinction between hardware and software implementations of a system configuration. The use of hardware or software is generally a design choice representing a cost-effectiveness trade-off (for example, in that, though not always, the choice between hardware and software can be important in certain contexts). Various vehicles (e.g., hardware, software, and / or firmware) may exist in which the processes and / or systems and / or other technologies described herein may be effective, and the preferred vehicle may vary depending on the context in which the processes and / or systems and / or other technologies are deployed. For example, if the implementer determines that speed and accuracy are paramount, the implementer may choose primarily hardware and / or firmware vehicles. If flexibility is paramount, the implementer may choose primarily software implementations. Alternatively, the implementer may choose any combination of hardware, software, and / or firmware.

[0177] The detailed description above illustrates various embodiments of devices and / or processes through the use of block diagrams, flowcharts, and / or examples. Those skilled in the art will understand that, insofar as such block diagrams, flowcharts, and / or examples include one or more functions and / or operations, each function and / or operation in such block diagrams, flowcharts, or examples may be implemented individually and / or collectively by a wide range of hardware, software, firmware, or substantially any combination thereof. Suitable processors include, by example, general-purpose processors, dedicated processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines.

[0178] While the features and elements are provided above in specific combinations, it will be understood by those with ordinary art in the art that each feature or element can be used individually or in any combination with other features and elements. This disclosure is not limited in terms of the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope of the invention. Any elements, actions, or instructions used in the description of this application should not be construed as important or essential to the invention unless expressly presented as such. In addition to those enumerated herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art from the above description. Such modifications and variations are intended to fall within the scope of the appended claims. This disclosure is limited only by the terms of the appended claims, and is limited in the same way as the full scope of the equivalents to which such claims are entitled. It should be understood that this disclosure is not limited to any particular method or system.

[0179] Furthermore, it should be understood that the terms used herein are for the sole purpose of describing specific embodiments and are not intended to limit the present invention. Where used herein, “Station” and its abbreviation “STA,” “User Equipment” and its abbreviation “UE” may mean (i) a radio transmission and / or receiving unit (WTRU) such as the infrastructure described herein, (ii) any of several embodiments of a WTRU such as the infrastructure described herein, (iii) a radio-enabled and / or wired (e.g., tetherable) device configured to have some or all of the structure and functions of a WTRU such as the infrastructure described herein, (iii) a radio-enabled and / or wired device configured to have less than all of the structure and functions of a WTRU such as the infrastructure described herein, or (iv) other. Details of exemplary WTRUs that may represent any UE enumerated herein are provided below with respect to Figures 1A to 1D.

[0180] In certain representative embodiments, some parts of the subject matter described herein may be implemented via application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and / or other integrated formats. However, it will be recognized by those skilled in the art that some aspects of the embodiments disclosed herein may be equivalently implemented in an integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or substantially any combination thereof, and that designing circuits and / or writing software and / or firmware code is within the scope of the art of those skilled in the art in light of this disclosure. In addition, it will be understood by those skilled in the art that the mechanisms of the subject matter described herein may be distributed as various forms of program products, and that the exemplary embodiments of the subject matter described herein are applicable regardless of the particular type of signal-carrying medium used to actually carry out the distribution. Examples of signal-carrying media include, but are not limited to, recordable media such as floppy disks, hard disk drives, CDs, DVDs, digital tapes, and computer memory, as well as transmitting media such as digital and / or analog communication media (e.g., optical fiber cables, waveguides, wired communication links, wireless communication links, etc.).

[0181] The subject matter described herein may, in some cases, depict different components that are contained within or connected to other different components. Such illustrated architectures are merely examples, and it should be understood that in practice, many other architectures can be implemented to achieve the same function. Conceptually, any arrangement of components to achieve the same function is effectively “associated” in such a way that the desired function can be achieved. Therefore, any two components combined herein to achieve a particular function, regardless of architecture or intermediate components, can be seen as “associated” with each other in such a way that the desired function can be achieved. Similarly, any two components thus associated can be considered “operably connected” or “operably coupled” with each other to achieve the desired function, and any two components that can be associated in such a way can be considered “operably coupled” with each other to achieve the desired function. Specific examples of operably coupled components include, but are not limited to, physically matable and / or physically interacting components, and / or wirelessly interactable and / or wirelessly interacting components, and / or logically interacting and / or logically interactable components.

[0182] With regard to the use of substantially any plural and / or singular terms herein, those skilled in the art can convert from plural to singular and / or singular to plural as appropriate to the context and / or use. For clarity purposes, various singular / plural rearrangements may be explicitly described herein.

[0183] In general, it will be understood by those skilled in the art that the terms used herein, and especially in the appended claims (e.g., in the body of the appended claims), are generally intended to be “non-limiting” terms (for example, the term “contains” should be interpreted as “contains but not limited to,” the term “has” should be interpreted as “has at least,” and the term “contains” should be interpreted as “contains but not limited to.”). Furthermore, it will be understood by those skilled in the art that if a particular number of claims introduced are intended to be described, such intent is explicitly stated in the claim, and if such statement is not present, such intent does not exist. For example, if only one item is intended, the term “single” or similar word may be used. To aid understanding, the following appended claims and / or descriptions herein may include the use of the introductory phrases “at least one” and “one or more” to introduce the description of the claims. However, the use of such phrases should not be interpreted as meaning that the introduction of a claim description by the indefinite article "a" or "an" limits any particular claim containing such introduced description to embodiments containing only one such description, even if the same claim contains the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an" (for example, "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"). The same applies to the use of definite articles used to introduce a claim description. In addition, it will be recognized by those skilled in the art that even if a particular number of descriptions in an introduced claim are explicitly stated, such description should be interpreted as meaning at least the number stated (for example, the simple statement "two descriptions" without other modifiers means at least two descriptions or two or more descriptions).

[0184] Furthermore, when a notation similar to "at least one of A, B, and C" is used, such a structure is generally intended to mean what a person skilled in the art would understand (for example, "a system having at least one of A, B, and C" includes, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together). When a notation similar to "at least one of A, B, or C" is used, such a structure is generally intended to mean what a person skilled in the art would understand (for example, "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together). It will be further understood by those skilled in the art that any substantially any disjunct word and / or phrase presenting two or more alternative terms in the specification, claims, or drawings should be understood as construed to include the possibility of including one of the terms, either of the terms, or both of the terms. For example, the phrase “A or B” should be understood to include the possibility of “A” or “B” or “A and B.” Furthermore, as used herein, the term “any of ~” followed by a list of multiple items and / or a list of categories of multiple items is intended to include “any of,” “any combination of,” “any number of,” and / or “any number of combinations of,” of the items and / or categories of items, individually or in combination with other items and / or categories of other items. Furthermore, as used herein, the term “set / group” or “cluster” is intended to include any number of items, including zero. In addition, as used herein, the term “number” is intended to include any number, including zero.

[0185] In addition, if any feature or aspect of the present disclosure is described from the perspective of the Markush group, a person skilled in the art will recognize that the present disclosure is also described from the perspective of any individual member or subgroup of a member of the Markush group.

[0186] For all purposes, including providing written explanations, as will be understood by those skilled in the art, all scopes disclosed herein also encompass any possible sub-scopes and combinations of sub-scopes. Any enumerated scope can be readily recognized as sufficiently explainable and enable that the same scope can be broken down into at least equal 1 / 2, 1 / 3, 1 / 4, 1 / 5, 1 / 10, etc. As a non-limiting example, each scope described herein can readily be broken down into the lower third, the middle third, the upper third, etc. Also, as will be understood by those skilled in the art, all words such as “up to,” “at least,” “greater than,” and “less than” include the number mentioned and mean a scope that can be further broken down into sub-scopes as described above. Finally, as will be understood by those skilled in the art, a scope includes each individual element. Thus, for example, a group having 1 to 3 cells refers to a group having 1, 2, or 3 cells. Similarly, a group having 1 to 5 cells refers to a group having 1, 2, 3, 4, or 5 cells, and so on.

[0187] Furthermore, unless otherwise specifically stated, the claims should not be read as being limited to the order or elements provided. In addition, in any claim, the use of the term “means for” is intended to appeal to Section 112, paragraph 6 of the U.S. Patent Act, or the means-plus-function claim format, and no claim without the term “means for” is intended to appeal in that way.

[0188] A software-related processor may be used to implement radio frequency transceivers for use in a radio transceiver unit (WTRU), user equipment (UE), terminal, base station, mobility management entity (MME), or evolved packet core (EPC), or any host computer. A WTRU may be used in conjunction with hardware and / or software-implemented modules such as software-defined radio (SDR), and may be implemented in other components such as cameras, video camera modules, video phones, speakerphones, vibration devices, speakers, microphones, television transceivers, hands-free headsets, keyboards, Bluetooth® modules, frequency modulation (FM) radio units, near-field communication (NFC) modules, LCD display units, organic light-emitting diode (OLED) display units, digital music players, media players, video game player modules, internet browsers, and / or wireless local area network (WLAN) or ultra-wideband (UWB) modules.

[0189] Although the present invention has been described in relation to a communication system, it is intended that the system may be implemented in software on a microprocessor / general-purpose computer (not shown). In certain embodiments, one or more functions of various components may be implemented in software that controls the general-purpose computer.

[0190] In addition, although the present invention is illustrated and described herein with reference to specific embodiments, it is not intended to be limited to the details shown. Rather, various modifications can be made in detail within the scope of the claims and their equivalents, without departing from the present invention.

[0191] Through this disclosure, those skilled in the art will understand that certain representative embodiments may be used in combination with alternative or other representative embodiments.

Claims

1. A method performed on a client device, The process involves receiving a first set of packets from a server, wherein the first set of packets carries video frames of video content. Before decoding and displaying, a first latency value is applied to the first plurality of packets, Sending a request message to the server, wherein the request message includes first information indicating a requested latency value determined based on frame rate fluctuations, Receiving a response message from the server, wherein the response message includes second information indicating a second latency value. Applying a second latency value to a second set of packets before decoding and display, wherein the second set of packets is received following the first set of packets. Methods that include...

2. The method according to claim 1, wherein the first latency value, the requested latency value, and the second latency value are, respectively, the first buffer latency value, the requested buffer latency value, and the second buffer latency value.

3. The method according to claim 1 or 2, wherein applying the first latency value and the second latency value to the first plurality of packets and the second plurality of packets, respectively, comprises (i) extracting the first plurality of packets and the second plurality of packets from a receiver buffer based on the first plurality of packets and the second latency value and (ii) timestamp information contained in the first plurality of packets and the second plurality of packets, respectively, wherein the timestamp information indicates the respective time associated with the storage of the first plurality of packets and the second plurality of packets in the transmitter buffer of the server.

4. The method according to claim 3, wherein the timestamp information contained in any of the first plurality of packets and the second plurality of packets indicates a first time when the packet was stored in the transmitter buffer, the packet is extracted at a second time corresponding to a time amount after the first time, the time amount is constant for consecutive packets and is associated with either the first latency value or the second latency value.

5. The method according to claim 3 or 4, wherein the frame pace variation is a variation in the pace at which the video frame is received, extracted from the receiver buffer, and decoded.

6. The method according to any one of claims 1 to 5, wherein the requested latency value is based on either the number of delayed frames or the number of dropped frames.

7. The method according to any one of claims 1 to 6, wherein the second latency value is applied to the second plurality of packets based on the second latency value being equal to the requested latency value, or based on the second latency value having a value between the requested latency value and the first latency value.

8. The method according to any one of claims 1 to 7, further comprising sending a handshake request message to the server before receiving the first plurality of packets from the server, wherein the handshake request message includes third information indicating the client's ability to support dynamic latency operation.

9. In response to the aforementioned handshake request message, the system receives a handshake response message from the server, The dynamic latency operation is performed on the condition that the handshake response message includes a fourth piece of information indicating the server's ability to support dynamic latency operation. The method according to claim 8, further comprising:

10. Before receiving the first set of packets, a handshake request message is received from the server, A handshake response message is sent to the server, provided that the handshake request message includes third information indicating the server's ability to support dynamic latency operation. Includes, The method according to any one of claims 1 to 7, wherein the handshake response message includes fourth information indicating the client's ability to support dynamic latency operation.

11. A client device comprising a circuit including one of a transmitter, receiver, processor, and memory, The process involves receiving a first set of packets from a server, wherein the first set of packets carries video frames of video content. Before decoding and displaying, a first latency value is applied to the first plurality of packets, Sending a request message to the server, wherein the request message includes first information indicating a requested latency value determined based on frame rate fluctuations, Receiving a response message from the server, wherein the response message includes second information indicating a second latency value. Applying a second latency value to a second set of packets before decoding and display, wherein the second set of packets is received following the first set of packets. A client device configured to perform the following actions.

12. The client device according to claim 11, wherein the first latency value, the requested latency value, and the second latency value are, respectively, the first buffer latency value, the requested buffer latency value, and the second buffer latency value.

13. The client device according to claim 11 or 12, wherein the circuit is configured to apply the first and second latency values ​​to the first and second packets, respectively, by extracting the first and second packets from a receiver buffer based on (i) the first and second latency values ​​and (ii) timestamp information contained in the first and second packets, respectively, the timestamp information indicating the respective time associated with the storage of the first and second packets in the server's transmitter buffer.

14. The client device according to claim 13, wherein the timestamp information contained in any of the first plurality of packets and the second plurality of packets indicates a first time when the packet was stored in the transmitter buffer, the packet is extracted at a second time corresponding to a time amount after the first time, the time amount is constant for consecutive packets and is associated with either the first latency value or the second latency value.

15. The client device according to claim 13 or 14, wherein the frame pace variation is a variation in the pace at which the video frame is received, extracted from the receiver buffer, and decoded.

16. The client device according to claim 11, wherein the requested latency value is further based on either the number of delayed frames or the number of dropped frames.

17. The client device according to any one of claims 11 to 16, wherein the circuit is configured to apply the second latency value to the second plurality of packets based on the second latency value equal to the requested latency value, or based on the second latency value having a value between the requested latency value and the first latency value.

18. The circuit is configured to send a handshake request message to the server before receiving the first plurality of packets from the server. The client device according to any one of claims 11 to 17, wherein the handshake request message includes third information indicating the client's ability to support dynamic latency operation.

19. The circuit receives a handshake response message from the server in response to the handshake request message, The dynamic latency operation is performed on the condition that the handshake response message includes a fourth piece of information indicating the server's ability to support dynamic latency operation. The client device according to claim 18, configured to perform the following:

20. A method implemented on the server, Sending a first plurality of packets carrying a first video frame of video content to a client device, wherein the first plurality of packets remain in a buffer for retransmission for a first amount of time associated with a first latency value. Receiving a request message from the client device, wherein the request message includes first information indicating the requested latency value, Sending a response message to the client device, wherein the response message includes second information indicating a second latency value, The process involves transmitting a second plurality of packets carrying a second video frame of the video content to the client device, wherein the second plurality of packets are transmitted following the first plurality of packets, and the second plurality of packets remain in a buffer for retransmission for a second amount of time associated with the second latency value. Methods that include...

Images