Congestion Control for Low-Latency Interactive Video Streaming

Abstract

The server transmits the encoded game frames as a set of packets over the network to each client system. In response to transmitting the set of packets, the server determines a bandwidth estimate based on a size of the encoded game frames and timing data associated with the transmitted set of packets. The server then compares the bandwidth estimate to a current video bitrate of the game stream it is transmitting from the server to each client device. In response to the comparison indicating network underutilization, the server increases the encoding bitrate. Further, in response to the comparison indicating network overutilization, the server decreases the encoding bitrate.

Description

[Background technology]

[0001] A cloud-based gaming platform executes a video game application, in part or in whole, on a remote server to facilitate playing the video game application on a local client device. The remote server provides audio and video rendered by the execution of the video game application as audio and video streams over a network to the local client device. In providing these audio and video streams, the remote server uses various encoder operations to compress the game frames (which constitute the rendered video) and audio in real-time, which are then streamed. After compressing the game frames and audio, the remote server transmits the compressed game frames and audio as a set of packets over the network to the local client device.

[0002] Unlike common video streaming applications, where some round-trip delay is acceptable in network communication, cloud-based games require very low round-trip delay to maintain interactivity. In addition, cloud-based games require high and stable network throughput due to visual artifacts introduced by video compression techniques. However, both the round-trip delay and throughput of the network may fluctuate due to network congestion. Therefore, to address these requirements, some cloud-based gaming platforms implement congestion control algorithms that adjust pacing rates to remove congestion on the network. However, such congestion control algorithms are not helpful in ensuring low latency and high bandwidth of the game stream under diverse dynamic network conditions such as noisy links and queue-generating cross traffic. Furthermore, any mismatch between the transmission rate of such congestion control algorithms and the bit rate of the encoded video of the game stream will cause sender queuing, and data will be queued at the remote server, adversely affecting the end-to-end frame delay and user experience.

[0003] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of a cloud-based gaming system for single-player or multiplayer (including massively multiplayer) games, according to some embodiments. [Diagram 2] FIG. 1 is a block diagram of a computing device configured to encode and stream frames in a cloud-based gaming system according to some embodiments. [Diagram 3]5 is a flow diagram of an example operation for varying an encoding bit rate to alleviate network congestion in accordance with an example embodiment. [Figure 4] FIG. 2 is a block diagram of a congestion control engine configured to vary an encoding bit rate to alleviate network congestion according to some embodiments. [Diagram 5] 5 is a flow diagram of an example operation for determining an updated bandwidth estimate according to an example embodiment. [Figure 6] 4 is a flow diagram of an example method for varying an encoding bit rate to alleviate network congestion according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The techniques and systems described herein address alleviating congestion on the network by modifying the encoding bitrate of one or more encoders. In particular, such techniques and systems described herein enable the server to modify the encoding bitrate based on an estimated bandwidth of the encoded game frames transmitted by the server to each client system. To this end, the server first transmits the encoded game frames over the network to each client system. In response to transmitting the encoded game frames, the server determines a bandwidth estimate for the transmitted encoded game frames based on a size of the encoded game frames and timing data associated with the transmission of the encoded game frames from the server to each client. The server then compares the bandwidth estimate to a current video bitrate (e.g., average bitrate) of the game stream it is transmitting from the server to each client device. In response to the comparison indicating underutilization of the network (e.g., utilization of less than the total available bandwidth of the network), the server increases the encoding bitrate so that the game stream makes better use of the available bandwidth of the network. Further, in response to the comparison indicating network over-utilization (e.g., utilization of more than the total available bandwidth of the network), the server reduces the encoding bitrate to reduce the network bandwidth used by the game stream and drain any bottleneck queues at the server to help relieve congestion on the network.

[0006] For ease of understanding, the techniques of this disclosure are described in the exemplary context of a cloud gaming system. A cloud-based or other remote server renders a stream of video frames representing the visual content of a video game instance running on that server or an associated server, and then encodes each frame using the multi-trie encoding process described herein to generate a bitstream. The bitstream represents a stream of encoded and rendered game frames for transmission over one or more networks to one or more client devices. However, it should be appreciated that the systems and techniques described herein are not limited to this exemplary context, and may instead be implemented using the guidelines provided herein in any of a variety of video stream encoding / decoding systems, including non-game video stream encoding / decoding systems. Thus, references to "game" or "gaming" may encompass other sources of rendered frame streams, and thus references to "game frames" may equally apply to rendered frames more generally, unless otherwise noted.

[0007] FIG. 1 illustrates a cloud-based gaming system 100 for single-player or multiplayer (including massively multiplayer) gaming, according to some embodiments. The cloud-based gaming system 100 includes one or more servers 102, each of which is communicatively connected to one or more client systems 112 by at least one network 110. The network 110 is configured to enable transmission and reception of data between any of the servers 102 and the client systems 112, and includes wired and wireless networks, such as, for example, Ethernet, Internet, Wi-Fi, or any combination thereof. In an embodiment, each of the servers 102 are communicatively interconnected via a high-bandwidth, low-latency inter-server messaging bus. The servers 102 are typically distributed in one or more data centers across a geographic region to reduce transmission latency due to physical proximity. In the illustrated embodiment, three servers 102-1, 102-2, and 102-3 are presented supporting four client systems 112-1, 112-2, 112-3, and 112-4, although in other embodiments any number of servers may be implemented supporting any number of client devices. Of course, in a typical real-world implementation, the number of servers 102 and the number of client systems 112 will typically be significantly greater than that presented in the example of FIG.

[0008] In the illustrated embodiment, each server 102 includes a computing device, e.g., a cloud server, a virtual server, or any combination thereof, configured to support one or more client game sessions executed on one or more client systems 112. As used herein, a "client game session" encompasses a game application being played, partially simulated, or fully simulated on a client system 112. Each server 102 is configured to support this client game session by executing a corresponding game platform instance 104. The corresponding game platform instance 104 facilitates the execution of at least a portion of the game application being played, partially simulated, or fully simulated on the client system 112. Such facilitation may include performing one or more operations related to the game application, such as rendering one or more frames related to the game application, performing calculations related to the game application (e.g., lighting calculations, artificial intelligence calculations, physics calculations, shading calculations, input calculations, etc.), providing access to files, or any combination thereof, to name a few. The game platform instance 104 provides various software and hardware resources to accomplish such facilitation, such as communication / network management, resource management, media rendering encoding, etc. In this manner, the gaming platform instance 104 simulates the execution of one or more actions of a gaming application for a corresponding player as if that gaming application were being played on a local gaming device, such as a personal computer (“PC”), gaming console, smartphone, tablet computer, automobile entertainment system, or the like.

[0009] Each client system 112 represents hardware and software resources that are utilized to receive player input through the operation of one or more input / output devices of at least one player, and further utilized to present video and audio content that respectively represent the visual and aural content of the gameplay of at least one player. Examples of client systems 112 include one or more desktop computers, notebook computers, tablet computers, virtual reality systems, augmented reality systems, computing-enabled mobile phones (i.e., "smartphones"), computing-enabled televisions (i.e., "smart televisions"), or any combination thereof, to name a few. As shown with reference to client system 112-1, each client system 112 includes one or more client devices 116. In the illustrated embodiment, the client system 112-1 includes a first client device 116-1 that is communicatively connected to or otherwise associated with a display 118, at least one input device 120 (e.g., gamepad, joystick, keyboard, mouse, touch screen), one or more network interfaces configured to connect to a network connecting the client system 112 to a corresponding server 102, one or more processors, memory, storage, speakers, and other computing resources to render, process, and display scenes of a virtual environment. As illustrated with reference to the client system 112-1, each client system 112 may include a decoder 114 configured to decode one or more frames associated with the virtual environment. The decoder 114 may include hardware and software configured to decode one or more encoded streams (e.g., game stream 108) received from the server 102 to generate a set of decoded frames or a decoded stream.Each decoder 114 is configured to decode any encoded frame encoded by any method or process disclosed herein. In an embodiment, client system 112-1 further comprises a smartphone client device 116-2 and a wearable virtual reality client device 116-3. Each of smartphone client device 116-2 and wearable virtual reality client device 116-3 can operate as an integrated mobile computing device having input, output, display, and communication capabilities similar to those described above with respect to client system 112-1. In a particular embodiment, client systems 112-1, 112-2, and 112-3 may include one or more mechanisms, such as an accelerometer, a global positioning system (GPS) device, etc., that are used to obtain motion data representative of the movement of the client device, and the speed or acceleration of such movement.

[0010] Although certain aspects described herein are described with particular reference to cloud gaming scenarios, it should be understood that in certain embodiments, the described techniques can be utilized in a variety of non-gaming scenarios, for example, one or more of the server 102 and client systems 112 can be operative to render, process, and display other types of informational, educational, recreational, and / or artistic content. Thus, it should be further understood that, although techniques are described herein for rendering content that may utilize specific examples related to cloud gaming and gaming content, such descriptions and techniques can also be applied to the non-gaming scenarios described above. Although the examples provided herein may refer to scenarios involving the rendering, processing, and display of gaming content due to certain bandwidth and network latency issues associated with gaming content, they should not be construed as indicating that the described techniques are limited to such scenarios.

[0011] During operation, each server 102 executes a game platform instance 104 for one or more client game sessions. Executing the game platform instance 104 includes rendering a set of game frames, including one or more game frames associated with a game application executing on one or more respective client systems 112. Each rendered game frame represents at least a portion of a virtual environment used by the game application executing on the client system 112. For example, each rendered game frame may represent at least a portion of a virtual environment displayed on a display 118 of the client system 112 during a client game session.

[0012] Each server 102 is configured to encode each rendered game frame via a respective encoder 106 to generate a respective set of encoded frames (also referred to herein as a “game stream” 108). Each server 102 is configured to encode the game stream 108, e.g., through compression, reorganization, and manipulation of each frame rendered by the game platform instance 104. In an embodiment, the encoder 106 of a server 102 implements one or more codecs to encode one or more rendered frames according to the one or more codecs. Such codecs may include H.264, H.265, VP9, ​​AV1, or any combination thereof, to name a few. Each resulting game stream 108 corresponds to a gaming application running on one or more client systems 112 and is provided to these corresponding client systems 112 over the network 110. Each corresponding client system 112 is configured to decode the received game stream 108 via a decoder 114 and display the resulting set of decoded frames 122, e.g., on a display 118. Each client system 112 is configured to decode each game stream 108 by compressing, reorganizing, and manipulating frames in the encoded stream according to one or more different video codecs, including lossless and lossy codecs. According to an embodiment, each client system 112 includes a decoder that implements one or more codecs and decodes the received game stream 108 according to the one or more codecs. Such codecs may include H.264, H.265, VP9, ​​AV1, or any combination thereof, to name a few. Although the illustrated embodiment shows three game streams 108-1, 108-2, 108-3, in other embodiments, the server 102 may generate any number of game streams 108, each corresponding to one or more client game sessions.

[0013] In an embodiment, each server 102 is configured to transmit each encoded game frame of the game stream 108 to each client system 112 as a set of packets. That is, each server 102 transmits a set of packets representing the encoded game frame to each client system 112. To mitigate congestion on the network 110 resulting from the game stream 108 (e.g., a set of packets representing encoded game frames) transmitted from one or more servers 102 to corresponding client systems 112, each server 102 includes a respective congestion control engine 124. The congestion control engine 124 includes hardware and software configured to modify the encoder bitrate of each encoder 106 in response to under-utilization of the network 110 by the game stream 108, over-utilization of the network 110, or both. For example, in response to under-utilization of the network 110 (e.g., utilization of less than a predetermined threshold amount of the bandwidth of the network 110), the congestion control engine 124 increases the encoder bitrate of each encoder 106. As another example, in response to over-utilization of the network 110 (e.g., utilization of the network 110 bandwidth above a predetermined threshold amount), the congestion control engine 124 reduces the encoder bit rate of each encoder 106. In this manner, the congestion control engine 124 reduces the encoding bit rate of each encoder 106 in response to congestion on the network 110 (e.g., over-utilization of the network 110). Reducing the encoding bit rate in this manner helps to drain any bottleneck queues caused by previously encoded frames, reducing the bandwidth of the game stream 108 and helping to alleviate congestion on the network 110. To determine under-utilization of the network 110 by the game stream 108, over-utilization of the network 110, or both, each congestion control engine 124 is configured to determine a bandwidth estimate of the network 110 (e.g., an estimate of the amount of bandwidth currently available on the network 110).

[0014] To determine the bandwidth estimate, the congestion control engine 124 first determines a bandwidth sample value for the network 110 (e.g., a sample value of the amount of available bandwidth for the network 110) based on, for example, interval data associated with the transmission of the encoded game frame. The interval data includes, for example, data representing a time interval between the transmission of the encoded game frame from the server 102 and the complete reception at each client system 112. As an example, the interval data includes a size of the encoded frame and an elapsed time between the transmission of the encoded game frame from the server 102 and the complete reception of the encoded game frame by each client system 112 (e.g., the time elapsed between when a first transmitted packet associated with the encoded game frame is transmitted by the server and when a last transmitted packet associated with the encoded game frame is received by each client system 112). In an embodiment, the congestion control engine 124 determines the bandwidth sample value from the interval data, for example, by dividing the frame size of the encoded frame by the elapsed time between the transmission of the encoded game frame from the server 102 and the complete reception of the encoded game frame by each client system 112. According to an embodiment, after determining the bandwidth sample value, the congestion control engine 124 smoothes the bandwidth sample value based on one or more previous bandwidth estimates to generate an updated bandwidth estimate. For example, the congestion control engine 124 uses the bandwidth sample value to update a current bandwidth estimate (e.g., a predetermined default bandwidth estimate or a previously determined bandwidth estimate) to generate an updated bandwidth estimate. In some embodiments, the bandwidth estimate determined by the congestion control engine 124 includes the bandwidth sample value, while in other embodiments, the bandwidth estimate determined by the congestion control engine includes the updated bandwidth estimate (e.g., the smoothed bandwidth sample value).In response to determining the bandwidth estimate (e.g., bandwidth sample value, updated bandwidth estimate), the congestion control engine 124 is configured to determine under-utilization of the network 110, over-utilization of the network 110, or both, e.g., by comparing the bandwidth estimate to a current video bitrate, which represents an average bitrate at which the game stream 108 is transmitted from the server 102 to each client system 112. As an example, in response to the bandwidth estimate being smaller than the current video bitrate, the congestion control engine 124 determines that there is over-utilization of the network because the current video bitrate of the game stream 108 exceeds the estimated available bandwidth of the network 110. Similarly, in response to the bandwidth estimate being larger than the current video bitrate, for example, the congestion control engine 124 determines that there is under-utilization of the network because the current video bitrate of the game stream 108 is below a predetermined threshold utilization of the estimated available bandwidth of the network 110.

[0015] In an embodiment, each congestion control engine 124 is also configured to set a pacing rate for one or more encoded game frames of the game stream 108. The pacing rate may, for example, include a value indicative of a maximum bandwidth for transmitting packets associated with each encoded game frame over the network 110 (e.g., a value indicative of a rate for transmitting a set of packets associated with the encoded game frames). That is, each congestion control engine 124 sets a pacing rate for packets of the game stream 108 transmitted from the server 102 to a corresponding client system 112. According to an embodiment, each congestion control engine 124 is configured to set a pacing rate for each encoder 106 based on a determined bandwidth estimate (e.g., a bandwidth sample value, an updated bandwidth estimate), for example, based on the transmission of the encoded game frames from the server 102 to each client system 112. For example, the congestion control engine 124 is configured to set the pacing rate as a multiple of the bandwidth estimate. In this manner, the congestion control engine 124 helps to mitigate congestion of the network 110 due to a pacing rate that does not utilize the network 110.

[0016] Referring now to FIG. 2, a computing device 200 is shown. The computing device 200 is configured to encode and stream frames in a cloud-based gaming system. In an embodiment, the computing device 200 implements aspects of the cloud-based gaming system 100 as shown in FIG. 1. For example, the computing device 200 may be similar to or the same as the server 102 shown in FIG. 1. The computing device 200 includes one or more software and hardware components for two-way communication, including components for encoding a set of game frames 250, such as to generate a game stream 108. In some embodiments, the computing device 200 is part of an electronic device that supports encoding a set of game frames 250. The electronic device may include, for example, a desktop computer, a notebook computer, a tablet, a server, or a game console, to name a few. In an embodiment, the computing device 200 includes a processor 226, a modem 228, and a memory 230. The memory 230 includes an electronic storage device, such as, for example, a solid state drive, a hard disk drive, a random access memory ("RAM"), a read only memory ("ROM"), an electronically erasable programmable ROM ("EEPROM"), an optical storage device, or any combination thereof. The memory 230 includes instructions and data (e.g., codecs, reference frames, game engines, game applications, constants, etc.) related to the operation of the game platform instance 204, the encoder 206, the congestion control engine 224, the rate control engine 232, and the transport layer 234. The modem 228 is configured to be communicatively coupled to one or more client systems 112 via the network 110, and is further configured to transmit the game stream 108 to the one or more client systems 112 using the network 110.According to an embodiment, the processor 226, modem 228, and memory 230 are in internal electronic communication via one or more interfaces (eg, bus 244).

[0017] According to the present embodiment, the processor 226 includes one or more control processing units ("CPU"), microprocessors, field programmable gate arrays ("FPGA"), graphics processing units ("GPU"), application specific integrated circuits (ASICs), or any combination thereof, and is configured to render and encode game frames for use in client game sessions on the cloud-based gaming system 100. The processor 226 operates to execute a game platform instance 204. The game platform instance 204 is the same as or similar to the game platform instance 104 and is configured to support game applications associated with a current client game session and executing on one or more client systems 112. The game platform instance 204 includes graphics hardware and software (not shown for clarity) for rendering a set of game frames 250 associated with the environment of the game application executing on one or more client devices. Such graphics hardware and software may include, for example, a graphics core, processing cores, pixel shaders, video random access memory ("VRAM"), GPU, physics engine, lighting engine, tessellation engine, etc. Each rendered game frame in the set of game frames 250 represents at least a portion of a virtual environment associated with a game application executing on a client device. For example, if the game application is a racing game, each game frame in the set of game frames 250 represents at least a portion of a race track, a car, or a surrounding area.

[0018] Each rendered game frame of the set of game frames 250 is provided to the encoder 206 for encoding into the game stream 108. The encoder 206 includes hardware and software configured to encode one more game frame 250 to generate the game stream 108, for example, according to interframe and intraframe techniques. To reduce the bandwidth required to transmit the game stream 108 between the computing device 200 and the one or more client systems 112, the encoder 206 encodes the game stream 108 by compressing one or more game frames of the set of game frames 250. Compressing the game frame includes comparing the game frame to one or more reference frames stored in the memory 230 and encoding one or more of the differences between the game frame and the one or more reference frames into the game stream 108. The encoder 206 is further configured to encode the used reference frames into the game stream 108. According to an embodiment, in response to the encoder 206 encoding a game frame to generate an encoded game frame, the encoder 206 is configured to provide the encoded game frame to the transport layer 234 for transmission to one or more client systems 112. The transport layer 234 includes hardware and software configured to packetize and transmit the one or more encoded game frames to the one or more client systems 112 over the network 110. For example, the transport layer 234 is configured to generate a set of packets representing the encoded game frame and transmit the set of packets to the client system 112. In an embodiment, in response to packetizing and transmitting the encoded game frame to the client system 112, the transport layer 234 is configured to generate a set of transmission timestamps.The set of transmission timestamps includes, for example, data indicating a time (e.g., year, month, week, day, hour, minute, second, millisecond, or any combination thereof) at which each packet of the set of packets generated from the encoded game frame is transmitted by the transport layer 234. Further, the transport layer 234 is configured to receive a set of reception timestamps from one or more client systems 112 in response to transmitting the set of packets to the client system 112. The set of reception timestamps includes data indicating a time (e.g., year, month, week, day, hour, minute, second, millisecond, or any combination thereof) at which each packet of the set of packets generated from the encoded game frame is received from the transport layer 234 by each client system 112. That is, in response to receiving a packet of the set of packets transmitted by the transport layer 234, each client system 112 is configured to transmit an acknowledgment to the transport layer 234 including a timestamp indicating that the packet was received by each client system 112.

[0019] To facilitate alleviating congestion on the network 110 caused by the game stream 108, the processor 226 includes a congestion control engine 224. The congestion control engine 224 is configured to help reduce congestion on the network 110 by modifying an encoding bit rate used by the encoder 106 to encode the game frames 250. In an embodiment, the congestion control engine 224 is configured to determine a bandwidth sample value (e.g., an amount of bandwidth available on the network 110) for the network 110 based on the encoded frames being transmitted from the transport layer 234 to each client system 112. The bandwidth sample value represents an amount of available bandwidth for the network 110 based on, for example, end-to-end transport of the encoded game frames from the transport layer 234 to each client system 112. To determine the bandwidth sample value, the congestion control engine 224 uses timing data and frame information associated with the transmitted encoded game frames. Such timing data may include, for example, a set of transmit timestamps generated by the transport layer 234 in response to transmitting a set of packets of the encoded game frame to each client system 112, and a set of receive timestamps received by the transport layer 234 in response to receiving the packets of the set of encoded game frames at each client system 112. Further, such frame data may include, for example, a size (e.g., in bits) of the encoded game frame. Based on the frame data and the timing data associated with the transmitted encoded game frames, the congestion control engine 224 may be configured to determine a bandwidth sample value, for example, according to the following equation:

number

[0020] In an embodiment, the congestion control engine 224 is further configured to take into account undershoots (e.g., a game stream 108 that does not utilize all of the available bandwidth on the network 110) when determining the bandwidth sample value. Such undershoots may occur, for example, due to a traditional rate control mechanism (e.g., implemented by the rate control engine 232) that acts as a bridge between the bandwidth sample value and the encoder bitrate and reduces the frequency of encoder bitrate updates, due to an application-specific maximum encoder bitrate setting, due to encoder undershoots when the video data is not complex enough to guarantee encoding of a frame at the required full target bitrate (e.g., low-motion scenes), or any combination thereof. To take undershoots into account, the congestion control engine 224 is configured to determine the bandwidth sample value, for example, according to the following formula:

number

number

[0021] In an embodiment, the congestion control engine 224 is configured to smooth the bandwidth sample values, for example, by using the bandwidth sample values ​​to determine an updated bandwidth estimate. For example, based on the bandwidth sample values ​​(e.g., S), the congestion control engine 224 is configured to update a current bandwidth estimate for the game stream 108 being transmitted by the transport layer 234. The current bandwidth estimate includes, for example, a predetermined default bandwidth estimate or a previously determined bandwidth estimate associated with the processing device 200. According to an embodiment, the congestion control engine 224 is configured to update the current bandwidth estimate, for example, according to the following formula:

number

[0022] According to an embodiment, the congestion control engine 224 is configured to provide the bandwidth sample value, the updated bandwidth estimate, or both to the rate control engine 232. The rate control engine 232 includes hardware and software configured to determine, set, and modify the encoding bit rate of the encoder 206. In some embodiments, the rate control engine 232 is included in the congestion control engine 224, while in other embodiments, the rate control engine 232 is separate from the congestion control engine 224. According to an embodiment, the rate control engine 232 is configured to modify the encoding bit rate of the encoder 206 based on a comparison of the bandwidth sample value, the updated bandwidth estimate, or both, to a current video bit rate of the game stream 108. For example, the current video bit rate represents an average bit rate at which the game stream 108 associated with the bandwidth sample value, the updated bandwidth estimate, or both, is transmitted by the transport layer 234 to each client system 112. In an embodiment, the congestion control engine 224, the rate control engine 232, or both, are configured to determine the current video bit rate, for example, according to the following formula:

number

[0023] According to an embodiment, the congestion control engine 224 is further configured to determine a pacing rate for packets transmitted by the transport layer 234. That is, the congestion control engine 224 determines a rate at which a set of packets of an encoded game frame is transmitted from the transport layer 234 to each client system 112. The congestion control engine 224 is configured to determine the pacing rate, for example, based on the determined updated bandwidth estimate. As an example, the congestion control engine 224 is configured to set the pacing rate as a multiple of the determined updated bandwidth estimate. Setting the pacing rate in this manner allows packets to be more distributed among other flows within the encoded game frame transmitted by the transport layer 234 while increasing the utilization of the network 110.

[0024] 3, an exemplary operation 300 for modifying an encoding bit rate to mitigate network congestion is presented. The operation 300 includes an encoder 306, similar to or the same as the encoder 106, 206, receiving a game frame 305, for example, from the set of game frames 250. In response to receiving the game frame 305, the encoder 306 encodes the game frame 305 to generate an encoded game frame 310. The encoder 306 further generates frame data 315 associated with the encoded game frame 310, for example, including a size of the encoded game frame 310. After encoding the game frame 305, the encoder 306 provides the encoded game frame 310 to a transport layer 334 and provides the frame data 315 to a congestion control engine 324. The operation 300 further includes a transport layer 334, similar to or the same as the transport layer 234, transmitting the encoded game frame 310 as a set of packets to each client system 112. In response to transmitting the encoded game frames 310 as a set of packets, the transport layer 334 generates a set of transmit timestamps, where each transmit timestamp in the set of transmit timestamps represents a time (e.g., year, month, week, day, hour, minute, second, millisecond) that each packet of the set of packets was transmitted by the transport layer 334 to each client system 112. Further, in response to transmitting the encoded game frames 310 as a set of packets, the transport layer 334 receives a set of receive timestamps from each client system 112. Each receive timestamp in the set of receive timestamps represents a time (e.g., year, month, week, day, hour, minute, second, millisecond) that each packet of the set of packets was received from the transport layer 334 by each client system 112.

[0025] In an embodiment, the operations 300 include the transport layer 334 providing timing data 320 to a congestion control engine 324, similar or the same as the congestion control engine 124, 224. The timing data 320 includes, for example, a set of transmit timestamps and a set of receive timestamps associated with the transmission of the encoded game frames 310. In response to receiving the timing data 320, the congestion control engine 324 determines an updated bandwidth estimate 330 (e.g., an amount of available bandwidth of the network 110). To determine the updated bandwidth estimate 330, the congestion control engine 324 determines a bandwidth sample value based on the frame data 315, the timing data 320, or both, according to any equation disclosed herein. After determining the bandwidth sample value, the congestion control engine 324 updates a current bandwidth estimate (e.g., a predetermined default bandwidth estimate, a previously determined bandwidth estimate) based on the bandwidth sample value to generate the updated bandwidth estimate 330. For example, the congestion control engine 324 updates the current bandwidth estimate based on the determined bandwidth sample value and according to any of the formulas disclosed herein to generate the updated bandwidth estimate 330 .

[0026] The operations 300 further include the congestion control engine 324 providing the updated bandwidth estimate 330 to a rate control engine 332 similar or the same as the rate control engine 232. In response to receiving the updated bandwidth estimate 330, the rate control engine 332 compares the updated bandwidth estimate 330 to a current video bitrate, which represents, for example, an average rate at which the game stream 108 associated with the updated bandwidth estimate 330 is transmitted by the transport layer 334 to each client system 112. Based on a comparison of the updated bandwidth estimate 330 to the current video bitrate, the rate control engine 332 is configured to modify an encoding bitrate of the encoder 306 to generate an encoding bitrate 335. For example, in response to the updated bandwidth estimate 330 being greater than the current video bitrate (e.g., indicating low utilization of the network 110), the rate control engine 332 is configured to increase an encoding bitrate of the encoder 306 to generate an encoding bitrate 335. As another example, in response to the updated bandwidth estimate 330 being less than the current video bitrate (e.g., indicating over-utilization of the network 110), the rate control engine 332 is configured to decrease the encoding bitrate of the encoder 306 to generate an encoding bitrate 335. Altering the bitrate of the encoder 306 in this manner helps to alleviate congestion on the network 110 transmitting the game stream 108 and also helps to prevent under-utilization of the network 110. After generating the encoding bitrate 335, the rate control engine 332 provides the encoding bitrate 335 to the encoder 306.

[0027] In an embodiment, the operations 300 further include the congestion control engine 324 determining a pacing rate 325 based on the updated bandwidth estimate 330. The pacing rate 325 represents, for example, a rate at which a set of packets of encoded game frames are transmitted from the transport layer 334 to each client system 112. According to an embodiment, the congestion control engine 324 determines the pacing rate 325 as a multiple of the updated bandwidth estimate 330. That is, the congestion control engine 324 multiplies the updated bandwidth estimate 330 by a predetermined constant value (e.g., 2) to determine the pacing rate 325. After determining the pacing rate 325, the congestion control engine 324 provides the pacing rate 325 to the transport layer 334.

[0028] 4, a congestion control engine 400 is presented. The congestion control engine 400 is similar or the same as the congestion control engine 124, 224, 324 and is configured to modify the encoder bit rate to alleviate network congestion. The congestion control engine 400 includes a spacing tracking engine 436, which includes hardware and software configured to determine spacing data (e.g., data representing a time interval required to transmit an encoded game frame) for encoded game frames transmitted by a transport layer similar or the same as the transport layer 234, 334. The spacing data for an encoded game frame includes, for example, a frame size of the encoded game frame, a time when the encoded game frame was first transmitted, and a time when the frame was completely received by each client system 112. In an embodiment, the spacing tracking engine 436 is configured to determine spacing data for an encoded game frame based on frame data similar or the same as the frame data 315 and timing data similar or the same as the timing data 320. For example, the interval tracking engine 436 is configured to determine interval data for the encoded game frames based on a frame size, a set of transmit timestamps, and a set of receive timestamps associated with the encoded game frames. As another example, the interval tracking engine 436 is configured to determine a frame size based on the frame data, determine a time the encoded game frame was first transmitted based on the set of transmit timestamps, and determine a time the frame was completely received by each client system 112 based on the set of receive timestamps. In response to determining the interval data for the encoded game frames, the interval tracking engine 436 is configured to provide the interval data to a bandwidth sampling engine 438.

[0029] The bandwidth sampling engine 438 includes hardware and software configured to determine a bandwidth sample value for the encoded game frame based on the interval data. For example, the bandwidth sampling engine 438 is configured to determine a bandwidth sample value based on a frame size of the encoded game frame, a time when the encoded game frame was first transmitted, and a time when the frame was completely received by each client system 112, according to any formula disclosed herein. In response to determining the bandwidth sample value, the bandwidth sampling engine 438 is configured to provide the bandwidth sample value to the bandwidth estimation engine 440. The bandwidth estimation engine 440 includes hardware and software configured to update a current bandwidth estimate (e.g., a predetermined default bandwidth estimate, a previously determined bandwidth estimate) based on the bandwidth sample to generate an updated bandwidth estimate. For example, the bandwidth estimation engine 440 updates the current bandwidth estimate based on the bandwidth sample value determined by the bandwidth sampling engine 438, according to any formula disclosed herein to generate an updated bandwidth estimate. In response to generating the updated bandwidth estimate, the bandwidth estimation engine 440 is configured to provide the updated bandwidth estimate to the rate control engine 432 and the pacing control engine 442 .

[0030] The rate control engine 432 is similar or the same as the rate control engine 232, 332 and is configured to modify an encoding bit rate of an encoder similar or the same as the encoder 106, 206, 306 based on a comparison of the updated bandwidth estimate and a current video bit rate. For example, the current video bit rate represents an average rate at which the game stream 108 associated with the updated bandwidth estimate is transmitted by the transport layer to each client system 112. As an example, in response to the updated bandwidth estimate being greater than the current video bit rate (e.g., indicating underutilization of the network 110), the rate control engine 432 is configured to increase the encoding bit rate of the encoder to generate a certain encoding bit rate. As another example, in response to the updated bandwidth estimate being less than the current video bit rate (e.g., indicating overutilization of the network 110), the rate control engine 432 is configured to decrease the encoding bit rate of the encoder to generate a certain encoding bit rate. The congestion control engine 400 further includes a pacing control engine 442. Pacing control engine 442 includes hardware and software configured to set a pacing rate for the transport layer. The pacing rate represents, for example, the rate at which sets of packets of encoded game frames are transmitted from the transport layer to each client system 112. In an embodiment, pacing control engine 442 is configured to determine the pacing rate based on an updated bandwidth estimate generated by bandwidth estimation engine 440. For example, pacing control engine 442 determines the pacing rate as a multiple of the updated bandwidth estimate generated by bandwidth estimation engine 440.

[0031] 5, an exemplary operation 500 for determining an updated bandwidth estimate is presented. In an embodiment, the operation 500 includes an interval tracking engine 536, similar or the same as the interval tracking engine 436, receiving frame data 505, similar or the same as the frame data 315, and timing data 510, similar or the same as the timing data 320. After receiving the frame data 505 and the timing data 510, the interval tracking engine 536 determines interval data 515 for the encoded game frames based on the frame data 505 and the timing data 510. The interval data 515 for the encoded game frames includes, for example, a frame size of the encoded game frames, a time when the encoded game frames were initially transmitted, and a time when the frames were completely received by each client system 112. As an example, the interval tracking engine 536 determines the interval data 515 for the encoded game frames based on the frame size from the frame data 505, a set of transmission timestamps from the timing data 510, and a set of reception timestamps from the timing data 510. In response to determining the interval data 515 for the encoded game frames, the interval tracking engine 536 provides the interval data to a bandwidth sampling engine 538, which may be similar or the same as the bandwidth sampling engine 438. In response to receiving the interval data 515, the bandwidth sampling engine 538 determines a bandwidth sample value 520 based on the interval data 515. For example, the bandwidth sampling engine 538 determines the bandwidth sample value 520 according to any of the formulas disclosed herein based on the frame size of the encoded game frames as indicated by the interval data 515, the time the encoded game frames were first transmitted, and the time the frames were completely received by each client system 112. After determining the bandwidth sample value 520, the bandwidth sampling engine 538 provides the bandwidth sample value 520 to a bandwidth estimation engine 540.

[0032] The bandwidth estimation engine 540 is similar or the same as the bandwidth estimation engine 440 and is configured to update a current bandwidth estimate (e.g., a predetermined default bandwidth estimate, a previously determined bandwidth estimate) based on the bandwidth sample values ​​520 to generate an updated bandwidth estimate 525. For example, the bandwidth estimation engine 540 updates the current bandwidth estimate based on the bandwidth sample values ​​520 according to any equation disclosed herein to generate an updated bandwidth estimate 525. In response to generating the updated bandwidth estimate 525, the bandwidth estimation engine 540 is configured to provide the updated bandwidth estimate 525 to the rate control engine 532 and the pacing control engine 542. In response to receiving the updated bandwidth estimate 525, a rate control engine 532, similar to or the same as the rate control engine 232, 332, 432, modifies an encoding bit rate of an encoder, similar to or the same as the encoder 106, 206, 306, based on a comparison of the updated bandwidth estimate 525 and the current video bit rate to generate an encoding bit rate 535 similar to or the same as the encoding bit rate 335. As an example, in response to the updated bandwidth estimate 525 being greater than the current video bit rate, the rate control engine 532 increases the encoding bit rate of the encoder to generate the encoding bit rate 535. As another example, in response to the updated bandwidth estimate 525 being less than the current video bit rate, the rate control engine 532 decreases the encoding bit rate of the encoder to generate the encoding bit rate 535.

[0033] According to an embodiment, operations 500 further include a pacing control engine 542, similar or the same as pacing engine 442, determining a pacing rate 530 for the transport layer. Pacing rate 530 represents, for example, a rate at which sets of packets of encoded game frames are transmitted from the transport layer to each client system 112. In an embodiment, pacing control engine 542 determines pacing rate 530 based on updated bandwidth estimate 525. For example, pacing control engine 542 determines pacing rate 530 as a multiple of updated bandwidth estimate 525.

[0034] 6, an exemplary operation 600 for varying an encoding bit rate to mitigate network congestion is presented. At step 605, an encoder similar to or the same as the encoder 106, 206, 306 encodes the game frames to generate encoded game frames. At step 610, a transport layer similar to or the same as the transport layer 234, 334 transmits the encoded game frames as a set of packets to each client system 112 over the network 110. Further, at step 610, the transport layer determines a set of transmit timestamps associated with the set of packets transmitted to each client system 112, and receives a set of receive timestamps associated with the set of packets received by each client system 112 from the transport layer. At step 615, the congestion control engine, similar to the congestion control engine 124, 224, 324, 400, determines a bandwidth estimate (e.g., a bandwidth sample value, an updated bandwidth estimate) based on the size of the transmitted encoded game frame and timing data associated with the transmitted set of packets of the encoded game frame. Such timing data includes, for example, a set of transmit timestamps and a set of receive timestamps. As one example, the congestion control engine determines a bandwidth sample value for the transmitted encoded game frame based on the frame size of the encoded game frame, the set of transmit timestamps, and the set of receive timestamps according to any equation disclosed herein. As another example, the congestion control engine determines an updated bandwidth estimate based on the bandwidth sample value determined according to any equation disclosed herein.

[0035] In step 620, the rate control engine, similar to the rate control engine 232, 332, 432, 532, modifies the encoder encoding bit rate based on the bandwidth estimate (e.g., bandwidth sample value, updated bandwidth estimate). For example, the rate control engine modifies the encoder encoding bit rate based on a comparison of the bandwidth estimate to a current video bit rate. The current video bit rate represents, for example, an average rate at which the game stream 108 associated with the bandwidth estimate is transmitted by the transport layer to each client system 112. In response to the bandwidth estimate being less than the current video bit rate, the rate control engine decreases the encoder encoding bit rate. In response to the bandwidth estimate being greater than the current video bit rate, the rate control engine increases the encoder encoding bit rate. Modifying the encoder bit rate in this manner helps to alleviate congestion on the network 110 transmitting the game stream 108 and also helps to prevent underutilization of the network 110.

[0036] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, cause the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices (e.g., flash memory, cache memory, random access memory (RAM) or other one or more non-volatile memory devices), and the like. The executable instructions stored in the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by the one or more processors.

[0037] A computer-readable storage medium may include any storage medium or combination of storage media that is accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache memory), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. A computer-readable storage medium may be incorporated into a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or a universal serial bus (USB)-based flash memory), or connected to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0038] It should be noted that not all of the operations or elements described above in the general description are required, some of the particular operations or devices may not be required, and one or more additional operations may be performed or elements may be included in addition to those described above. Furthermore, the order in which the operations are listed is not necessarily the order in which they are performed. Additionally, the concepts have been described with reference to specific embodiments. However, it will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope of the disclosure as set forth in the claims below. Thus, the specification and drawings should be regarded in an illustrative rather than restrictive sense, and all such modifications are intended to be included within the scope of the disclosure.

[0039] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, and solutions to problems, as well as any feature or features that may cause or make any benefit, advantage, or solution more pronounced, should not be construed as critical, necessary, or essential features of any or all of the claims. Moreover, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in equivalent and different manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design shown herein, other than as set forth in the claims below. It will thus be apparent that the particular embodiments disclosed above may be altered or modified, and such variations are deemed to be within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.

Claims

1. 1. A computer-implemented method for cloud-based gaming, comprising: in response to the encoder encoding game frames in the stream of game frames according to an encoding bit rate to generate encoded game frames, transmitting the encoded game frames as a set of packets over a network; determining a bandwidth estimate based on timing data associated with the set of packets transmitted over the network; and varying the encoding bit rate based on the bandwidth estimate.

2. The method of claim 1 , wherein the timing data comprises a set of transmit timestamps and a set of receive timestamps associated with the set of packets.

3. Changing the encoding bit rate comprises: The method of claim 1 or 2, comprising decreasing the encoding bit rate in response to the bandwidth estimate being lower than a current video bit rate.

4. Changing the encoding bit rate comprises: A method according to any preceding claim, comprising increasing the encoding bitrate in response to the bandwidth estimate being higher than a current video bitrate.

5. Determining the bandwidth estimate further comprises: determining interval data representing a time interval for transmitting the transmitted encoded game frames based on the timing data; and determining said bandwidth estimate based on said interval data.

6. Determining the bandwidth estimate further comprises: The method of any one of claims 1 to 5, comprising smoothing the bandwidth estimate based on a step size.

7. determining a pacing rate based on the bandwidth estimate; 7. The method of claim 1, further comprising: in response to the encoder encoding a second game frame of the stream of game frames to generate a second encoded game frame, transmitting the second encoded game frame as a second set of packets over the network in accordance with the pacing rate.

8. Transmitting the encoded game frame comprises: transmitting the set of packets over the network to a client system associated with the current client game session; A method according to any preceding claim, comprising receiving from the client system a set of receipt timestamps associated with said set of packets.

9. 9. The method of claim 8, further comprising: decoding, at the client system, the encoded game frames to generate decoded game frames, the decoded game frames representing at least a portion of a virtual environment associated with the current client game session.

10. The method of claim 8 , further comprising: rendering the game frames based on a game application associated with the current client game session.

11. 1. A system for cloud-based gaming, comprising: an encoder configured to encode game frames in the stream of game frames according to an encoding bit rate to generate encoded game frames; a transport layer configured to transmit the encoded game frames as a set of packets over a network; a congestion control engine, the congestion control engine comprising: determining a bandwidth estimate based on timing data associated with the set of packets transmitted over the network; The system is configured to vary the encoding bit rate of the encoder based on the bandwidth estimate.

12. 12. The system of claim 11, wherein the timing data comprises a set of transmit timestamps and a set of receive timestamps associated with the set of packets.

13. The congestion control engine further comprises: The system of claim 11 or 12, configured to reduce the encoding bit rate of the encoder in response to the bandwidth estimate being lower than a current video bit rate.

14. The congestion control engine further comprises: A system according to any one of claims 11 to 13, configured to increase the encoding bitrate of the encoder in response to the bandwidth estimate being higher than a current video bitrate.

15. The congestion control engine further comprises: determining interval data representing a time interval for transmitting the transmitted encoded game frames based on the timing data; and determining the bandwidth estimate based on the interval data.

16. The congestion control engine further comprises: The system of any one of claims 11 to 15, configured to smooth the bandwidth estimate based on a step size.

17. The congestion control engine is further configured to determine a pacing rate based on the bandwidth estimate; 17. The system of claim 11, wherein the transport layer is further configured, in response to the encoder encoding a second game frame of the stream of game frames to generate a second encoded game frame, to transmit the second encoded game frame as a second set of packets over the network in accordance with the pacing rate.

18. a client system associated with the current client game session, the transport layer further comprising: transmitting said set of packets to said client system; A system according to any one of claims 11 to 17, configured to receive from the client system a set of receipt timestamps associated with the set of packets.

19. The client system includes:

20. The system of claim 18, configured to decode the encoded game frames to generate decoded game frames, the decoded game frames representing at least a portion of a virtual environment associated with the current client game session.

20. 1. A system for cloud-based gaming, comprising: a cloud-based game server configured to render a set of game frames based on a current client game session, the cloud-based game server comprising: an encoder configured to encode game frames of a set of game frames in a stream of game frames according to an encoding bit rate to generate encoded game frames; a transport layer configured to transmit the encoded game frames as a set of packets over a network to each client system based on the current client game session; a congestion control engine, the congestion control engine comprising: determining a bandwidth estimate based on timing data associated with the set of packets transmitted over the network to each of the client systems; The system is configured to vary the encoding bit rate of the encoder based on the bandwidth estimate.

21. 1. A system comprising: an encoder configured to encode frames of the stream of frames according to an encoding bit rate to generate encoded frames; a transport layer configured to transmit the encoded frames as a set of packets over a network; a congestion control engine, the congestion control engine comprising: determining a bandwidth estimate based on timing data associated with the set of packets transmitted over the network; The system is configured to vary the encoding bit rate of the encoder based on the bandwidth estimate.

Images