Content-adaptive signal-to-noise ratio guided two-pass video coding for cloud gaming

Abstract

A technique for encoding game content for real-time cloud gaming, more specifically, uses an adaptive statistical model over a moving window of previous frames up to the current frame to adaptively calculate a quantization parameter (QP) lower bound corresponding to a selected peak signal-to-noise ratio (PSNR) target. The result of this PSNR-guided encoding is a significant reduction in the size of the encoded frames (i.e., below the target bitrate when there is no subjective quality loss for those frames, using their PSNR as guidance). These reductions alleviate bandwidth issues and improve the overall user experience, as fewer bits transferred means faster network transmission and decoding and less load on the platform.

Description

[Background technology]

[0001] background Various cloud gaming platforms support the use of encoding operations to compress game content in real time into a video stream that is delivered to users' devices over the Internet. Typically, an encoder uses a quantization parameter (QP) to control how the game content is encoded, particularly how much information is preserved during the encoding operation. Some encoders select a QP for each frame (also called a media frame, video frame, image, etc.) depending on the size of the frame. Thus, these encoders select a QP to meet a target bitrate (i.e., frame size). However, these encoders result in encoding operations on the game content that cause additional and unnecessary bandwidth usage for the encoder (or devices associated with the encoder) and, in some cases, increase end-to-end latency, thereby negatively impacting the user experience. Summary of the Invention [Problem to be solved by the invention]

[0002] overview This disclosure particularly relates to encoding media content within frames based on a determined Peak Signal to Noise Ratio (PSNR). For example, a proposed solution may include using an adaptive statistical model over a moving window of previous frames up to the current frame to adaptively calculate a lower bound on the QP corresponding to a PSNR target. The result of the corresponding PSNR-guided encoding may be a significant reduction in the size of the encoded frames (i.e., below the target bitrate when there is no subjective quality loss for those frames, using their PSNR as guidance). These reductions alleviate bandwidth issues and improve the overall user experience, as fewer bits transferred means faster network transmission and decoding and less platform load. [Means for solving the problem]

[0003] According to one aspect, a method for encoding video for real-time cloud gaming comprises determining a PSNR associated with encoding a downscaled version of each frame of a stream of frames in a first encoding pass, and encoding each frame of the stream of frames in a second encoding pass based on the PSNR in response to the first encoding pass.

[0004] The method may also include determining, based on a statistical linear regression model, a minimum QP associated with the selected target PSNR for encoding each frame of the stream of frames. In some embodiments, encoding the frames of the stream of frames is based on a minimum QP for the encoded frames to meet the selected target PSNR.

[0005] The method may also include determining a linear mapping from PSNR to a minimum QP for each frame by fitting a linear curve to PSNR and QP pairs in a moving window of previous frames in the deque. In some embodiments, the linear mapping is updated after the second encoding pass of each frame.

[0006] The method may also include determining a minimum QP floor value based on the selected target PSNR and the updated linear mapping for each frame comprising the stream of frames.

[0007] The method may also include determining an intersection point and a slope of a linear mapping from PSNR to a minimum QP based on a moving window of a previous frame.

[0008] The method may also include maintaining a moving window size of the previous frame associated with the PSNR and QP pair.

[0009] The method may also include updating the statistical model based on a PSNR associated with each frame of the stream of frames. Encoding each frame of the stream of frames is based on a minimum QP from the statistical model.

[0010] The method may also include calculating, by the encoder, a PSNR associated with each frame of the stream of frames between the uncompressed frames and the reconstructed frames in the first encoding pass, and calculating, by the encoder, a PSNR associated with each frame of the stream of frames between the uncompressed frames and the reconstructed frames in the second encoding pass.

[0011] In some embodiments, the statistical model includes a linear regression model based on the PSNR and QP pair.

[0012] The method may also include determining a target PSNR threshold. Encoding each frame of the stream of frames is based on the target PSNR threshold.

[0013] The method may also include downsampling each frame of the stream of frames based on a downsampling parameter, and encoding the downsampled frame of the stream of frames based on a target bitrate parameter in response to the downsampling.

[0014] According to another aspect, an apparatus for encoding video for real-time cloud gaming comprises one or more processors; a memory coupled to the one or more processors; and instructions executable by the one or more processors to determine a minimum QP based on a selected target PSNR and encode each frame of a stream of frames based on the minimum QP.

[0015] In some embodiments, the instructions are further executable by the one or more processors to update the statistical model based on a peak signal-to-noise ratio associated with each frame of the stream of frames. In some embodiments, the statistical model comprises a linear regression model. In some other embodiments, the linear regression model is based on one or more pairs of PSNR and QP.

[0016] In some embodiments, the instructions are further executable by the one or more processors to downsample each frame of the stream of frames based on the downsampling parameters, and, in response to the downsampling, encode the downsampled frames of the stream of frames based on the target bitrate parameters.

[0017] According to yet another aspect, a non-transitory computer-readable medium having stored thereon code for encoding game content for real-time cloud gaming on a device, the code comprising instructions executable by a processor of the device to determine a minimum QP based on a selected target PSNR and encode each frame of the stream of frames based on the minimum QP.

[0018] In some embodiments, the instructions are further executable by a processor of the device to update a statistical model based on a PSNR associated with each frame of the stream of frames. The statistical model includes a linear regression model. The linear regression model is based on one or more pairs of a peak signal-to-noise ratio and a quantization parameter.

[0019] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram of a cloud-based gaming system that supports real-time game content encoding, according to some embodiments. [Figure 2] 2 is a block diagram of a device of the system of FIG. 1 that supports content-adaptive encoding, according to some embodiments. [Figure 3] 1 is a flowchart of a method for PSNR-guided gaming content adaptive encoding, according to some embodiments. [Figure 4] 4 is a flowchart illustrating additional aspects of the method of FIG. 3, including a statistical model for PSNR to QP mapping, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0021] Detailed Description Cloud-based gaming systems allow end users to access and experience gaming applications, sometimes originally configured for non-portable devices (e.g., personal computers or gaming console setups), on remote devices such as portable client devices (e.g., laptops, smartphones). In cloud-based gaming systems, a network device (e.g., a game server) performs operations such as graphics rendering, capturing, encoding, and streaming of game content to a communications network. The portable client device receives an encoded bitstream of the game content over the communications network, decodes the encoded bitstream, and displays the resulting images to the user.

[0022] To enhance the user experience, it may be desirable for a cloud-based gaming system to generate and deliver high-resolution, high-frame-rate video streams to client devices. However, such high-quality video streams consume significant network bandwidth and encoding resources. For example, a 60-frames-per-second (FPS) stream of 1080p frames (e.g., 1080p video) using a particular codec requires network bandwidth that provides a consistent data rate of greater than 25 megabits per second (Mbps) to avoid degradation of the game content. Furthermore, network bandwidth may vary based on several factors, such as the underlying technology supporting a particular network connection. For example, digital subscriber line (DSL) connections provide data rates of 10-25 Mbps, cable connections provide data rates of 50-100 Mbps, fiber connections provide data rates of 100 Mbps or more, and satellite connections provide data rates of 15-25 Mbps. Therefore, depending on the particular network connection, a particular codec may degrade performance, resulting in degradation of game content quality, thereby adversely affecting the user experience.

[0023] To meet bandwidth requirements and maintain low-latency video streaming, some gaming systems use hardware-accelerated encoders on various cloud-based platforms. These encoders for cloud-based applications focus on maximizing objective game quality to meet target bandwidth as closely as possible. Due to strict end-to-end latency requirements, preventing exceeding the target frame size for every frame is typically the highest priority for these cloud gaming systems. However, the human visual system is much less sensitive to additional subjective quality gains when the objective video quality of the encoded game content exceeds a threshold. That is, objective video quality, as measured by peak signal-to-noise ratio (PSNR) (among other metrics), and subjective video quality, as measured by mean opinion score (MOS), indicate that subjective video quality is maximized above a certain PSNR level.

[0024] Various aspects of the present disclosure relate to techniques for adaptive encoding of game content, and more particularly, to peak signal-to-noise ratio (PSNR)-guided encoding of game content. An encoder receives, for example, a stream of frames related to game content and outputs an encoded version of the stream of frames for streaming over a network (e.g., the Internet). The encoder receives and encodes input frames, which may be downsampled, to obtain first encoding pass information. This allows the encoder to accurately predict the current PSNR before the input frames are encoded in the second encoding pass (i.e., the final encoding stage). The encoder adaptively calculates a lower bound (minimum QP) in decibels of a QP corresponding to a selected PSNR target using a content-adaptive statistical model (e.g., a linear regression model) in a moving window of previous frames up to the current frame, and uses the lower bound QP in the rate control process for encoding the current frame. The result of this PSNR-guided encoding is to reduce the size of encoded frames while staying below the target bitrate if the PSNR indicates that those frames are unlikely to have subjective quality loss. These reductions alleviate bandwidth issues and improve overall game performance, as fewer bits transferred means faster network transmission, faster decoding, and much less load on the platform, all while maintaining the overall quality of the game stream.

[0025] 1 is a block diagram of a cloud-based gaming system 100 that supports encoding of real-time game content, according to some embodiments. The cloud-based gaming system 100 includes a client device 102, a server 104, and a database 106. Although the cloud-based gaming system 100 illustrates a single client device 102, a single server 104, a single database 106, and a single network 108, this disclosure applies to any cloud-based gaming system architecture having one or more devices 102, servers 104, databases 106, and networks 108. The client device 102, server 104, and database 106 communicate with each other over the network 108 using one or more communication links (e.g., wireless connections, wired connections).

[0026] The server 104 may be a data server, a cloud server, a server associated with a game subscription provider, a proxy server, a web server, an application server, or any combination thereof. The server 104, in some embodiments, includes a cloud gaming platform 110. The cloud gaming platform 110 enables client devices 102 to discover, browse, share, and stream dynamic game content 112 that is rendered on a server over a network 108 using one or more communication links, or static game content 112 that is stored in a database 106 by real-time digital distribution over the network 108. Thus, digital distribution may be a form of distribution of some static game content 112, such as audio, video, or images, over an online distribution medium, such as the Internet, without using a physical medium.

[0027] The database 106 may store various information, such as instructions or commands (e.g., game-related information). For example, the database 106 may store game content 112. The client device 102 may retrieve the game content 112 from the database 106 over the network 108 using one or more communication links. The network 108 may provide encryption, access permissions, tracking, Internet Protocol (IP) connectivity, and other access, computation, modification, and / or functionality. Examples of the network 108 may include any combination of a cloud network, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a wireless network (e.g., using 802.11), a cellular network (using a third generation (3G), fourth generation (4G), long-term evolved (LTE), or new radio (NR) system (e.g., fifth generation (5G)), etc.). The network 108 may include the Internet.

[0028] The one or more communication links illustrated in the cloud-based gaming system 100 include uplink transmissions from the client devices 102 to the server 104 and database 106, and / or downlink transmissions from the server 104 and database 106 to the client devices 102. These communication links may carry bidirectional and / or unidirectional communications. In some embodiments, the communication links may be wired or wireless connections, or both. For example, the communication links may include one or more connections including, but not limited to, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), cellular, Z-WAVE, 802.11, peer-to-peer, LAN, wireless local area network (WLAN), Ethernet, FireWire, fiber optic, and / or other connection types associated with wireless communication systems.

[0029] The client device 102, in some embodiments, is a cellular phone, a smartphone, a tablet computer, a laptop computer, a display device (e.g., a monitor), etc. that supports various types of video codecs, including h.264, vp9, or HEVC. In some embodiments, the server 104 and the client device 102 together implement a cloud gaming session for a user of the client device 102. Illustratively, the server 104 executes a game program 114 that generates a stream of frames 116 (images) based on a game state of a game implemented by the game program 114. The client device 102 executes a game application 118 that can receive the stream of frames from the server 104 and can receive input data from a user via a user interface (e.g., a game controller, a touchscreen, etc., or any combination thereof). The client device 102 provides the input data to the server 104, and the server 104 updates the game state of the game program 114 and generates more frames for the stream of frames based on the updated game state.

[0030] To execute their respective programs and implement the game session, client device 102 and server 104 each include one or more of a general-purpose processor, a digital signal processor (DSP), an image signal processor (ISP), a CPU, a GPU, a microcontroller, an ASIC, an FPGA, discrete gate or transistor logic components, discrete hardware components, or any combination thereof, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described in this disclosure. For example, client device 102 may decode encoded game content 112 (e.g., image data, video data, audio data) streamed from server 104.

[0031] The server 104 may also be configured to provide enhancement, restoration, analysis, and compositing, among other functions, on the game content 112. For example, the server 104 may perform white balancing, cropping, scaling (e.g., game compression), resolution adjustment, stitching, color processing, spatial game filtering, artifact removal, frame rate adjustment, etc. As a further example, the server 104 may process the game content 112 in accordance with the techniques described herein and support adaptive encoding of the game content 112.

[0032] For example, during a game session, the server 104 may receive and encode input frames associated with a stream of frames 116 that may be downsampled, obtaining first encoding pass information to help accurately predict the PSNR of a current or subsequent frame in the stream of frames 116 before it is encoded in a second encoding pass (i.e., the final encoding stage). The server 104 adaptively calculates a lower bound on the QP corresponding to a selected PSNR target using a content-adaptive statistical model (e.g., a linear regression model) in a moving window of previous frames up to the current frame. This PSNR-guided encoding results in a significant reduction in the size of encoded frames below the target bitrate, depending on whether those frames have no subjective quality loss using their PSNR as guidance. These reductions mitigate bandwidth issues and improve overall game performance, as fewer bits transferred results in faster network transmission, faster decoding, and a much smaller platform load.

[0033] FIG. 2 is a block diagram of a device 200 of the system of FIG. 1 that supports adaptive encoding of game content, according to some embodiments. In some embodiments, device 200 implements aspects of cloud-based gaming system 100 as illustrated in FIG. 1. For example, device 200 may be an example of server 104 as illustrated in FIG. 1. Device 200 includes one or more software and hardware components for interactive communication, including components for encoding game content 236. In some embodiments, device 200 is part of an electronic device that supports encoding game content 236, including a desktop computer, a server, a game console, etc. In the example of FIG. 2, device 200 includes at least a processor 202, a memory 204, and a modem 206. These components are in electronic communication via one or more interfaces (e.g., a bus 228).

[0034] Processor 202 is a processing device (e.g., a general-purpose processor, CPU, GPU, ASIC, FPGA, etc.) that generally supports the execution of a set of instructions (e.g., a computer program) to perform designated tasks on behalf of device 200. In some embodiments, processor 202 is configured to interface with memory 204 using a memory controller integrated with processor 202. Memory 204 includes RAM and ROM. In some embodiments, memory 204 stores computer-readable, computer-executable code including instructions that, when executed, cause processor 202 to perform various functions described herein. In some embodiments, memory 204 includes, among other features, a basic input / output system (BIOS), which controls basic hardware or software operations such as interaction with peripheral components of device 200. The code is stored in a non-transitory computer-readable medium, such as system memory or other type of memory. In some other embodiments, memory 204 additionally or alternatively stores statistical models 230 related to improving the performance of encoding game content 236 as described herein. The statistical model 230 may include a linear regression model that may map corresponding PSNR values ​​to corresponding QP values, as described herein.

[0035] Modem 206 communicates bidirectionally via one or more antennas, wired, or wireless communication links (e.g., Wi-Fi communication links, cellular communication links). In some embodiments, modem 206 is configured to modulate packets carrying encoded game content, provide the modulated packets to an antenna for transmission, and demodulate packets carrying encoded game content received from the antenna. In some embodiments, modem 206 represents a wireless transceiver configured to communicate bidirectionally with another device, such as client device 102 or server 104 as described in FIG. 1.

[0036] In the illustrated embodiment, processor 202 includes a game content module 208, a frame generation module 210, a frame analysis module 212, and an encoder 214. In some embodiments, game content module 208 is configured to generate game content 236 (e.g., images, video, audio) on behalf of encoder 214 for communication with another computing device via modem 206. For example, in some embodiments, game content module 208 is a program that executes on processor 202 and generates game content for communication with another device.

[0037] To generate the game content 236, the game content module 208 signals the frame generation module 210 via the control signal 226 to generate one or more frames 216 associated with the game content 236 (e.g., images, video, audio). For example, the game content module 208 signals the frame generation module 210 via the control signal 226 to generate one or more frames associated with a cloud-based gaming application, etc. The frame generation module 210 is implemented by the processor 202 or other components of the device 200. The frame generation module 210 signals the one or more generated frames 216 to the encoder 214 via the control signal 226. The encoder 214 is implemented by the processor 202 or other components of the device 200. In some embodiments, the encoder 214 encodes the one or more generated frames 216 into an encoded frame stream 232 according to a specified encoding technique, called a codec. In some embodiments, the encoder 214 is configured to forward the encoded frame stream 232 to the modem 206, for example, for transmission to another device (e.g., the client device 102 or the server 104 as illustrated in FIG. 1). For example, the encoder 214 generates the frame stream 232, which may be an uncompressed video stream from a GPU renderer (e.g., on the server 104 as illustrated in FIG. 1), and outputs the encoded video stream for streaming over the Internet via the modem 206.

[0038] The processor 202 or other components of the device 200 signal the frame generation module 210 via a control signal 226 to analyze one or more frames 216 associated with the game content 236 (e.g., images, video, audio) to support adaptive encoding of the game content 236. The frame analysis module 212 initializes one or more parameters associated with analyzing the one or more frames 216 associated with the frame stream 232. These parameters include one or more of a PSNR parameter 218, a minimum QP 220, a bitrate parameter 222, or a frame size parameter 224, or any combination thereof.

[0039] In some embodiments, the frame analysis module 212 determines a target PSNR threshold based on one or more factors. For example, the frame analysis module 212 may determine a target PSNR threshold of 45 decibels (dB) based on the underlying technology supporting a particular network connection (e.g., DSL, cable, fiber, satellite, etc.) and store this value in the target PSNR parameters 218. Additionally or alternatively, the frame analysis module 212 may determine a target PSNR threshold of 45 dB based on a default configuration (e.g., game settings) associated with a gaming application such as illustrated in FIG. 1 and store this value in the target PSNR parameters 218. In some embodiments, the frame analysis module 212 determines a target bitrate in Mbps. For example, the frame analysis module 212 may determine a target bitrate of 28 Mbps and store this value in the bitrate parameters 222. In some embodiments, the frame analysis module 212 determines a target frame size (e.g., in bits). The frame analysis module 212 may determine the target frame size based on a target bitrate or a frame rate, or a combination thereof. Additionally or alternatively, the target frame size may be based on a fixed resolution associated with a default configuration (e.g., game settings) associated with the gaming application, such as illustrated in Figure 1. The frame analysis module 212 stores the value of the target frame size in the frame size parameters 224.

[0040] In some embodiments, the encoder 214 encodes the frame 216-1 and provides the encoding information to the frame analysis module 212. The encoding information may include the frame size of the frame 216-1 and the PSNR associated with the frame 216-1 (e.g., the PSNR of the luma plane). In some embodiments, the frame analysis module 212 or the encoder 214 downscales the frame 216-1 to a fraction of its original resolution. The encoder 214 encodes the downscaled frame 216-1 using a target bitrate that can provide a PSNR estimate before encoding the current frame.

[0041] The frame analysis module 212 selects the minimum QP 220 for frame 216-1. For example, in some embodiments, the frame analysis module 212 compares the estimated PSNR of frame 216-1 with a target PSNR. If the PSNR of frame 216-1 is greater than the selected target PSNR, the frame analysis module 212 calculates a lower-bound QP using a linear mapping according to equation (1) based on a statistical model 230, which is described further below, and assigns the result as the lower-bound value of the minimum QP 220.

[0042] QP=f(PSNR)=slope*PSNR+ intercept ··(1) Therefore, the linear mapping is a function of PSNR. In some embodiments, the value of the slope may be predefined based on an initialization. For example, the value of the slope may be initially set to zero. In some embodiments, section The value of may be predefined based on initialization. For example, section The value of may be initially set to zero. The frame analysis module 212, in some embodiments, provides an indication or sets a minimum QP 220 for the encoder 214.

[0043] In some embodiments, the frame analysis module 212 may support rate control to adjust encoder parameters associated with the encoder 214 to achieve a target bit rate. In some embodiments, encoding the frames 216 may be an inherently lossy process. For example, the encoder 214 may achieve compression not only by removing truly redundant information from the frame stream 232 (bit stream), but also by making small quality compromises in a manner intended to be minimally perceptible. The minimum QP 220 provides a lower bound on the QP that the rate control can use. If the minimum QP 220 is very small, the rate control can preserve almost all of its detail. As the minimum QP 220 increases, some of that detail is aggregated to reduce the bit rate, but at the cost of some objective quality loss with some increase in objective distortion and minimal subjective quality loss, which may be selected such that the PSNR target is sufficiently high. The frame analysis module 212 may thereby dynamically vary the minimum QP 220 based on the PSNR estimate of the frames 216 so that each frame 216 has an appropriate bit allocation to maximize the target quality. Thus, subjective quality loss is minimized because the minimum QP 220 is determined by a high target PSNR.

[0044] As an example, frame analysis module 212 may subsequently receive frame 216-2 and perform rate control associated with encoding frame 216-2, thereby managing the quality of the encoded frame 216-2. In some embodiments, frame analysis module 212 selects a minimum QP 220 for frame 216-2 based on encoding information associated with frame 216-1. In some other embodiments, frame analysis module 212 selects a minimum QP 220 for frame 216-2 based on a statistical model 230, which may indicate a minimum QP 220 for a current window of frames that includes frame 216-2. Frame analysis module 212 signals the selected minimum QP 220 to encoder 214, and encoder 214 encodes frame 216-2 based on the selected minimum QP 220 for frame 216-2. The encoder 214 may determine and pass to the frame analysis module 212 coding information associated with the frame 216-2, such as the PSNR associated with the frame 216-2, after the current frame is encoded.

[0045] The encoder 214 provides the PSNR associated with frame 216-2 to the frame analysis module 212, which updates the statistical linear regression model 230 with the encoding information, including the selected QP for frame 216-2. For example, the frame analysis module 212 uses the encoding information to fit a linear curve in the linear regression model to the current window of frames associated with frame stream 232, as described in more detail in FIG. 4 . The encoder 214 and frame analysis module 212 may repeat the above operations for all frames 216 associated with frame stream 232. The frame analysis module 210 thereby determines a minimum QP for the current frame and upper-bounds its PSNR to the target PSNR threshold. The device 200 thereby supports a functional mapping from PSNR to QP to support adaptive encoding of the game content 236.

[0046] Thus, as explained above, the encoder 214 of the device 200 uses a content-adaptive linear regression model over a moving window of previous frames 216 up to the current frame 216 to adaptively calculate the minimum QP 220 corresponding to a selected target PSNR parameter 218. The result of this PSNR-guided two-pass encoder is often to significantly reduce the size of the encoded frames 216 by using the PSNR as guidance to stay below the target bitrate in the absence of subjective quality loss for these frames. These bitrate reductions not only alleviate bandwidth issues, but also improve overall performance.

[0047] 3 is a flowchart of a method for adaptive encoding of game content according to some embodiments. In some embodiments, the method is implemented by an encoder or components thereof, such as those described with reference to FIG. 1. For example, the encoder is implemented or configured to perform a method for improving game content encoding in cloud-based gaming applications to reduce latency and increase reliability of the encoding of the game content. In some embodiments, the encoder executes a set of instructions for controlling functional elements of the encoder to perform the operations described below. Additionally or alternatively, the encoder uses dedicated hardware to perform aspects of the operations described below.

[0048] In the example of Figure 3, the encoder performs one or more initialization operations and a first encoding operation. As part of the initialization phase, in some embodiments, it determines a default target PSNR, a default initial minimum QP, or both. In some other embodiments, it also determines a maximum number of frames for a frame range. In other embodiments, it determines a target PSNR threshold (e.g., 45 dB), a target bitrate parameter (e.g., 28 Mbps), or a target frame size parameter, or any combination thereof. The encoder may also be pre-configured with one or more of the above parameters as part of the initialization operation of the method for adaptive encoding of game content.

[0049] As part of a first encoding stage, in block 302, an encoder receives a current frame from a stream of frames associated with, for example, a cloud-based gaming application. In block 304, the encoder downscales the current frame. For example, the encoder downscales the current frame by a fraction of the current frame's original resolution. In some embodiments, the encoder downscales the current frame by half (½) the current frame's original resolution. In some other embodiments, the encoder downscales the current frame by one-quarter (¼) the current frame's original resolution. In other embodiments, the encoder downscales by a different percentage of the current frame's original resolution. In block 306, the encoder encodes the current frame downscaled during the first encoding pass. In some embodiments, the encoder encodes the current frame downscaled during the first encoding pass based on a target bitrate.

[0050] In block 308, the encoder determines a target bit rate based on a default setting or configuration of the encoder. In some embodiments, the encoder thereby determines a target bit rate, which can be expressed in Mbps, such as 28 Mbps, based on a default setting or configuration of the encoder. The encoder encodes the current frame downscaled during the first encoding pass based on the target bit rate of 28 Mbps. In some embodiments, the encoder encodes the current frame downscaled during the first encoding pass based on a fraction of the target bit rate (e.g., one-quarter (1 / 4) of the target bit rate). For example, the encoder encodes the current frame downscaled during the first encoding pass based on a fraction of the target bit rate of 28 Mbps (e.g., 7 Mbps). In block 310, the encoder additionally or alternatively determines a frame size of the current frame based on the first encoding pass. In block 312, the encoder determines a PSNR of the current frame by estimation based on the first encoding pass. For example, the encoder uses dedicated hardware and software techniques to determine the PSNR of the luma plane during the first encoding pass of the current frame.

[0051] Figure 4 is a flowchart illustrating additional aspects of the method of Figure 3 according to some embodiments. In some embodiments, the method is implemented by an encoder or components thereof, such as those described with reference to Figure 1. In some embodiments, the encoder executes a set of instructions to control functional elements of the encoder to perform the operations described below. Additionally or alternatively, the encoder uses dedicated hardware to perform aspects of the operations described below.

[0052] In the example of FIG. 4, the encoder performs an encoding operation that may be the second encoding operation of a two-pass encoding operation. In block 402, the encoder encodes a current frame during a second encoding pass. The encoding during the second encoding pass may be based on one or more parameters (e.g., a minimum quantization parameter, a target bit rate parameter). In block 404, the encoder determines a target bit rate. In block 406, the encoder determines previous frame information. In block 408, the encoder determines previous PSNR information. Thereby, the encoding of the current frame during the second encoding pass may be based on the target bit rate, previous frame information, previous PSNR information, an estimated PSNR from the first encoding pass, or any combination thereof.

[0053] In block 410, the encoder determines the frame size of the current frame based on the second encoding pass. In block 412, after the second encoding pass, the encoder determines the PSNR of the current frame. In block 414, the encoder calculates QP=f(PSNR)=slope*PSNR+intercept( intercept ) mapping, a statistical model, for example a linear regression model, is updated based on the {current psnr, current qp} pair.

[0054] The encoder maintains a double-ended queue of {PSNR, QP} pairs in a moving window of previous frames up to the current frame. After the second encoding pass of the current frame, the {PSNR, QP} pairs for the current frame are pushed to the end of the queue. When the queue size reaches a maximum size determined by the encoder's initialization stage, block 414 fits a linear curve defined by QP=a*PSNR+b to the current queue of {PSNR, QP} pairs, where a is Tilt and b is sectionIn some embodiments, the average PSNR and average QP are calculated for the queue. For each pair of {PSNR, QP} in the queue, the difference between the average PSNR and the PSNR is squared, and the result is then accumulated in s_xx. Similarly, the difference between the average QP and the QP is multiplied by the difference in the average PSNR, and then accumulated in s_xy. In some embodiments, the linear curve section (b) and slope (a) are determined by the following formula:

[0055] a=s_xy / s_xx b=Average QP-Slope*Average PSNR In some embodiments, the encoder finally removes the {PSNR, QP} pair at the top of the queue.

[0056] In some embodiments, some aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software comprises one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and some data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage, solid-state storage such as flash memory, cache, RAM, or other non-volatile storage, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by one or more processors.

[0057] A computer-readable storage medium may include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, or magnetic hard drives), volatile memory (e.g., RAM or cache), non-volatile memory (e.g., ROM or flash memory), or micro-electromechanical systems (MEMS)-based storage media. A computer-readable storage medium may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or USB-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0058] It should be noted that not all of the activities or elements described above in the general description are required, that some of the specific activities or devices may not be required, and that one or more additional activities may be performed or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure, as set forth in the claims below. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0059] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, benefits, advantages, solutions to problems, and any features that may cause or make more pronounced any benefit, advantage, or solution should not be construed as critical, necessary, or essential features of any or all claims. Moreover, the specific embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design shown herein, other than as set forth in the claims below. It is therefore apparent that the specific embodiments disclosed above may be altered or modified, and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the following claims.

Claims

1. 1. A method for encoding video for real-time cloud gaming, comprising: determining a peak signal-to-noise ratio associated with encoding a downscaled version of each frame of the stream of frames in a first encoding pass; encoding each frame of the stream of frames in a second encoding pass based on the peak signal-to-noise ratio in response to the first encoding pass; determining, based on a statistical linear regression model, a minimum quantization parameter associated with a selected target peak signal-to-noise ratio for encoding each frame of the stream of frames; encoding a frame of the stream of frames based on the minimum quantization parameter for the encoded frame to meet the selected target PSNR; determining a linear mapping from the peak signal-to-noise ratio of each frame to the minimum quantization parameter by fitting a linear curve to pairs of peak signal-to-noise ratios and quantization parameters in a moving window of previous frames in the deque; The method, wherein the linear mapping is updated after the second encoding pass of each frame.

2. 2. The method of claim 1 , further comprising determining a floor value for the minimum quantization parameter based on the selected target peak signal-to-noise ratio and the updated linear mapping for each frame comprising the stream of frames.

3. The method of claim 1 , further comprising determining an intercept and a slope of the linear mapping from the peak signal-to-noise ratio to the minimum quantization parameter based on the moving window of a previous frame.

4. The method of claim 3 , further comprising maintaining the size of the moving window of a previous frame relative to a peak signal-to-noise ratio and quantization parameter pair.

5. The method further comprises updating a statistical model based on the peak signal-to-noise ratio associated with each frame of the stream of frames; The method of claim 1 , wherein encoding each frame of the stream of frames is based on the smallest quantization parameter from the statistical model.

6. The method comprises: calculating the peak signal-to-noise ratio associated with each frame of the stream of frames between uncompressed frames and reconstructed frames in the first encoding pass; 6. The method of claim 1, further comprising: calculating the peak signal-to-noise ratio associated with each frame of the stream of frames between the uncompressed frames and the reconstructed frames in the second encoding pass.

7. The method of claim 5 , wherein the statistical model comprises a linear regression model based on pairs of peak signal-to-noise ratios and quantization parameters.

8. The method further comprises determining a target peak signal-to-noise ratio threshold; The method of any one of claims 1 to 5, wherein encoding each frame of the stream of frames is based on the target peak signal-to-noise ratio threshold.

9. The method comprises: downsampling each frame of the stream of frames based on a downsampling parameter; The method of any one of claims 1 to 5, further comprising: in response to said downsampling, encoding said downsampled frames of said stream of frames based on a target bitrate parameter.

10. 1. An apparatus for encoding video for real-time cloud gaming, comprising: one or more processors; a memory coupled to the one or more processors; by the one or more processors determining a minimum quantization parameter based on the selected target peak signal-to-noise ratio; encoding each frame of the stream of frames based on the smallest quantization parameter; instructions executable to determine a linear mapping from the peak signal-to-noise ratio of each frame to the minimum quantization parameter by fitting a linear curve to pairs of peak signal-to-noise ratios and quantization parameters in a moving window of previous frames in a double-ended queue; The apparatus, wherein the linear mapping is updated after a first encoding pass in which encoding based on the target peak signal-to-noise ratio is performed followed by a second encoding pass in which encoding is performed.

11. The apparatus of claim 10 , wherein the instructions are further executable by the one or more processors to update a statistical model based on the target peak signal-to-noise ratio associated with each frame of the stream of frames.

12. The apparatus of claim 11 , wherein the statistical model comprises a linear regression model.

13. The apparatus of claim 12 , wherein the linear regression model is based on one or more pairs of peak signal-to-noise ratios and quantization parameters.

14. The instruction: downsampling each frame of the stream of frames based on a downsampling parameter; and The apparatus of claim 10 , further executable by the one or more processors to, in response to the downsampling, encode the downsampled frames of the stream of frames based on a target bit rate parameter.

15. A computer program for causing a computer to execute the method according to any one of claims 1 to 9.

16. a memory storing the computer program according to claim 15; and one or more processors for executing said computer programs.

Images