Low-latency multi-path frame-level rate control using shared reference frames

Abstract

A multi-pass encoding operation using a shared reference frame is performed to encode one or more game frames into a game stream. The multi-pass encoding operation includes generating a shared reference frame based on a second pass reference frame used by a second pass encoding of the multi-pass encoding operation. The encoding operation also includes performing a first encoding pass on a current frame using the shared reference frame. As a result of the first encoding pass, an estimated complexity of the current frame is determined. A second pass encoding is then performed on the current frame according to the second pass reference frame and the estimated complexity, resulting in an encoded frame. The encoded frame is then transmitted to a client system as part of the stream.

Description

[Background technology]

[0001] background A cloud-based gaming platform relates to executing a video game application, in part or in whole, on a remote server to facilitate playing of the video game application on a local client device. The remote server provides rendered audio and video from the execution of the video game application to the local client device as audio and video streams over a network. In providing these audio and video streams, the remote server uses various encoder operations to compress the game frames and audio in real-time before they are streamed. For low-latency cloud gaming applications, it is desirable to ensure that the encoder operations do not compress the game frames with a number of bits that would cause the stream to require more bandwidth than the available network allows.

[0002] Unlike typical video streaming applications, which can tolerate some latency, cloud-based games require real-time interactivity to be maintained. To address this, some cloud-based gaming platforms require all frames to be encoded very close to a target frame size that matches the currently available network bandwidth. Encoding game frames at a size other than the target frame size can result in undershoot (fewer bits than allowed) or overshoot (more bits than allowed), which can cause packet loss and latency, resulting in a degraded gaming experience. Summary of the Invention

[0003] Overview of the embodiment Techniques and systems described herein are directed to multi-pass encoding operations using a shared reference frame. According to an exemplary embodiment, a computer-implemented method can include comparing a first pass resolution of a first pass encoding of a multi-pass (e.g., two-pass) encoding process with a second pass resolution of a second pass encoding of the multi-pass encoding process. In addition, the method can include obtaining a second pass reference frame associated with the second pass encoding based on the comparison to generate a shared reference frame. This refers to a method that includes, for example, obtaining a second pass reference frame used for second pass encoding of a first game frame to be encoded and using the second pass reference frame as a first pass reference frame for use in the first pass encoding of the first game frame based on the resolution comparison, or adapting the second pass reference frame to generate a first pass reference for use in the first pass encoding of the first game frame. In any case, the second pass reference frame is used as a basis for the first pass reference frame when performing the first pass encoding, so that the second pass reference frame forms a shared reference frame that is used not only in the second pass encoding but also for the first pass encoding. The method may include performing first pass encoding on the first game frame at the first pass resolution using the shared reference frame to determine an estimated complexity of the first game frame of the stream of game frames. The method may additionally include performing second pass encoding on the first game frame at the second pass resolution using the second pass reference frame and according to the estimated complexity to generate an encoded frame. Thus, the proposed method may additionally include performing second pass encoding on the first game frame at the second pass resolution using the second pass reference frame and the estimated complexity to generate an encoded frame.As used herein, the "complexity" of a current frame refers to an amount or value of information in the current frame that is different from one or more reference frames associated with the current frame (e.g., a reference frame used to encode the current frame). For example, the complexity of the current frame includes the difference in signal-to-noise ratio between the current frame and one or more reference frames, the difference in energy between the current frame and one or more reference frames, the difference in entropy between the current frame and one or more reference frames, or any combination thereof, to name a few. Thus, the more complex the current frame is, the more information is required to encode it.

[0004] In this method, obtaining the second pass reference frame may include obtaining the second pass reference frame from a memory buffer associated with the second pass encoding. Additionally, obtaining the second pass reference frame may include downsampling the second pass reference frame to the first pass resolution to generate a downsampled reference frame in response to the first pass resolution being lower than the second pass resolution, and storing the downsampled reference frame in a second memory buffer associated with the first pass encoding to generate the shared reference frame. Obtaining the second pass reference frame may also include storing the second pass reference frame in a second memory buffer associated with the first pass encoding to generate the shared reference frame in response to the first pass resolution being equal to the second pass resolution.

[0005] In this method, the second pass reference frames can be obtained according to pointers stored in a memory buffer associated with the first pass encoding. Further, the memory buffer can include a plurality of pointers each associated with a respective second pass reference frame stored in a second pass memory buffer associated with the second pass encoding.

[0006] In another exemplary embodiment, a computer-implemented method can include generating a shared reference frame for use in a first pass encoding of a multi-pass encoding process based on a second pass reference frame for use in a second pass encoding of a multi-pass encoding process. Generating the shared reference frame can include comparing a first pass resolution of the first pass encoding to a second pass resolution of the second pass encoding and downsampling the second pass reference frame based on the comparison of the first pass resolution and the second pass resolution. The method can also include performing a first pass encoding at the first pass resolution on a first game frame of the stream of game frames using the shared reference frame and determining an estimated complexity of the first game frame based on the first pass encoding of the first game frame. In addition, the method can include performing a second pass encoding on the first game frame at the second pass resolution using the second pass reference frame and based on the estimated complexity to generate an encoded frame.

[0007] In the method, generating the shared reference frame may also include accessing a memory buffer associated with the second pass encoding according to a pointer in the second memory buffer associated with the first pass encoding. The second memory buffer may include a plurality of pointers each associated with a respective second pass reference frame stored in the memory buffer associated with the second pass encoding. Generating the shared reference frame may further include, in response to the first pass resolution being lower than the second pass resolution, downsampling the second pass reference frame to the first pass resolution to generate a downsampled reference frame, and storing the downsampled reference frame in the second memory buffer to generate the shared reference frame. Generating the shared reference frame may also include, in response to the first pass resolution being equal to the second pass resolution, storing the second pass reference frame in the second memory buffer associated with the first pass encoding to generate the shared reference frame.

[0008] In general, the first pass encoding may also include downsampling the first game frame to a first pass resolution. Determining an estimated complexity of the first game frame based on the first pass encoding of the first game frame may also include determining a quantization parameter for the first game frame. Further, a second pass encoding may be performed based on the quantization parameter. Performing the first pass encoding may generate a first pass encoded frame having a bit size, and determining the estimated complexity may be further based on the bit size.

[0009] Further, in general, the first game frame may represent at least a portion of a virtual environment of a game session associated with the client system. Any method herein may also include encoding the stream of game frames according to a multi-pass encoding process to generate an encoded stream of game frames and transmitting the encoded stream of game frames to the client system. Any method herein may also include decoding the stream of game frames to generate a decoded stream of game frames and displaying the decoded stream of game frames. Further, any method herein may include rendering the stream of game frames according to a client game session associated with the client system.

[0010] According to an example embodiment, a cloud-based gaming system can include one or more cloud-based gaming servers communicatively coupled to one or more client systems, each cloud-based gaming server including one or more processors and a memory coupled to the one or more processors and storing executable instructions configured to operate the one or more processors to perform one or more of the methods disclosed herein.

[0011] Additionally, as an example embodiment, a cloud-based gaming system can include a network interface couplable to a network and an encoder (e.g., of a game server of the gaming system) coupled to the network interface, the system configured to perform any method disclosed herein. Yet providing an additional example embodiment, the cloud-based gaming system can include an interface configured to receive encoded frames (e.g., for a client system of the gaming system) encoded according to any method disclosed herein, and one or more decoders configured to decode the encoded frames.

[0012] The present disclosure may be better understood, and its numerous features and advantages made apparent, to those skilled in the art by reference to the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of a cloud-based gaming system that applies a multi-pass encoding technique using one or more shared frames of reference, according to some embodiments. [Diagram 2] FIG. 1 is a block diagram of a computing device configured to encode and stream frames within a cloud-based gaming system, according to some embodiments. [Diagram 3] FIG. 2 is a flow diagram illustrating a method for multi-pass encoding of game frames of a stream of game frames according to some embodiments. [Figure 4] FIG. 2 illustrates an example of a multi-pass encoding process using one or more shared reference frames, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Detailed Description The techniques and systems described herein address the demand for providing low-latency video streams from one or more cloud-based game servers to client devices while maintaining video quality. To this end, the cloud-based game server first renders a set of game frames for a game application and encodes the rendered game frames to generate a game stream. To better ensure that the game stream approaches a target bit rate, the cloud-based game server performs a multi-pass (e.g., two-pass) encoding operation when encoding each rendered game frame. Such a multi-pass encoding operation includes, for example, performing a first pass encoding on the game frame to estimate a complexity of the game frame, and performing a second pass encoding of the game frame based on the estimated complexity of the game frame. To this end, the cloud-based game server first performs a first pass encoding of the game frame by encoding the game frame using a first pass reference frame. Based on the results of the first pass encoding (e.g., the first pass encoded frames), the cloud-based game server estimates a complexity of the game frame, and determines a quantization parameter of the game frame based on the estimated complexity. The cloud-based game server then performs second pass encoding of the game frame using the second pass reference frame and the quantization parameters (e.g., using the quantization parameters in the second pass encoding to determine a quantization step size) to generate a (e.g., second pass) encoded game frame. To limit the encoded game frame from being assigned more bits than allowed during encoding (i.e., overshoot), the cloud-based game server uses one or more shared reference frames for the first pass encoding. That is, the cloud-based game server uses a shared reference frame for the first pass encoding that is based on the second pass reference frame (e.g., a reference frame used in the second pass encoding).In this way, the estimated complexity determined from the first pass encoding more accurately represents the actual complexity of the game frames, and the system can more accurately allocate the number of bits to encode each game frame, thereby reducing the risk that the resulting stream will be larger than the network bandwidth and minimizing packet loss over the network.

[0015] For ease of understanding, the techniques of the present disclosure are described in the exemplary context of a cloud gaming system. A cloud-based or other remote server renders a stream of video game frames representing the visual content of a video game instance running on that server or an associated server, and then encodes each game frame using a multi-pass encoding process described herein to generate a bitstream representing the stream of encoded and rendered game frames for transmission to one or more client devices over one or more networks. However, it will be appreciated that the systems and techniques described herein are not limited to this exemplary context, and instead may be implemented in any of a variety of video stream encoding / decoding systems using the guidelines provided herein.

[0016] FIG. 1 illustrates a cloud-based gaming system 100 for single-player or multiplayer (including massively multiplayer) gaming, according to some embodiments. The cloud-based gaming system 100 includes one or more servers 102, each of which is communicatively coupled to one or more client systems 112 by at least one network 110. The network 110 is configured to enable transmission and reception of data between any of the servers 102 and the client systems 112, and includes, for example, wired and wireless networks, such as Ethernet, Internet, Wi-Fi, or any combination thereof. In an embodiment, each of the servers 102 is communicatively interconnected via a high-bandwidth, low-latency, server-to-server messaging bus. The servers 102 are typically distributed in one or more data centers across a geographic region to reduce transmission latency due to physical proximity. In the illustrated embodiment, three servers 102-1, 102-2, and 102-3 are shown supporting four client systems 112-1, 112-2, 112-3, and 112-4, although in other embodiments any number of servers may be implemented supporting any number of client devices. It will be appreciated that in a typical real-world implementation, the number of servers 102 and the number of client systems 112 will typically be significantly greater than shown in the example of FIG.

[0017] In the depicted embodiment, each server 102 includes a computing device, such as, for example, a cloud server, a virtual server, or any combination thereof, configured to support one or more client game sessions executed at one or more client systems 112. As used herein, a "client game session" includes a game application being played, partially simulated, or fully simulated at the client system 112. Each server 102 is configured to support this client game session by executing a corresponding game platform instance 104 that facilitates the execution of at least a portion of the game application being played, partially simulated, or fully simulated at the client system 112. Such facilitation may include, for example, performing one or more operations related to the game application, including rendering one or more frames related to the game application, performing calculations (e.g., lighting calculations, artificial intelligence calculations, physics calculations, shading calculations, input calculations, etc.) related to the game application, providing access to files, or any combination thereof, to name a few. The game platform instance 104 provides various software and hardware resources to accomplish such facilitation, such as communication / network management, resource management, media rendering encoding, etc. In this manner, the gaming platform instance 104 simulates the execution of one or more actions of a gaming application for a corresponding player as if that gaming application were being played on a local gaming device, such as a personal computer (“PC”), gaming console, smartphone, tablet computer, in-vehicle entertainment system, etc.

[0018] Each client system 112 represents hardware and software resources utilized to receive player input through the operation of one or more input / output devices for at least one player, and to present video and audio content representing the visual and aural content of each of the at least one player's gameplay. Examples of client systems 112 include one or more desktop computers, notebook computers, tablet computers, virtual reality systems, augmented reality systems, computationally enabled cellular telephones (i.e., "smartphones"), computationally enabled televisions (i.e., "smart TVs"), or combinations thereof, to name a few. As illustrated with reference to client system 112-1, each client system 112 includes one or more client devices 116. In the illustrated embodiment, the client system 112-1 comprises a first client device 116-1 communicatively coupled to or otherwise associated with a display 118, at least one input device 120 (e.g., gamepad, joystick, keyboard, mouse, touch screen), one or more network interfaces configured to couple to a network connecting the client system 112 to a corresponding server 102, one or more processors, memory, storage, speakers, and other computing resources for rendering, processing, and displaying scenes of the virtual environment. As illustrated with reference to the client system 112-1, each client system 112 may include a decoder 114 configured to decode one or more frames associated with the virtual environment. The decoder 114 may include hardware and software configured to decode one or more encoded streams (e.g., game stream 108) received from the server 102 to generate a decoded set or stream of game frames.Each decoder 114 is configured to decode any encoded frame encoded by any method or process disclosed herein. In an embodiment, client system 112-1 further comprises a smartphone client device 116-2 and a wearable virtual reality client device 116-3, each of which may operate as an integrated mobile computing device having input, output, display, and communication capabilities similar to those described above with respect to client system 112-1. In a particular embodiment, client systems 112-1, 112-2, and 112-3 may include one or more features, such as an accelerometer, a global positioning system (GPS) device, etc., used to obtain motion data representative of motion of the client device as well as the speed or acceleration of such motion.

[0019] Although certain aspects described herein are discussed with particular reference to cloud gaming scenarios, it will be appreciated that in certain embodiments, the described techniques may be utilized in a variety of scenarios other than gaming, such as, for example, when one or more of the server 102 and client systems 112 operate to render, process, and display other types of informational, educational, recreational, and / or artistic content. Thus, while techniques relating to rendering of content that may utilize specific examples related to cloud gaming and gaming content are discussed herein, it will be further appreciated that such discussions and techniques may be applied to such non-gaming scenarios. The examples provided herein may refer to scenarios involving rendering, processing, and displaying gaming content due to certain bandwidth and network latency issues associated with gaming content, and should not be construed as indicating that the described techniques are limited to those scenarios.

[0020] During operation, each server 102 executes a game platform instance 104 for one or more client game sessions. Executing the game platform instance 104 includes rendering a set of game frames including one or more game frames associated with a game application executing at one or more respective client systems 112. Each rendered game frame represents at least a portion of a virtual environment used by the game application executing at the client system 112. For example, each rendered game frame may represent at least a portion of a virtual environment displayed on a display 118 of a client system 112 during a client game session.

[0021] Each server 102 is configured to encode each rendered game frame via an encoder 106 to generate a respective encoded set of game frames (also referred to herein as a "game stream" 108). Each server 102 is configured to encode the game stream 108, for example, by compressing, reconstructing, and manipulating each frame rendered by the game platform instance 104. In an embodiment, each encoder 106 of a server 102 implements one or more codecs to encode one or more rendered frames according to the one or more codecs. Such codecs may include H.264, H.265, VP9, ​​AV1, or any combination thereof, to name a few. According to an embodiment, each server 102 is configured to encode each frame rendered by the game platform instance 104 using one or more shared reference frames for the first pass encoding based on one or more respective reference frames used in the second pass encoding. As discussed in detail below with reference to Figure 3, this multi-pass encoding operation includes obtaining one or more reference frames associated with the second pass encoding to generate one or more respective shared reference frames, performing a first pass encoding on the game frames using one or more of the shared reference frames to estimate a complexity, and performing a second, subsequent encoding pass on the game frames according to the estimated complexity. Each resulting game stream 108 corresponds to a game application running on one or more client systems 112 and is provided to the corresponding client systems over the network 110. Each corresponding client system 112 is configured to decode the received game stream 108 via a decoder 114 and display the resulting decoded set of game frames 122, for example on a display 118.Each client system 112 is configured to decode a respective game stream 108 by compressing, reorganizing, and manipulating game frames within the encoded stream according to one or more different video codecs, including lossless and lossy codecs. According to an embodiment, each client system 112 includes a decoder that implements one or more codecs to decode the received game stream 108 according to the one or more codecs. Such codecs may include H.264, H.265, VP9, ​​AV1, or any combination thereof, to name a few. In the illustrated embodiment, three game streams 108-1, 108-2, 108-3 are depicted, but in other embodiments, the server 102 may generate any number of game streams 108, each corresponding to one or more client game sessions.

[0022] Referring now to FIG. 2, a computing system 200 configured to encode and stream frames in a cloud-based gaming system is illustrated. In an embodiment, the computing system 200 implements aspects of the cloud-based gaming system 100 described in FIG. 1. For example, the computing system 200 may be similar to or the same as the server 102 described in FIG. 1. The computing system 200 includes one or more software and hardware components for two-way communication, including components for encoding a set of game frames 250, such as generating a game stream 108. In some embodiments, the computing system 200 is part of an electronic device that supports encoding of the set of game frames 250, including, for example, a desktop computer, a notebook computer, a tablet, a server, a game console, to name a few. In an embodiment, the computing system 200 includes a processor 238, a modem 240, and a memory 242. The memory 242 includes an electronic storage device, such as, for example, a solid state drive, a hard disk drive, a random access memory ("RAM"), a read only memory ("ROM"), an electronically erasable programmable ROM ("EEPROM"), an optical storage device, or any combination thereof. The memory 242 includes instructions and data related to the operation of the game platform instance 204, the encoder 206, the downsampler 234, and the rate control unit 224, such as, for example, codecs, reference frames, game engines, game applications, constants, etc. The modem 240 is configured to be communicatively coupled to one or more client systems 112 via the network 110, and further configured to transmit the game stream 108 to the one or more client systems 112. According to an embodiment, the processor 238, the modem 240, and the memory 242 electronically communicate internally via one or more interfaces (e.g., a bus 244).

[0023] According to an embodiment, the processor 238 includes one or more control processing units ("CPUs"), microprocessors, field programmable gate arrays ("FPGAs"), graphics processing units ("GPUs"), application specific integrated circuits (ASICs), or any combination thereof, and is configured to render and encode game frames for use in client game sessions in the cloud-based gaming system 100. The processor 238 operates to execute a gaming platform instance 204, the same as or similar to the gaming platform instance 104, associated with a current client game session and configured to support a gaming application executing on one or more client systems 112. The gaming platform instance 204 includes graphics hardware and software (not shown for clarity) for rendering a set of game frames 250 associated with an environment of a gaming application executing on one or more client devices. Such graphics hardware and software may include, for example, a graphics core, processing cores, pixel shaders, video random access memory ("VRAM"), GPU, physics engines, lighting engines, tessellation engines, and the like. Each rendered game frame of the set of game frames 250 represents at least a portion of a virtual environment associated with a game application executing on a client device. For example, if the game application is a racing game, each game frame of the set of game frames 250 represents at least a portion of a race track, a car, or a surrounding area.

[0024] The game platform instance 204 provides each rendered game frame of the set of game frames 250 to the encoder 206 for encoding into the game stream 108. The encoder 206 includes one or more software and / or hardware encoders configured to encode the game stream 108, for example, according to inter-frame and intra-frame techniques. To reduce the bandwidth required to transmit the game stream 108 between the computing system 200 and the one or more client systems 112, the encoder 206 encodes the game stream 108 by compressing one or more game frames of the set of game frames 250. Compressing the game frame includes comparing the game frame to one or more reference frames stored in the memory buffer 228 and encoding one or more differences between the game frame and the one or more reference frames into the game stream 108. The encoder 206 is further configured to encode the reference frames used into the game stream 108. Furthermore, to match the available bandwidth, the encoder 206 is configured to encode each frame according to a rate control scheme such that the size of each encoded frame is compressed to approach a predetermined target bit size. The rate control scheme is implemented by a rate control unit 224, which includes one or more processors, hard-coded logic, programmable logic, or a combination thereof, configured to control or adjust the bitrate and respective number of bits (i.e., the degree of compression) for encoding a current frame of the set of game frames 250. The rate control unit 224 controls or adjusts the bitrate and respective number of bits by determining an appropriate quantization parameter ("QP") for the current frame as a function of the complexity of the current frame.As used herein, the "complexity" of a current frame refers to an amount or value of information in the current frame that is different from one or more reference frames associated with the current frame (e.g., a reference frame used to encode the current frame). For example, the complexity of the current frame includes the difference in signal-to-noise ratio between the current frame and one or more reference frames, the difference in energy between the current frame and one or more reference frames, the difference in entropy between the current frame and one or more reference frames, or a combination thereof, to name a few. Thus, the higher the complexity of the current frame, the more information is required for encoding in the game frame encoder 206. Therefore, it is important that the rate control unit 224 determines a sufficiently high QP to help prevent the resulting encoded frame from having a number of bits greater than a predetermined target size associated with the target bit rate. Those skilled in the art will recognize that such a determined QP will cause more aggressive quantization of details for a current frame with high complexity, resulting in an encoded frame with a reduced number of encoded bits. Similarly, for a current frame having low complexity, the encoder 206 may select a QP for less aggressive quantization, reducing the risk of potential overshoot, since there is less new information to encode.

[0025] In an embodiment, rate control unit 224 determines the QP for the current frame according to the following formula: [Formula 1] t=(a1 / QP+a2 / QP 2 )*c where t represents a predetermined desired frame size or degree of compression of the encoding frame, a1 and a2 represent predetermined constants derived from past encoding frames, and c represents the complexity or distortion of the current frame (e.g., c represents a complexity that indicates the difference in signal-to-noise ratio between the current frame and one or more reference frames, the difference in energy between the current frame and one or more reference frames, the difference in entropy between the current frame and one or more reference frames, or any combination thereof).

[0026] According to an embodiment, the encoder 206 determines the complexity of the current frame by performing a multi-pass encoding operation. The multi-pass encoding operation includes performing a first pass encoding of the current frame of the set of game frames 250 using one or more shared reference frames obtained from the first pass memory buffer 230. In an embodiment, the first pass encoding is performed at a first pass resolution (e.g., a predetermined resolution). According to an embodiment, performing the first pass encoding at the first pass resolution includes the downsampler 234 reducing the resolution of the current frame, the shared reference frame, or both, to the first pass resolution. In an embodiment, the first pass encoding is performed according to a predetermined first pass QP determined from one or more codecs implemented by the encoder 206. According to an embodiment, performing the first pass encoding on the current frame results in a first pass encoded frame having a bit size. After the first pass encoding, the rate control unit 224 determines an estimated complexity 226 for the current frame as a function of the bit size of the first pass encoded frame. The multi-pass encoding operation further includes a second pass encoding of the current frame using one or more second pass reference frames obtained from the second pass memory buffer 232. According to an embodiment, the encoder 206 performs the second pass encoding at a second pass resolution. That is, when the encoder 206 performs the second pass encoding on the current frame, the current frame and the second pass reference frames are at the second pass resolution. In some embodiments, the rate control unit 224 calculates a quantization parameter for the current frame as a function of the estimated complexity 226. The rate control unit 224 then passes the quantization parameter to the encoder 206, which determines a corresponding quantization step size. The encoder 206 uses the quantization step size to control or adjust the output bits when encoding the current frame.

[0027] In an embodiment, to more accurately determine the estimated complexity 226, the encoder 206 is configured to use one or more shared reference frames for the first pass encoding. The shared reference frames include, for example, reference frames based on the respective second pass reference frames used in the second pass encoding of the current frame. According to an embodiment, the encoder 206 generates one or more shared reference frames for the first pass memory buffer 230, each including one or more pointers associated with a current frame of the set of game frames 250. For the current frame, the associated pointers include the location of the respective second pass reference frames (e.g., the second pass reference frames used in the second pass encoding of the current frame) in the second pass memory buffer 232. In an embodiment, to generate the shared reference frames, the encoder 206 is configured to obtain the second pass reference frames from the second pass memory buffer 232 according to the associated pointers in the first pass memory buffer 230. Obtaining the second pass reference frame includes storing the second pass reference frame in the first pass memory buffer 230 to generate a shared reference frame for use in the first pass encoding. In other words, obtaining the second pass reference frame includes storing the second pass reference frame in the first pass memory buffer 230 and also using the second pass reference frame in the first pass encoding such that the second pass reference frame forms a shared reference frame used in both the second pass encoding and the first pass encoding (e.g., with or without downsampling).

[0028] According to an embodiment, the encoder 206 is configured to compare a first pass resolution of the first pass encoding for the current frame with a second pass resolution of the second pass encoding for the current frame. In response to the first pass resolution being lower than the second pass resolution, the downsampler 234 is configured to downsample a second pass reference frame associated with the current frame to the first pass resolution to generate a downsampled reference frame 236. That is, in response to the first pass resolution being lower than the second pass resolution, the downsampler 234 is configured to downsample a second pass reference frame obtained according to an associated pointer stored in the first pass memory buffer 230 to the first pass resolution to generate a downsampled reference frame 236. The downsampler 234 includes one or more processors, hard-coded logic, programmable logic, or any combination thereof configured to downsample and compress one or more rendered game frames 250, shared reference frames, second pass reference frames, or any combination thereof. In an embodiment, the encoder 206 is configured to store the downsampled reference frame 236 in the first pass memory buffer 230 to generate a shared reference frame for use in the first pass encoding.

[0029] Referring now to FIG. 3, an exemplary multi-pass (e.g., two-pass) encoding operation 300 using a shared reference frame is illustrated. For ease of explanation, the multi-pass encoding operation 300 is described with reference to the computing system 200 of FIG. 2 implemented as the server 102 in the cloud-based gaming system 100 of FIG. 1. In the operation 300, the gaming platform instance 204 renders a set of game frames 305 for a client gaming session associated with a gaming application executing on one or more client systems 112. Each frame of the set of game frames 305 represents at least a portion of a virtual environment associated with the gaming application. To facilitate transmission of the set of game frames 305 to a client device executing the gaming application, the encoder 206 encodes each frame of the set of game frames 305 using a multi-pass encoding operation using one or more shared reference frames. In doing so, the amount of bandwidth required to transmit the encoded set of game frames to the client device more closely matches the actual available bandwidth.

[0030] For a first game frame 310 of the set of game frames 305, the encoder 206 performs shared reference frame acquisition 315 to generate one or more shared reference frames. The shared reference frame acquisition 315 includes acquiring one or more second pass reference frames associated with the first game frame 310, i.e., second pass reference frames used in the second pass encoding 330 of the first game frame 310, from the second pass memory buffer 232. In an embodiment, the shared reference frame acquisition 315 includes the encoder 206 accessing the first pass memory buffer 230, which includes one or more pointers associated with the first game frame 310. For example, the pointers were associated with the first pass encoding 320 of the first game frame 310. The pointers in the first pass memory buffer 230 include the locations of the respective second pass reference frames stored in the second pass memory buffer 232. That is, the pointers in the first pass memory buffer 230 each include a location of a respective second pass reference frame used in the second pass encoding 330 of the first game frame 310. Based on these pointers, the encoder 206 is configured to first retrieve a respective second pass reference frame from the second pass memory buffer 232 to generate one or more retrieved reference frames by simply accessing the first pass memory buffer 230. According to an embodiment, the shared reference frame acquisition 315 further includes the encoder 206 comparing a first pass resolution used for the first pass encoding 320 of the first game frame 310 with a second pass resolution used for the second pass encoding 330 of the first game frame 310. In an embodiment, in response to the first pass resolution being lower than the second pass resolution, the downsampler 234 is configured to downsample the retrieved reference frames to the first pass resolution to generate one or more downsampled (e.g., second pass) reference frames.The encoder 206 then stores the downsampled (e.g., second pass) reference frame in the first pass memory buffer 230 to generate a shared reference frame for use as a first pass reference frame in the first pass encoding 320 of the first game frame 310. In response to the first pass resolution being equal to the second pass resolution, the encoder 206 stores the obtained (e.g., second pass) reference frame in the first pass memory buffer 230 to generate a shared reference frame for use in the first pass encoding 320 of the first game frame 310. In this manner, the encoder 206 generates the shared reference frame based on a comparison of the first pass resolution used for the first pass encoding 320 of the first game frame 310 and the second pass resolution used for the second pass encoding 330 of the first game frame 310. That is, the encoder 206 generates a shared reference frame by storing the obtained second pass reference frame (e.g., downsampled or not downsampled based on the comparison) in a first pass memory buffer, and the shared reference frame is made available as a first pass reference frame to be used in the first pass encoding 320 of the first game frame 310. By using the second pass reference frame also for the first pass encoding 320, the second pass reference frame becomes a shared reference frame used in both the first pass encoding and the second pass encoding. This allows the complexity estimation 325 to more accurately estimate the complexity of the first game frame 310.

[0031] In an embodiment, the first pass encoding 320 includes downsampling the first game frame 310 by reducing the resolution of the first game frame 310 to the first pass resolution, such as by the downsampler 234. After downsampling the first game frame, the encoder 206 encodes the first game frame 310 using the second pass reference frame including the resulting shared reference frame and a predetermined (first) quantization parameter, resulting in a first pass encoded frame. After the first pass encoding 320, the rate control unit 224 determines a complexity estimate 325 based on the first pass encoded frame. By using the shared reference frame in the first pass encoding 320, the bit size of the first pass encoded frame can more accurately reflect the bit size of the encoded frame 335, improving the complexity estimate. In an embodiment, the rate control unit 224 determines the estimated complexity as a scale factor version of the bit size of the first pass encoded frame.

[0032] After the rate control unit 224 determines the estimated complexity from the complexity estimation 325, the encoder 206 performs a second pass encoding 330 of the first game frame 310. The second pass encoding 330 includes determining a quantization parameter (e.g., a second quantization parameter) for the first game frame 310 as a function of the estimated complexity from the complexity estimation 325. From the determined quantization parameter (e.g., the second quantization parameter) based on the second pass encoding, the encoder 206 determines a quantization step size to be used to encode the first game frame 310. The encoder 206 then performs the second pass encoding 330 on the first game frame 310 according to the second pass reference frame and with the determined (second) quantization parameter to generate an encoded frame 335, the encoded frame 335 having a size close to a target bit size, e.g., t in Equation 1. Increasing the accuracy of the estimated complexity by using a shared reference frame in the first pass encoding 320 helps to eliminate the possibility that the encoded frame 335 has a bit size larger than the target bit size.

[0033] As shown with reference to FIG. 4, a method 400 for multi-pass encoding using a shared reference frame is illustrated. For ease of explanation, the method 400 is described with reference to the computing system 200 of FIG. 2 implemented as the server 102 of the system 100 of FIG. 1. At block 405, an encoder the same as or similar to the encoder 206 receives a first game frame. The first game frame represents at least a portion of a virtual environment associated with a client game session. At block 410, the encoder retrieves a second pass reference frame associated with the first game frame from the second pass memory buffer 232 and generates a retrieved reference frame. For example, the encoder retrieves a second pass reference frame used in the second pass encoding of the first game frame. In an embodiment, retrieving the second pass reference frame includes accessing the second pass memory buffer 232 according to one or more pointers stored in the first pass memory buffer 230. The one or more pointers stored in the first path memory buffer 230 include the location of a second path reference frame associated with the first game frame in the second path memory buffer 232 .

[0034] At block 415, the encoder determines whether a first pass resolution associated with the first pass encoding of the multi-pass encoding is lower than a second pass resolution associated with the second pass encoding of the multi-pass encoding. In response to the first pass resolution being lower than the second pass resolution, the computing system 200 proceeds to block 420. At 420, the downsampler 234 downsamples the obtained reference frame to the first pass resolution to generate a downsampled reference frame. The encoder 206 then stores the downsampled reference frame in the first pass memory buffer 230 and generates a shared reference frame for use in the first pass encoding of the first game frame. As shown again with reference to block 410, in response to the first pass resolution being equal to or greater than the second pass resolution, the encoder stores the obtained reference frame in the first pass memory buffer 230 and generates a shared reference frame for use in the first pass encoding of the first game frame. The computing system 200 then proceeds to block 425.

[0035] In block 425, the encoder performs a first pass encoding on the first game frame. In an embodiment, the first pass encoding includes a downsampler 234 that downsamples the first game frame to the first pass resolution, such as by reducing the resolution of the first game frame to the first pass resolution and encoding the first game frame using a shared reference frame. The first pass encoding generates a first pass encoded frame with a large number of bits. In block 430, a rate control unit, the same as or similar to the rate control unit 224, determines an estimated complexity of the first game frame based on the number of bits in the first pass encoded frame. Further in block 430, the rate control unit determines a quantization parameter of the first game frame according to the estimated complexity. By using a shared reference frame in the first pass encoding, the estimated complexity more accurately represents the actual complexity of the first game frame, and the risk of the encoded frame exceeding the target number of bits is reduced. At block 435, the encoder performs a second pass encoding operation on the first game frame using the second pass reference frame and the determined quantization parameter.

[0036] A computer-readable storage medium may include any storage medium, or combination of storage media, that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. A computer-readable storage medium may be embedded in a computing system (e.g., system RAM or ROM), permanently attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or a universal serial bus (USB)-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0037] It should be noted that not all of the activities or elements described above in the general description are required, that some of the specific activities or devices may not be required, and that one or more additional activities may be performed or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, these concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure, as set forth in the following claims. Thus, the specification and drawings should be regarded in an illustrative rather than restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0038] The preposition "or" used in the context of "at least one of A, B, or C" is used herein to mean "inclusive or." That is, in the above and similar contexts, or is used to mean "at least one of them or any combination thereof." For example, "at least one of A, B, and C" is used to mean "at least one of A, B, C, or any combination thereof."

[0039] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any features that cause or make any benefit, advantage, or solution more prominent should not be construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments disclosed above are merely exemplary, and the disclosed subject matter may be modified and implemented in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design shown herein, except as set forth in the following claims. It is therefore apparent that the specific embodiments disclosed above may be altered or modified, and all such variations are deemed to be within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the following claims.

Claims

1. A method for cloud-based video frame encoding executed by one or more processors, comprising: comparing a first pass resolution of a first pass encoding of a multi-pass encoding process with a second pass resolution of a second pass encoding of said multi-pass encoding process; generating a shared reference frame from a second pass reference frame used in the second pass encoding based on the comparison; and performing the first pass encoding on a first video frame of a stream of video frames at the first pass resolution using the shared reference frame to determine an estimated complexity of the first video frame; performing the second pass encoding on the first video frame at the second pass resolution using the second pass reference frame and according to the estimated complexity to generate an encoded frame.

2. The method of claim 1 , further comprising obtaining the second pass reference frame from a memory buffer associated with the second pass encoding.

3. generating the shared frame of reference In response to the first pass resolution being lower than the second pass resolution, 3. The method of claim 2, further comprising: downsampling the second pass reference frame to the first pass resolution and storing the downsampled second pass reference frame in a memory buffer associated with the first pass encoding to generate the shared reference frame.

4. In response to the first pass resolution being equal to the second pass resolution, 4. The method of claim 1, further comprising storing the second pass reference frame in a memory buffer associated with the first pass encoding to generate the shared reference frame.

5. The method of claim 1 , further comprising obtaining the second pass reference frame according to a pointer stored in a memory buffer associated with the first pass encoding.

6. 6. The method of claim 5, wherein the memory buffer associated with the first pass encoding includes a plurality of pointers each associated with a respective second pass reference frame stored in a second memory buffer associated with the second pass encoding.

7. 1. A computer-implemented method comprising: generating a shared frame of reference for use in a first pass encoding of a multi-pass encoding process based on a second pass frame of reference for use in a second pass encoding of the multi-pass encoding process, wherein generating the shared frame of reference includes: comparing a first pass resolution of the first pass encoding with a second pass resolution of the second pass encoding; and downsampling the second pass frame of reference based on the comparison of the first pass resolution and the second pass resolution, wherein generating the shared frame of reference further comprises: performing the first pass encoding on a first video frame of a stream of video frames at the first pass resolution using the shared reference frame to determine an estimated complexity of the first video frame; determining the estimated complexity of the first video frame based on the first pass encoding of the first video frame; and performing the second pass encoding on the first video frame at the second pass resolution using the second pass reference frame and the estimated complexity to generate an encoded frame.

8. generating the shared frame of reference 8. The method of claim 7, further comprising: accessing a memory buffer associated with the second pass encoding according to a pointer in a second memory buffer associated with the first pass encoding.

9. 9. The method of claim 8, wherein the second memory buffer includes a plurality of pointers each associated with a respective second pass reference frame stored in the memory buffer associated with the second pass encoding.

10. generating the shared frame of reference In response to the first pass resolution being lower than the second pass resolution, 10. The method of claim 8 or 9, further comprising downsampling the second pass reference frame to the first pass resolution and storing the downsampled second pass reference frame in the second memory buffer to generate the shared reference frame.

11. generating the shared frame of reference In response to the first pass resolution being equal to the second pass resolution, 10. The method of claim 8 or 9, further comprising storing the second pass reference frame in the second memory buffer associated with the first pass encoding to generate the shared reference frame.

12. The first pass encoding comprises: The method of claim 1 , further comprising downsampling the first video frames to the first pass resolution.

13. The method of claim 1 , further comprising determining a quantization parameter for the first video frame.

14. The method of claim 13 , wherein the second pass encoding is further performed based on the quantization parameter.

15. 4. The method of claim 1, wherein performing the first pass encoding produces a first pass encoded frame having a bit size, and the estimated complexity is based on the bit size.

16. The method of claim 1 , wherein the first video frame represents at least a portion of a virtual environment of a gaming session associated with a client system.

17. encoding the stream of video frames according to the multi-pass encoding process to generate an encoded stream of video frames; 20. The method of claim 16, further comprising transmitting the encoded stream of video frames to the client system.

18. A method for decoding said stream of video frames, comprising:

20. The method of claim 17, further comprising: displaying the decoded stream of video frames.

19. 20. The method of claim 18, further comprising rendering the stream of video frames in accordance with a client game session associated with the client system.

20. The method of claim 1 , wherein the first video frame is rendered upon which the first pass encoding and the second pass encoding are performed.

21. 4. The method of claim 1, further comprising: encoding the first video frames according to a rate control scheme such that the encoded frames of the second pass encoding are compressed to fit a predetermined target bit size.

22. communicatively coupled to one or more client systems, each of which one or more processors; and a memory coupled to the one or more processors and storing executable instructions configured to operate the one or more processors to perform the method of any of claims 1 to 3.

23. 1. A cloud-based system, comprising: a network interface capable of being coupled to a network; and an encoder coupled to the network interface, the cloud-based gaming system configured to perform the method according to any one of claims 1 to 3.

Images