Providing scene change hints for encoding a game application at a cloud game server

Abstract

A method for encoding includes executing game logic built on a game engine of a video game to generate video frames at a cloud game server. The method includes executing scene change logic to predict a scene change in the video frames based on game state collected during execution of the game logic. The method includes identifying a range of video frames predicted to include the scene change. The method includes generating a scene change hint using the scene change logic, where the scene change hint identifies the range of video frames, where the range of video frames includes a first video frame. The method includes delivering the first video frame to an encoder. The method includes sending the scene change hint from the scene change logic to the encoder. The method includes encoding the first video frame as an I-frame based on the scene change hint.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to streaming systems configured for streaming content across a network, and more particularly to providing scene change hints to an encoder for video frames generated during execution of a video game in order to reduce latency between a cloud gaming server and a client, provide consistent latency between the cloud gaming server and the client, and improve the smoothness of the client display of the video. BACKGROUND

[0002] In recent years, there has been a constant push for online services that allow online or cloud gaming in a streaming format between a cloud gaming server and a client connected through a network. Streaming format is increasingly popular because of the availability of game titles on demand, the ability to network between players for multiplayer games, sharing of assets between players, sharing of instant experiences between players and / or spectators, allowing friends to watch one friend play a video game, letting one friend join another friend in a game in progress, etc. Unfortunately, this demand is also pushing the limits of network connection capabilities and the responsiveness of the processing performed at the server and the client to render high quality images upon delivery to the client. For example, the results of all game activities performed on the server need to be compressed and transmitted back to the client with low millisecond latency for the best user experience. The round-trip latency can be defined as the total time between a user’s controller input and the display of a video frame at the client; it can include the processing and transmission of control information from the controller to the client, the processing and transmission of control information from the client to the server, the use of the input at the server to generate a video frame in response to the input, the processing of the video frame and its transfer to an encoding unit (e.g. scan-out), the encoding of the video frame, the transmission of the encoded video frame back to the client, the reception and decoding of the video frame and any processing or staging of the video frame before its display. The one-way latency can be defined as the portion of the round-trip latency that consists of the time from when the video frame is transferred to an encoding unit (e.g. scan-out) at the server to when the video frame is displayed at the client. The portion of the round-trip and one-way latency is associated with the time it takes for data to flow through the communication network from the client to the server and from the server to the client. The other portion is associated with the processing at the client and the server; improvements in these operations, such as advanced strategies related to frame decoding and display, can result in a significant reduction in the round-trip and one-way latency between the server and the client and provide a higher quality experience for users of the cloud gaming service.

[0003] It is in this context that embodiments of the present disclosure arise. SUMMARY

[0004] Embodiments of the present disclosure relate to streaming systems configured for streaming content (e.g., games) across a network, and more particularly to providing scene change hints to an encoder for video frames generated during execution of a video game in order to reduce latency and provide more consistent latency between a cloud game server and a client, as well as to improve smoothness of client display of the video.

[0005] Embodiments of the present disclosure disclose a method for encoding. The method includes executing game logic built on a game engine of a video game at a cloud game server to generate a plurality of video frames. The method includes executing scene change logic to predict a scene change in the plurality of video frames, wherein the prediction is based on game state collected during execution of the game logic. The method includes identifying a range of video frames in the plurality of video frames predicted to include the scene change. The method includes generating a scene change hint using the scene change logic, wherein the scene change hint identifies the range of video frames, wherein the range of video frames includes a first video frame. The method includes delivering the first video frame to an encoder. The method includes sending the scene change hint from the scene change logic to the encoder. The method includes encoding the first video frame as an I-frame based on the scene change hint.

[0006] In another embodiment, a non-transitory computer-readable medium storing a computer program for encoding is disclosed. The computer-readable medium includes program instructions for executing game logic built on a game engine of a video game at a cloud game server to generate a plurality of video frames. The computer-readable medium includes program instructions for executing scene change logic to predict a scene change in the plurality of video frames, wherein the prediction is based on game state collected during execution of the game logic. The computer-readable medium includes program instructions for identifying a range of video frames in the plurality of video frames predicted to include the scene change. The computer-readable medium includes program instructions for generating a scene change hint using the scene change logic, wherein the scene change hint identifies the range of video frames, wherein the range of video frames includes a first video frame. The computer-readable medium includes program instructions for delivering the first video frame to an encoder. The computer-readable medium includes program instructions for sending the scene change hint from the scene change logic to the encoder. The computer-readable medium includes program instructions for encoding the first video frame as an I-frame based on the scene change hint.

[0007] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to perform a method for encoding. The method includes executing game logic built on a game engine of a video game at a cloud game server to generate a plurality of video frames. The method includes executing scene change logic to predict a scene change in the plurality of video frames, where the prediction is based on game state collected during execution of the game logic. The method includes identifying a range of video frames in the plurality of video frames predicted to include the scene change. The method includes generating a scene change hint using the scene change logic, where the scene change hint identifies the range of video frames, where the range of video frames includes a first video frame. The method includes delivering the first video frame to an encoder. The method includes sending the scene change hint from the scene change logic to the encoder. The method includes encoding the first video frame as an I-frame based on the scene change hint.

[0008] Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrated by way of example in the following description and is set forth in the claims. BRIEF DESCRIPTION OF DRAWINGS

[0009] The disclosure can be best understood by reference to the following description taken in conjunction with the accompanying drawing figures in which:

[0010] Figure 1A is a diagram of a VSYNC signal at the beginning of a frame period according to one embodiment of the disclosure.

[0011] Figure 1B is a diagram of the frequency of a VSYNC signal according to one embodiment of the disclosure.

[0012] Figure 2A is a diagram of a system for providing a game between one or more cloud game servers and one or more client devices over a network in various configurations according to one embodiment of the disclosure, where VSYNC signals can be synchronized and offset to reduce one-way latency.

[0013] Figure 2B is a diagram of a system for providing a game between two or more peer devices according to one embodiment of the disclosure, where VSYNC signals can be synchronized and offset to achieve optimal timing of receiving controllers and other information between devices.

[0014] Figure 2CVarious network configurations that benefit from proper synchronization and offset of VSYNC signals between a source device and a target device are shown in accordance with one embodiment of the disclosure.

[0015] Figure 2D A multi-tenant configuration between a cloud game server and multiple clients that benefit from proper synchronization and offset of VSYNC signals between a source device and a target device is shown in accordance with one embodiment of the disclosure.

[0016] Figure 3 Changes in one-way latency between a cloud game server and a client due to clock drift when streaming video frames generated from a video game executing on the server are shown in accordance with one embodiment of the disclosure.

[0017] Figure 4 A network configuration including a cloud game server and a client when streaming video frames generated from a video game executing on a server, synchronization and offset of VSYNC signals between the server and the client to allow overlapping operations at the server and the client, and to reduce one-way latency between the server and the client are shown.

[0018] Figure 5A A cloud game server system configured for use of scene change cues when encoding video frames when streaming content from a video game executing on the cloud game server to a client across a network is shown in accordance with one embodiment of the disclosure.

[0019] Figure 5B A scan-out operation performed to generate modified video frames for delivery to an encoder when streaming content from a video game executing at a cloud game server to a client across a network in accordance with one embodiment of the disclosure, where the encoder is configured for use of scene change cues when encoding corresponding video frames is shown.

[0020] Figure 6A A flowchart showing a method for encoding video frames when performing cloud gaming in accordance with one embodiment of the disclosure, where one or more scene change cues are used by an encoder to process corresponding video frames is shown.

[0021] Figure 6B A flowchart showing a method for encoding video frames when performing cloud gaming in accordance with one embodiment of the disclosure, including generating one or more scene change cues when executing a video game or game application that is available for efficient encoding of corresponding video frames is shown.

[0022] Figure 7 Components of an exemplary device that can be used to perform aspects of the various embodiments of the disclosure are shown. DETAILED DESCRIPTION

[0023] While the following detailed description includes numerous specific details for the purposes of explanation and by way of example, one of ordinary skill in the art will appreciate that many modifications and substitutions can be made in the details without departing from the scope of the disclosure. Accordingly, one should not limit the aspects of the disclosure described below to any particular description without losing the generality of the disclosure.

[0024] Generally, various embodiments of the present disclosure describe methods and systems configured to reduce latency and / or reduce latency instability between a source device and a target device when streaming media content (e.g., streaming audio and video from a video game). Latency instability can be introduced in the one-way latency between a server and a client due to additional time required to generate complex frames (e.g., scene changes) at the server, increased time for encoding / compressing complex frames at the server, variable communication paths on the network, and increased time for decoding complex frames at the client. The introduction of latency instability can also be due to clock differences at the server and the client, which can cause drift between the server VSYNC signal and the client VSYNC signal. In embodiments of the present disclosure, the one-way latency between a server and a client can be reduced in a cloud gaming application by providing a scene change hint to an encoder for video frames generated during execution of a video game. The encoder can perform different encoding for video frames that are scene changes (e.g., switch to I-frames instead of encoding with P-frames in MPEG compression format used to stream content between a cloud gaming server and a client). In particular, this scene change hint can be used in the encoding process when the game application notifies the encoder that a scene change has occurred (e.g., through an API, such as a GPU API). In this way, because the scene change hint is provided to the encoder, the encoder does not need to detect scene changes, which can introduce jitter and increased latency for scene change detection and re-encoding of video frames with higher complexity. That is, the encoder can directly encode video frames that are identified as scene changes using the scene change hint with appropriate complexity (e.g., I-frames) without needing one or more additional frame periods for scene change detection and re-encoding of video frames with higher complexity. This results in reduced one-way latency, smoother frame rate, and more reliable and / or consistent one-way latency between a cloud gaming server and a client.

[0025] With the above general understanding of various embodiments, exemplary details of embodiments will now be described with reference to the various drawings.

[0026] Throughout this specification, reference to a "game" or "video game" or "gaming application" is intended to mean any type of interactive application that is directed by the execution of input commands. For illustrative purposes only, interactive applications include applications for gaming, word processing, video processing, video game processing, etc. Moreover, the above-introduced terms are interchangeable.

[0027] Cloud gaming includes executing a video game at a server to generate game-rendered video frames that are then sent to a client for display. Operation timing at both the server and the client can be associated with respective vertical sync (VSYNC) parameters. When the VSYNC signals are properly synchronized and / or offset between the server and / or the client, the operations performed at the server (e.g., generating and transmitting video frames over one or more frame periods) are synchronized with the operations performed at the client (e.g., displaying video frames on a display at a display frame or refresh rate that corresponds to the frame period). In particular, a server VSYNC signal generated at the server and a client VSYNC signal generated at the client can be used to synchronize the operations at the server and the operations at the client. That is, when the server VSYNC signal and the client VSYNC signal are synchronized and / or offset, the server generates and sends video frames that are synchronized with the way those video frames are displayed by the client.

[0028] VSYNC signaling and vertical blanking intervals (VBIs) have been combined for generating video frames and displaying those video frames when streaming media content between a server and a client. For example, a server seeks to generate a game-rendered video frame within one or several frame periods defined by a corresponding server VSYNC signal (e.g., generating one video frame per frame period results in a 60 Hz operation and generating one video frame every two frame periods results in a 30 Hz operation if the frame period is 16.7 milliseconds), and subsequently encodes and transmits the video frame to a client. At the client, the received encoded video frame is decoded and displayed, where the client displays each video frame rendered for display beginning with a corresponding client VSYNC.

[0029] To illustrate, Figure 1A It is shown how a VSYNC signal 111 can indicate the beginning of a frame period, where various operations can be performed at the server and / or the client during the corresponding frame period. When streaming media content, a server can use a server VSYNC signal to generate video frames and encode the video frames, and a client can use a client VSYNC signal to display the video frames. The VSYNC signal 111 is generated at a defined frequency that corresponds to the defined frame period 110, as shown in FIG. 1. The server VSYNC signal and the client VSYNC signal are synchronized and / or offset, as shown in FIG. 1, to synchronize the operations performed at the server and the operations performed at the client. Figure 1BThe VBI 105 defines a period of time between when the last raster line of the current frame is drawn on the display and when the first raster line (e.g., the top) is drawn to the display. As shown, after the VBI 105, the video frame that is rendered for display is shown by raster scan line 106 (e.g., from left to right, raster line by raster line).

[0030] Further, various embodiments of the present disclosure are disclosed for reducing one-way latency and / or reducing latency instability between a source device and a target device, such as when streaming media content (e.g., video game content). For illustrative purposes, various embodiments for reducing one-way latency and / or reducing latency instability within a server and client network configuration are described. However, it should be appreciated that various techniques disclosed for reducing one-way latency and / or reducing latency instability can be implemented within other network configurations and / or over a peer-to-peer network, such as Figures 2A-2D For illustrative purposes, various embodiments for reducing one-way latency and / or reducing latency instability within a server and client network configuration are described. However, it should be appreciated that various techniques disclosed for reducing one-way latency and / or reducing latency instability can be implemented within other network configurations and / or over a peer-to-peer network, such as

[0031] Figure 2A is an illustration of a system 200A for providing games in various configurations between one or more cloud gaming networks 290 and / or servers 260 and one or more client devices 210 over a network 250 in accordance with an embodiment of the present disclosure, where server VSYNC signals and client VSYNC signals can be synchronized and offset, and / or where dynamic buffering is performed on the client, and / or where encoding and transmission operations on the server can overlap, and / or where receiving and decoding operations at the client can overlap, and / or where decoding and display operations on the client can overlap to reduce one-way latency between the server 260 and the client 210. In particular, system 200A provides games over a cloud gaming network 290 in accordance with an embodiment of the present disclosure, where the games are being executed remotely from the client devices 210 (e.g., thin clients) of the corresponding users playing the games. System 200A can provide game control to one or more users playing one or more games over the cloud gaming network 290 in single player mode or multi-player mode over the network 250. In some embodiments, the cloud gaming network 290 can include a plurality of virtual machines (VMs) running on a hypervisor of a host machine, where one or more virtual machines are configured to execute game processor modules with hardware resources available to the hypervisor of the host machine. The network 250 can include one or more communication technologies. In some embodiments, the network 250 can include fifth generation (5G) network technology with advanced wireless communication systems.

[0032] In some embodiments, communications can be facilitated using wireless technology. Such technology can include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which service areas covered by a provider are divided into small geographic areas called cells. Analog signals representing sounds and images are digitized in the phone, converted by an analog-to-digital converter and transmitted as a bitstream. All 5G wireless devices in a cell communicate through radio waves over a frequency channel from a local array of antennas and low-power automatic transceivers (transmitters and receivers) assigned from a pool of frequencies that are reused in other cells. The local antennas are connected with the phone network and the Internet through high-bandwidth fiber-optic or wireless backhaul. As with other cell networks, a mobile device crossing from one cell to another is automatically transferred to the new cell. It will be appreciated that 5G networks are just one example type of communication network, and embodiments of the present disclosure can utilize previous generation wireless or wired communications, as well as next generation wired or wireless technologies after 5G.

[0033] As shown, the cloud gaming network 290 includes game servers 260 that provide access to a plurality of video games. The game servers 260 can be any type of server computing device available in the cloud and can be configured as one or more virtual machines executing on one or more hosts. For example, the game servers 260 can manage virtual machines that support game processors instantiated for users. In this way, a plurality of game processors of the game servers 260 associated with a plurality of virtual machines are configured to execute a plurality of instances of one or more games associated with game play of a plurality of users. In this way, the backend servers support streaming of media (e.g., video, audio, etc.) of game play of a plurality of game applications to a plurality of corresponding users. That is, the game servers 260 are configured to stream data (e.g., rendered images and / or frames of corresponding game play) back to corresponding client devices 210 over the network 250. In this way, computationally complex game applications can be executed at the backend servers in response to controller inputs received and forwarded by the client devices 210. Each server is able to render images and / or frames, which are then encoded (e.g., compressed) and streamed to the corresponding client device for display.

[0034] For example, multiple users can access a cloud gaming network 290 through the communication network 250 using corresponding client devices 210 configured for receiving streaming media. In one embodiment, the client devices 210 can be configured as thin clients providing interface connections with back-end servers (e.g., game servers 260 of the cloud gaming network 290) configured for providing computing functionality (e.g., including a game title processing engine 211). In another embodiment, the client devices 210 can be configured with a game title processing engine and game logic for at least some local processing of a video game, and can further be used to receive streaming content generated by the video game executed at a back-end server, or other content provided by the back-end server. For local processing, the game title processing engine includes basic processor-based functionality for executing a video game and services associated with the video game. The game logic is stored on the local client device 210 and used to execute the video game.

[0035] In particular, a client device 210 of a corresponding user (not shown) is configured for requesting access to a game through the communication network 250, such as the Internet, and for rendering display images generated by a video game executed by the game server 260, where encoded images are delivered to the client device 210 for display in association with the corresponding user. For example, a user can interact through the client device 210 with an instance of a video game executed on a game processor of the game server 260. More specifically, the instance of the video game is executed by the game title processing engine 211. Corresponding game logic (e.g., executable code) 215 implementing the video game is stored and accessible through a data store (not shown), and used to execute the video game. The game title processing engine 211 is capable of supporting multiple video games using multiple game logic, each of which can be selected by a user.

[0036] For example, the client device 210 is configured to interact with the game name processing engine 211 in association with game play of a corresponding user, such as through input commands for driving the game play. In particular, the client device 210 can receive input from various types of input devices, such as game controllers, tablets, keyboards, gestures captured by a video camera, mice, touch pads, etc. The client device 210 can be any type of computing device having at least a memory and a processor module capable of connecting to the game server 260 over the network 250. The backend game name processing engine 211 is configured for generating rendered images that are delivered over the network 250 for display at a corresponding display in association with the client device 210. For example, through a cloud-based service, the game rendered images can be delivered by an instance of the corresponding game executing on the game execution engine 211 of the game server 260. That is, the client device 210 is configured for receiving encoded images (e.g., encoded from game rendered images generated by executing a video game) and for displaying the images rendered as display 11. In one embodiment, the display 11 includes an HMD (e.g., displaying VR content). In some embodiments, the rendered images can be streamed to a smartphone or tablet in a wireless or wired manner, directly from the cloud-based service or through the client device 210 (e.g., Remote Play). Remote Play) to a smartphone or tablet.

[0037] In one embodiment, the game server 260 and / or the game name processing engine 211 includes processor-based functions for executing the game and services associated with the game application. For example, the processor-based functions include 2D or 3D rendering, physics, physics simulation, script processing, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. Further, the services of the game application include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with a social network of friends, communication channels, texting, instant messaging, chat support, etc.

[0038] In one embodiment, the cloud game network 290 is a distributed game server system and / or architecture. In particular, a distributed game engine that executes game logic is configured as a corresponding instance for a corresponding game. Generally, the distributed game engine takes each function of a game engine and distributes those functions for execution by multiple processing entities. Individual functions can be further distributed across one or more processing entities. The processing entities can be configured in different configurations, including physical hardware, and / or as virtual components or virtual machines, and / or as virtual containers, where a container is different from a virtual machine in that a container virtualizes an instance of a game application running on a virtualized operating system. The processing entities can utilize and / or rely on servers and their underlying hardware on one or more servers (compute nodes) of the cloud game network 290, where the servers can be located on one or more racks. Coordination, distribution, and management of these functions for various processing entities is performed by a distribution synchronization layer. In this way, execution of those functions is controlled by the distribution synchronization layer to implement generation of media (e.g., video frames, audio, etc.) for the game application in response to player controller inputs. The distribution synchronization layer is able to efficiently execute (e.g., through load balancing) those functions across the distributed processing entities such that critical game engine components / functions are distributed and reassembled for more efficient processing.

[0039] The game name processing engine 211 includes a central processing unit (CPU) and a group of graphics processing units (GPUs) that can be configured to perform multi-tenant GPU functions. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application executing on a corresponding CPU.

[0040] Figure 2B is an illustration of providing a game between two or more peer devices in accordance with one embodiment of the disclosure, where VSYNC signals can be synchronized and offset to achieve optimal timing of receiving controllers and other information between the devices. For example, a head-to-head game can be executed using two or more peer devices connected through a network 250 or directly through peer-to-peer communication (e.g., Bluetooth, local area network, etc.).

[0041] As shown, the game is executed locally in each of the corresponding user's client devices 210 (e.g., game consoles) that are playing the video game, where the client devices 210 are in communication through a peer-to-peer network. For example, an instance of the video game is executed by the game name processing engine 211 of the corresponding client device 210. Game logic 215 (e.g., executable code) that implements the video game is stored on the corresponding client device 210 and used to execute the game. For illustration purposes, the game logic 215 can be delivered to the corresponding client device 210 through portable media (e.g., optical media) or through a network (e.g., downloaded from a game provider over the Internet).

[0042] In one embodiment, the game name processing engine 211 of the corresponding client device 210 includes processor-based functionality for executing games and services associated with the game application. For example, the processor-based functionality includes 2D or 3D rendering, physics, physics simulation, script processing, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. In addition, the services of the game application include memory management, multi-threading management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with the social network of friends, communication channels, texting, instant messaging, chat support, etc.

[0043] The client device 210 can receive input from various types of input devices, such as game controllers, tablets, keyboards, gestures captured by a video camera, mice, touchpads, etc. The client device 210 can be any type of computing device having at least a memory and a processor module, and configured for generating rendered images executed by the game name processing engine 211 and for displaying the rendered images on a display (e.g., the display 11, or a display 11 including a head-mounted display (HMD), etc.). For example, the rendered images can be associated with an instance of a game being executed locally on the client device 210 to implement game play of a corresponding user, such as through input commands for driving the game play. Some examples of the client device 210 include a personal computer (PC), a game console, a home theater device, a general purpose computer, a mobile computing device, a tablet, a phone, or any other type of computing device that can execute an instance of a game.

[0044] Figure 2C Various network configurations that benefit from proper synchronization and offset of VSYNC signals between a source device and a target device are shown in accordance with embodiments of the present disclosure, including Figures 2A-2BThe illustrated configurations. In particular, the various network configurations benefit from proper alignment of the frequency of the server VSYNC signal and the client VSYNC signal, and a timing offset of the server VSYNC signal and the client VSYNC signal, in order to reduce the one-way latency between the server and the client and / or reduce the latency variability between the two. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration. In one embodiment, the client can include a WebRTC client configured to provide audio and video communication inside a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Another network device configuration includes a client (e.g., source) to client (target) configuration, where the clients can each be a game console providing, for example, a competitive game.

[0045] In particular, alignment of the VSYNC signals can include synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and can also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to eliminate drift, and / or to maintain a desired relationship between the server VSYNC signal and the client VSYNC signal to reduce one-way latency and / or reduce latency variability. In one embodiment, to achieve proper alignment, the server VSYNC signal can be tuned to achieve proper alignment between the pair of server 260 and client 210. In another embodiment, the client VSYNC signal can be tuned to achieve proper alignment between the pair of server 260 and client 210. Once the client VSYNC signal and the server VSYNC signal are aligned, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency, and are offset from each other by a timing offset that can be adjusted from time to time. In another embodiment, alignment of the VSYNC signals can include synchronizing the VSYNC frequencies of two clients, and can also include adjusting the timing offset between their VSYNC signals to eliminate drift and / or to achieve optimal timing of receipt of control and / or other information; either VSYNC signal can be tuned to achieve such alignment. In yet another embodiment, alignment can include synchronizing the VSYNC frequencies of multiple servers, and can also include synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal and adjusting the timing offset between the client VSYNC signal and the server VSYNC signal, e.g., for a battle cloud gaming. In server-to-client and client-to-client configurations, alignment can include both frequency synchronization between the server VSYNC signal and the client VSYNC signal and providing proper timing offset between the server VSYNC signal and the client VSYNC signal. In server-to-server configurations, alignment can include frequency synchronization between the server VSYNC signal and the client VSYNC signal without setting a timing offset.

[0046] Figure 2D A multi-tenant configuration between a cloud gaming server 260 and one or more clients 210 that benefit from proper synchronization and offset of VSYNC signals between a source device and a target device is shown in accordance with one embodiment of the present disclosure. In server-to-client configurations, alignment can include both frequency synchronization between the server VSYNC signal and the client VSYNC signal and providing proper timing offset between the server VSYNC signal and the client VSYNC signal. In one embodiment, in multi-tenant configurations, the client VSYNC signal is tuned at each client 210 to achieve proper alignment between the pair of server 260 and client 210.

[0047] For example, in one embodiment, a graphics subsystem can be configured to perform multi-tenancy GPU functionality, where one graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, the graphics subsystem is shared among multiple games that are being executed. In particular, in one embodiment, a game title processing engine can include a CPU and GPU bank configured to perform multi-tenancy GPU functionality, where one CPU and GPU bank can implement graphics and / or rendering pipelines for multiple games. That is, the CPU and GPU bank is shared among multiple games that are being executed. The CPU and GPU bank can be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application executing on a corresponding CPU.

[0048] Figure 3A general procedure is shown for executing a video game at a server to generate game rendered video frames and sending these video frames to a client for display. Conventionally, multiple operations at the game server 260 and client 210 are performed within frame periods defined by respective VSYNC signals. For example, the server 260 seeks to generate a game rendered video frame in one or more frame periods defined by a corresponding server VSYNC signal 311 at 301. The video frame is generated by the game in response to control information (e.g., input commands of a user) delivered from an input device at operation 350, or not driven by control information game logic. There can be transmission jitter 351 when control information is sent to the server 260, where the jitter 351 measures variations in network latency from the client to the server (e.g., when input commands are sent). As shown, a thick arrow shows a current delay when control information is sent to the server 260, but due to jitter, there can be a range of arrival times (e.g., a range bounded by dashed arrows) of the control information at the server 260. At flip time 309, the GPU arrives at a flip command, which indicates that a corresponding video frame has been completely generated and placed into a frame buffer at the server 260. Thereafter, the server 260 performs scan-out / scan-in for the video frame (operation 302, where scan-out can be aligned with the server VSYNC signal 311) within a subsequent frame period defined by the server VSYNC signal 311 (VBI is omitted for clarity). Subsequently, the video frame is encoded (operation 303) (e.g., encoding starts after the occurrence of the server VSYNC signal 311, and the end of encoding can not be aligned with the server VSYNC signal) and transmitted (operation 304) to the client 210 (where transmission can not be aligned with the server VSYNC signal 311). At the client 210, the encoded video frame is received (operation 305) (where reception can not be aligned with the client VSYNC signal 312), decoded (operation 306) (where decoding can not be aligned with the client VSYNC signal 312), buffered, and displayed (operation 307) (where the start of display can be aligned with the client VSYNC signal 312). In particular, the client 210 displays each video frame that is rendered for display, which starts with a corresponding occurrence of the client VSYNC signal 312.

[0049] One-way latency 315 can be defined as the latency from the encoding unit (e.g., scan-out 302) at the server where the video frame is transmitted to the display of the video frame at the client 307. That is, the one-way latency is the time from the server scan-out to the client display, taking into account the client buffering. Individual frames have a latency from scan-out 302 to the completion of decoding 306, which can vary from frame to frame due to server operations such as encoding 303 and transmission 304, network transmission between the server 260 and the client 210 with accompanying jitter 352, and the highly variable client reception 305. Transmission jitter 352 measures the variation in one-way latency from the server 260 to the client 210, with lower jitter values exhibiting a more stable connection. The variation in latency can be due to operations at the server that exceed the frame period and networking issues that introduce latency when transmitting the video frames to the client 210. As shown, the straight bold arrow illustrates the current latency when the corresponding video frame is sent to the client 210, but due to jitter 352, there can be a range of arrival times (e.g., the range bounded by the dashed arrows) of the video frame at the client 210. Since the one-way latency must be relatively stable (e.g., remain fairly consistent) to achieve a good gaming experience, the result of the buffering 320 is traditionally that the display of individual frames with low latency (e.g., from scan-out 302 to the completion of decoding 306) is delayed by several frame periods. That is, if there is network instability or unpredictable encoding / decoding times, additional buffering is needed so that the one-way latency remains consistent.

[0050] According to one embodiment of the disclosure, the one-way latency between the cloud gaming server and the client can vary in part due to clock drift when streaming video frames generated from a video game executing on the server. That is, the frequency difference between the server VSYNC signal 311 and the client VSYNC signal 312 can cause the client VSYNC signal to drift 390 relative to the frames arriving from the server 260. The drift 390 between the relative timing of the server VSYNC signal 311 and the client VSYNC signal 312 can be due to a very slight difference in the crystal oscillator used in each of the respective clocks at the server and the client. The variation in latency can be extended by the reception and decoding operations at the client 210, where one or more buffers 320 can be implemented to address the variation in one-way latency. Further, the one-way latency can be reduced by performing one or more synchronization and offset of the VSYNC signals to align between the server and the client, providing dynamic buffering on the client, overlapping encoding and transmission of video frames at the server, overlapping reception and decoding of video frames at the client, and overlapping decoding and display of video frames at the client

[0051] Further, during encoding of video frames (operation 303), in prior art, an encoder determines how much change exists between a current video frame being encoded and one or more previously encoded frames to determine whether a scene change exists (e.g., corresponding to a complex image generated for the video frame). That is, a scene change hint can be inferred from the difference between the current frame to be encoded and the previously encoded frames. When content is streamed from a server to a client over a network, an encoder at the server can decide to encode video frames detected as a scene change with a complexity. Otherwise, the encoder will encode video frames not detected as a scene change with a lower complexity. However, detection of a scene change at the encoder can take up to one frame period (e.g., adding jitter) because the video frame is initially encoded with a lower complexity (in a first frame period), but then re-encoded with a higher complexity (in a second frame period) once it is determined that a scene change exists. Further, detection of a scene change can be unnecessarily triggered (such as by a slight explosion in an image) because the difference between the current encoded video frame and the previously encoded video frame can exceed a threshold difference value even though a scene change does not exist. Thus, when a scene change is detected at the encoder, additional latency is introduced at the encoder due to jitter to accommodate performing scene change detection and re-encoding the video frame with a higher complexity.

[0052] Figure 4 A data flow is shown when streaming video frames generated from a video game executing on a server through a network configuration including a highly optimized cloud game server 260 and a highly optimized client 210, in accordance with embodiments of the present disclosure, where overlapping server operations and client operations reduce one-way latency, and synchronization and offset of VSYNC signals between the server and the client reduce one-way latency and reduce variability of one-way latency between the server and the client. In particular, Figure 4 A desired alignment between the server VSYNC signal and the client VSYNC signal is shown. In one embodiment, such as in a server and client network configuration, tuning of the server VSYNC signal 311 is performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal. In another embodiment, such as in a multi-tenant server to multiple clients network configuration, tuning of the client VSYNC signal 312 is performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal. For illustration purposes, in the server and client network configuration, the server VSYNC signal 311 is shown as being tuned to align with the client VSYNC signal 312. Figure 4Tuning of the server VSYNC signal 311 to synchronize the frequency of the server VSYNC signal and the client VSYNC signal, and / or adjust the timing offset between the corresponding client VSYNC signal and the server VSYNC signal, is described in the middle, but it should be understood that the client VSYNC signal 312 can also be used for tuning. In the context of this patent, “synchronize” should be understood to mean tuning the signals so that their frequencies match, but the phases can be different; “offset” should be understood to mean a time delay between the signals, e.g., the time between when one signal reaches its maximum and when the other signal reaches its maximum.

[0053] As shown, Figure 4 An improved process is shown in which a video game is executed at a server in embodiments of the disclosure to generate rendered video frames and send those video frames to a client for display. The process is shown with respect to generating and displaying a single video frame at the server and the client. In particular, the server generates a game rendered video frame at 401. For example, the server 260 includes a CPU configured to execute the game (e.g., game name processing engine 211). The CPU generates one or more draw calls for the video frame, where a draw call includes commands placed into a command buffer for a corresponding GPU of the server 260 to execute in a graphics pipeline. The graphics pipeline can include one or more shader programs that are placed on the vertices of objects within a scene to generate texture values for rendering of the video frame for display, where the operations are performed in parallel by the GPU for efficiency. At flip time 409, the GPU reaches a flip command in the command buffer that indicates that the corresponding video frame has been completely generated and / or rendered and placed into a frame buffer at the server 260.

[0054] At 402, the server performs scan-out of the game rendered video frame to an encoder. In particular, scan-out is performed scan line by scan line or in groups of consecutive scan lines, where a scan line refers to a single horizontal line of a screen, e.g., of a display, from edge to edge of the screen. These scan lines or groups of consecutive scan lines are sometimes referred to as slices, and are referred to as screen slices in this specification. In particular, scan-out 402 can include multiple processes that modify the game rendered frame, including: covering it with another frame buffer, or shrinking it so that it is surrounded by information from another frame buffer. During scan-out 402, the modified video frame is then scanned into an encoder for compression. In one embodiment, scan-out 402 is performed at the occurrence 311a of the VSYNC signal 311. In other embodiments, scan-out 402 can be performed before the VSYNC signal 311 occurs, such as at flip time 409.

[0055] At 403, the game rendered video frame (which can have undergone modifications) is encoded at the encoder on a per-encoder slice basis to generate one or more encoded slices, where the encoded slices are not tied to scanlines or screen slices. Thus, the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scanout 402 process for the corresponding video frame has completed. Furthermore, the start and / or end of the encoding 403 can or can not be aligned with the server VSYNC signal 311. The boundaries of the encoded slices are not limited to a single scanline and can be comprised of a single scanline or multiple scanlines. Furthermore, the end of an encoded slice and / or the start of the next encoder slice can not necessarily occur at the edge of the display screen (e.g., can occur somewhere in the middle of the screen or in the middle of a scanline), such that the encoded slices do not need to traverse the display screen completely from edge to edge. As shown, one or more encoded slices can be compressed and / or encoded, including the compressed “encoded slice A” with hash marks.

[0056] At 404, the encoded video frame is transmitted from the server to the client, where the transmission can occur on a per-encoded slice basis, where each encoded slice is an encoder slice that has been compressed. In one embodiment, the transmission process 404 begins before the encoding process 403 for the corresponding video frame has completed. Furthermore, the start and / or end of the transmission 404 can or can not be aligned with the server VSYNC signal 311. As shown, the compressed encoded slice A is transmitted to the client independently of other compressed encoder slices of the rendered video frame. The encoder slices can be transmitted one at a time or in parallel.

[0057] At 405, the client again receives the compressed video frame on a per-encoded slice basis. Furthermore, the start and / or end of the receiving 405 can or can not be aligned with the client VSYNC signal 312. As shown, the compressed encoded slice A is received by the client. Transmission jitter 452 can exist between the server 260 and the client 210, where the jitter 452 measures the variation in network latency from the server 260 to the client 210. Lower jitter values exhibit a more stable connection. As shown, the thick straight arrow shows the current latency when the corresponding video frame is sent to the client 210, but due to jitter, there can be a range of arrival times (e.g., the range bounded by the dashed arrows) of the video frame to the client 210. The variation in latency can also be due to one or more operations at the server, such as the encoding 403 and the transmission 404, as well as networking issues that introduce latency in transmitting the video frame to the client 210.

[0058] At 406, the client again decodes the compressed video frame on a per- encoded slice basis, resulting in a decoded rendered video frame A that is now ready for display (displayed without hash marks). In one embodiment, the decoding process 406 begins after the receiving process 405 for the corresponding video frame has completed entirely. Further, the beginning and / or end of the decoding 406 can or can not be aligned with the client VSYNC signal 312. At 407, the client displays the decoded rendered video frame on a display at the client. That is, the decoded video frame is placed in a display buffer, which is streamed to, for example, a display device on a per-scan line basis. In one embodiment, the display process 407 (i.e., the streaming to the display device) begins after the decoding process 406 for the corresponding video frame has completed entirely (i.e., the decoded video frame resides entirely in the display buffer). In another embodiment, the display process 407 begins before the decoding process 406 for the corresponding video frame has completed entirely. That is, the streaming to the display device begins from the address of the display buffer each time, at which point only a portion of the decoded frame buffer resides in the display buffer. The display buffer is then timely updated or filled with the remaining portion of the corresponding video frame for display, such that the updating of the display buffer is performed before those portions are streamed to the display. Further, the beginning and / or end of the display 407 is aligned with the client VSYNC signal 312.

[0059] In one embodiment, the one-way latency 416 between the server 260 and the client 210 can be defined as the time elapsed between when the scan-out 402 begins and when the display 407 begins. Embodiments of the present disclosure are capable of aligning (e.g., synchronizing the frequency and adjusting the offset) the VSYNC signals between the server and the client to reduce the one-way latency between the server and the client, and to reduce the variability of the one-way latency between the server and the client. For example, embodiments of the present disclosure are capable of calculating the optimal adjustment to the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312 such that the decoded rendered video frame is available in time for the display process 407 even in the case of near worst-case server processing (such as encoding 403 and transmission 404) time required, near worst-case network latency between the server 260 and the client 210, and near worst-case client processing (such as receiving 405 and decoding 406). That is, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC; it is sufficient to adjust the offset such that the decoded rendered video frame is available in time for the display process.

[0060] In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 can be aligned through synchronization. Synchronization is achieved by tuning either the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, tuning with respect to the server VSYNC signal 311 is described, but it should be understood that tuning can alternatively be performed with respect to the client VSYNC signal 312. For example, as shown in FIG. 4, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), which indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal. Figure 4

[0061] To maintain synchronization of the frequencies of the server VSYNC signal and the client VSYNC signal, the timing of the server VSYNC signal 311 can be manipulated. For example, the vertical blanking interval (VBI) in the server VSYNC signal 311 can be increased or decreased for a certain period of time, such as to account for drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulating the vertical blanking (VBLANK) lines in the VBI allows for adjusting the number of scan lines of the VBLANK for one or more frame periods of the server VSYNC signal 311. Decreasing the number of scan lines of the VBLANK decreases the corresponding frame period (e.g., time interval) between occurrences of the server VSYNC signal 311. Conversely, increasing the number of scan lines of the VBLANK increases the corresponding frame period (e.g., time interval) between occurrences of the VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies between the client VSYNC signal 311 and the server VSYNC signal 312 to be substantially the same. Also, the offset between the server VSYNC signal and the client VSYNC signal can be adjusted by increasing or decreasing the VBI for a short period of time, followed by restoring the VBI to its original value. In one embodiment, the server VBI is adjusted. In another embodiment, the client VBI is adjusted. In yet another embodiment, instead of two devices (server and client), there are multiple connected devices, each of which can have a corresponding VBI that is adjusted. In one embodiment, each of the multiple connected devices can be a standalone peer device (e.g., a serverless device). In another embodiment, the multiple devices can include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, a multi-tenant server / client architecture, or some combination thereof. ​

[0062] Alternatively, in one embodiment, the server's pixel clock (e.g., located at the southbridge of the server's north / southbridge core logic chipset) can be manipulated to perform a coarse and / or fine tuning of the frequency of the server VSYNC signal 311 over a period of time, thereby restoring the frequency synchronization between the server VSYNC signal 311 and the client VSYNC signal 312 for alignment. Specifically, the pixel clock in the server's southbridge can be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In this way, the frequency of the server VSYNC signal 311 is adjusted to make the frequency alignment between the client VSYNC signal 311 and the server VSYNC signal 312 substantially the same frequency. The offset between the server VSYNC and the client VSYNC can be adjusted by increasing or decreasing the client server pixel clock for a short period of time, after which the pixel clock is returned to its original value. In one embodiment, the server pixel clock is adjusted. In another embodiment, the client pixel clock is adjusted. In yet another embodiment, instead of two devices (server and client), there are multiple connected devices, each of which can have a corresponding pixel clock that is adjusted. In one embodiment, each of the multiple connected devices can be a standalone peer device (e.g., a serverless device). In another embodiment, the multiple connected devices can include one or more server devices and one or more client devices arranged in one or more server / client architectures, multi-tenant server / client architectures, or some combination thereof.

[0063] Figure 5A A transition between two scenes of a video game executing on a cloud gaming server is shown in accordance with one embodiment of the disclosure. In particular, the scenes can be part of a virtualized environment of the video game, where the virtualized environment can have many scenes. For example, the scenes can be described as a cave environment in which a character that the user is playing is participating in a boss battle. A particular scene can be rendered from different points of view. For example, as the character moves back and forth in the scene of the cave environment, different images are generated from different points in the cave. Generally, movement within a scene is not considered a scene change because the variability between points of view of a scene can not change much.

[0064] As shown, there are two scenes 511 and 512 in the virtualized environment of the corresponding video game that the user is playing. For illustrative purposes, scene 511 can be the cave environment in the example provided above. The cave environment can be characterized using dark and monotonous colors. Scene 512 can be outside of the cave environment, such as a bright beach scene with a blue sky, a blue ocean, and a white sand beach. The character that the user is playing can traverse along a path 514 in the virtualized environment, where the path 514 traverses both scenes 511 and 512. For example, in scene 511, the character moves from point A to point B, then from point B to point C, and then from point C to point D. Because the viewpoint of the character along the path 514 changes little within scene 511, the corresponding video frames generated can be encoded with lower complexity, such as generating a low percentage of I-frames and a high percentage of P-frames, such as those used in MPEG or H.264 compression formats.

[0065] The path 514 of the character moves from point D in scene 511 to point E in scene 512. That is, as the character moves from scene 511 inside the cave environment to scene 512 to enter the beach environment, the path between point D and point E includes a scene change 513. The scene change 513 indicates a transition of the scene being displayed, which typically indicates that a new scene with new resources must be drawn during the extraction of the video game. For example, the scene change 513 can be an in-game scene switch, where the game transitions from one scene to another between two video frames (e.g., a scene change in a movie sequence, or starting interactive game play after a series of menus). That is, as the scene changes, one or more corresponding video frames representing the transition between scene 511 and scene 512 can be generated with high complexity (e.g., as reference frames), such as generated as I-frames, such as those used in MPEG or H.264 compression formats.

[0066] After the scene change 513, the character continues along the path 514 and moves from point E to point F within scene 512. Again because the viewpoint of the character along the path 514 changes little within scene 512, the corresponding video frames generated (e.g., between point E and point F) can be encoded with lower complexity, such as generating a low percentage of I-frames and a high percentage of P-frames.

[0067] Figure 5B A cloud gaming system is shown that includes a cloud game server 260 and a client 210. According to one embodiment of the present disclosure, the cloud game server 260 is configured to generate and use scene change hints when encoding video frames when streaming content from a video game executing on the cloud game server 260 to the client 210 across a network 250.

[0068] In particular, the game logic 215 of the video is built on top of a game engine 211 that, when executed by the cloud game server 260, generates game rendered video frames for streaming back to the client 210. In particular, the game engine 211 includes core functionality that can be used by the game logic to build a game environment of a video game. For example, some functionality of the game engine 211 can include a physics engine for simulating physical forces and collisions acting on objects in the video game, a rendering engine for 2D or 3D graphics, collision detection, sound, animation, artificial intelligence, networking, streaming, etc. In this way, the game logic 215 does not have to build core functionality provided by the game engine 211 from scratch. Moreover, the core functionality of the game engine 211 can be transported to and used by other game logic of other video games.

[0069] The game logic 215 in conjunction with the game engine 211 is executed by the CPU 501 and the GPU 502 (e.g., configured to implement a graphics pipeline), where the CPU 501 and the GPU 502 can be configured as a rendering pipeline for generating game rendered video frames. The rendering pipeline includes the CPU 501 and the GPU 502 and a memory (e.g., system memory, shared memory, vertex buffer, index buffer, depth or Z buffer, frame buffer for storing rendered video frames, etc.) accessible by both. The rendering pipeline outputs game rendered images as image frames suitable for display and includes corresponding color information for each pixel in a virtualized display.

[0070] The CPU 501 is configured for executing a video game (e.g., game logic built on top of the game engine 211) to generate a plurality of video frames. For example, the CPU 501 generates draw calls for frames of the video frames, which include commands stored in a corresponding command buffer that are executed by the GPU in a GPU pipeline. There can be multiple draw calls generated by the CPU 501 and executed by the GPU 502 through the graphics pipeline for a particular video frame. Subsequent video frames are generated and / or rendered for display using similarly configured command buffers, where the video frames are output from the GPU pipeline.

[0071] In particular, the commands in the one or more command buffers can be executed by the GPU 502 to generate a corresponding video frame. For example, a graphics pipeline can be implemented by the GPU 502 to execute a shader program on vertices of objects within a scene to generate texture values for display pixels, where operations are performed in parallel through the GPU 502 for efficiency. Generally, the graphics pipeline receives input geometry (e.g., vertices of objects within a game world). A vertex shader constructs polygons or primitives that make up objects within a scene. Vertex shaders or other shader programs can perform lighting, shading, shadowing, and other operations on the polygons. A depth or z-buffer can be performed to determine which objects are visible in a scene rendered from a corresponding viewpoint. Rasterization is performed to project objects in a three-dimensional world to a two-dimensional plane defined by a viewpoint. Pixels sized fragments are generated for objects, where one or more fragments can contribute to a color for a corresponding pixel when the image is displayed. Fragments are combined and / or blended to determine a combined color for each pixel in a corresponding video frame, and can be stored in a frame buffer.

[0072] In one embodiment, each of the game rendered video frames can be composited and blended with additional user interface (UX) features (e.g., as an overlay). Example UX features can include user interfaces, system user interfaces, texting, messaging, menus, communications, additional game viewpoints, e-sports information, etc. For example, a scan-out process can be implemented to composite and blend each of the game rendered video frames with a corresponding feature overlay to generate a modified video frame. Additional operations can be performed in generating the modified video frame, such as decompression of a DCC compressed surface, resolution scaling to a target display, color space conversion, de-gamma, HDR expansion, gamut remapping, LUT shaping, tone mapping, blend gamma, blending, etc. so that the one or more layers of the input frame buffer composited and blended and modified using other operations are then optionally placed into a display buffer, which is then scanned to an encoder (e.g., scanned from the display buffer).

[0073] A plurality of game rendered video frames and / or modified video frames for a video game are generated. The modified video frames are then scanned to an encoder 570 for compression, after which the modified video frames are streamed to a client over a network. For example, a corresponding modified video frame can be compressed into one or more encoded slices (compressed encoder slices), which can be further packetized for network streaming. In one embodiment, a corresponding modified video frame is encoded on a per-encoder slice basis to generate one or more encoded slices for the corresponding modified video frame. The modified video frames that have been compressed and / or packetized into encoded slices are then optionally stored into a buffer (e.g., a first-in-first-out or FIFO buffer). Streamer 575 is configured to transmit the encoded slices from the optional buffer to client 210 over network 250. Streamer 575 can be configured to operate at the application layer of the Transmission Control Protocol / Internet Protocol (TCP / IP) computer network model.

[0074] According to one embodiment of the disclosure, game logic 215 can be made aware of scene changes while the CPU is executing the video game. In particular, scene change logic 520 can be integrated within game logic 215 and / or provided as an add-on to game logic 215. Scene change logic 520 is configured to determine, anticipate, and / or predict when a video frame being generated includes a scene change (e.g., a transition from one scene to another scene). For example, game logic 215 and / or scene change logic 520 can include code that indicates that a particular video frame is a scene change. In another embodiment, scene change logic 520 is configured to track gameplay during execution of the video game to predict when a scene change is forthcoming.

[0075] Specifically, scene change logic 520 analyzes game state data collected during the execution of the video game to determine and / or anticipate and / or predict when a scene change will occur in the next X frames or for a given identified video frame. For example, game state data defines the game state at that point and may include game characters, game objects, game object attributes, game attributes, game object states, graphics overlays, the character's position within the game world the player is playing, the scene or game environment, the level of the game application, the character's assets (e.g., weapons, tools, bombs, etc.), equipment, the character's skill set, game level, character attributes, character position, remaining lives, possible total available lives, armor, trophies, time counter values, and other asset information. In this way, game state data allows for the generation of a game environment existing at the corresponding point in the video game. Thus, scene change logic 520, when executed, is configured to predict when a scene change will occur and is further configured to identify which video frame (e.g., by frame number) includes a scene change, or within which range of video frames a scene change may occur.

[0076] Specifically, in one embodiment, scene change awareness of scene change logic 520 and / or game logic 215 allows for notification of scene changes to an encoder used to stream content from a video game executed on a cloud gaming server across a network to a client device. For example, notifications can be provided from scene change logic 520 as scene change cues 509 via an API to other components (e.g., the encoder), where the API can be used for communication between components of cloud gaming server 260 or between applications running on those components. In one embodiment, the API can be a GPU API. For example, the API can run on or be invoked by scene change logic 520 and / or game logic 215 to communicate with another component, such as GPU 502, system memory, command cache, encoder 570, etc. In one embodiment, scene change cues 509 can be provided as data control packets formatted such that all components receiving the data control packets can understand what type of information is included in the data control packets and understand the appropriate reference to the corresponding rendered video frame. In one implementation, the communication protocol, data control packets, or message formatting used for the API can be defined in the corresponding software development kit (SDK) for the video game.

[0077] Through the Figures 2A-2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 6AFIG. 6A illustrates a flowchart 600A of a method for encoding video frames when performing cloud gaming, in which one or more scene change cues are used by an encoder to process corresponding video frames, according to one embodiment of the present disclosure. The encoder can select to encode video frames differently depending on whether a scene change cue is received. In this way, the one-way latency between the cloud gaming server and the client is reduced because the encoder no longer has to determine whether a video frame to be encoded is a scene change. Moreover, because the frame period is no longer used to determine scene changes and re-encode video frames that have been determined to be scene changes, the one-way latency becomes more consistent because the encoding of most video frames is performed within a single frame period. Furthermore, because the frame period is no longer used to determine scene changes and re-encode video frames that have been determined to be scene changes, the jitter is reduced, which improves the smoothness of the client display of the video. Although the flowchart 600A is described as providing scene change cues to an encoder configured to compress video frames, it should be understood that the scene change cues can also be used by an encoder configured to compress encoded slices of corresponding video frames.

[0078] In particular, at 601, a video frame that has been generated by a video game executing on a CPU and GPU is received at an encoder when the content is streamed from the cloud gaming server to the client. As previously described, the game-generated video frame can be synthesized and mixed with additional user interface features into a modified video frame that is scanned into the encoder.

[0079] The encoder is configured to compress the modified video frame based on a desired format. For example, the Moving Picture Experts Group (MPEG) or H.264 standard can be used to stream media content from the cloud gaming server to the client. The encoder can perform compression on a video frame (e.g., MPEG-2) or can perform compression by encoding slices of a video frame, where each video frame can be compressed into one or more encoded slices. Generally, a video frame can be compressed as an I-frame (intra-frame) or a P-frame (predicted frame), each of which can be divided into encoded slices.

[0080] In particular, an I-frame contains an entire image and can be encoded without reference to any other video frame. That is, an I-frame can stand alone as a key image and can be used as a reference for encoding and / or decoding other video frames. An I-frame can also be decoded without reference to any other frame. Video frames that are determined to be scene changes are generally encoded as I-frames. However, I-frames generally require more bits to encode than other frame types, such as P-frames.

[0081] P-frames are encoded and / or decoded with reference to one or more previous video frames. Generally, a video frame that is encoded as a P-frame contains only the differences between the currently encoded frame and the provided one or more previous video frames. That is, the redundant information between video frames is not encoded in the P-frame. When decoded, the redundant information can be recovered because the decoded P-frame references a previously decoded video frame that contains the redundant information.

[0082] In one embodiment, when streaming, video frames are encoded as P-frames until a scene change occurs or the currently encoded frame can no longer reference a key frame (e.g., a previous I-frame), so that the next video frame is then encoded as another I-frame. In particular, in an embodiment, a group of pictures (GOP) mode can be disabled, where the GOP mode requires one I-frame for every video frame period (e.g., 16 video frames). Further, in an embodiment, scene change detection can also be disabled at the encoder. By disabling the GOP mode and scene change detection, most video frames are encoded as P-frames until it is no longer possible to generate a P-frame (e.g., cannot reference a previous I-frame). In the present embodiment, when a corresponding scene change hint is received at the encoder, a video frame is compressed as an I-frame.

[0083] At decision step 603, the encoder determines whether a scene change hint has been received for the corresponding video frame. As previously described, the scene change hint can be provided by the video game executing on the cloud gaming server (e.g., generated by the scene change logic 520 and delivered through the API). The behavior of the encoder differs depending on whether a scene change hint is received in relation to the current video frame to be encoded. In particular, when a scene change hint is not received, the encoder assumes that the video frame is not a scene change and normally encodes the video frame. In one embodiment, the GOP mode is disabled and scene change detection is also disabled because the encoder relies on the scene change hint to determine when a video frame is a scene change and needs to be encoded as an I-frame. Typically, the encoder will compress the corresponding video frame as a P-frame unless it is no longer possible to do so.

[0084] On the other hand, when a scene change cue is received, the corresponding video frame is ultimately compressed into an I-frame. Specifically, at 607, the method determines whether to encode the next video frame as a P-frame (i.e., still referencing the previous I-frame). If the video frame is to be encoded as a P-frame, then at 609, after receiving the scene change cue, the corresponding video frame received by the encoder is compressed into an I-frame. On the other hand, if the video frame is to be encoded as an I-frame, then it is still encoded as an I-frame. Because the encoder relies on the scene change cue, GOP mode or scene change detection is not required at the encoder. In particular, the scene change cue is provided by the CPU when executing the video game to generate video frames. That is, the video game detects when a video frame is considered a scene change. For example, code in the video game can be used to recognize scene changes so that the corresponding video frame can also be recognized as a scene change when it is generated.

[0085] Through the Figures 2A-2D Detailed description of various client devices 210 and / or cloud gaming networks 290 (e.g., in game servers 260), Figure 6B Flowchart 600B illustrates a method for encoding video frames during cloud gaming according to one embodiment of this disclosure, including generating one or more scene change cues while executing a video game or game application that can be used for efficient encoding of the corresponding video frames. Typically, a video game executed by a cloud gaming server can be configured to provide cues about upcoming scene changes in the next video frame to be encoded, where the scene change cues can be communicated to the encoder by the application (e.g., game logic) as it executes via an API. The encoder can optionally perform different encodings (e.g., switching from P-frames to I-frames during scene changes). This process provides smoother frame rates and more reliable latency, reducing and making more consistent one-way latency between the cloud gaming server and the client, thereby improving the smoothness of the client-side display of the video. Although flowchart 600B is described as providing scene change cues to an encoder configured to compress video frames, it should be understood that scene change cues can also be used by an encoder configured to compress encoded slices of the corresponding video frames.

[0086] At 610, the method includes executing game logic built on a game engine of a video game at a cloud game server to generate a plurality of video frames. The game logic is executed on a CPU and a GPU implementing a graphics pipeline. For example, the cloud game server can be executing the video game in a streaming mode such that the game logic built on the game engine is executed in response to input commands from a user in order to generate game rendered video frames using a graphics pipeline available for streaming. In particular, the CPU executing the game logic built on the game engine in cooperation with the GPU graphics pipeline is configured to generate the plurality of video frames. In cloud gaming, the video frames generated by the game are typically rendered for display on a virtual display.

[0087] At 620, the method includes executing scene change logic to predict a scene change in the plurality of video frames, where the prediction is based on game state collected during execution of the game logic. In one embodiment, the scene change logic can be integrated within the game logic. In another embodiment, the scene change logic can be provided as an add-on to the game logic. In particular, while the game logic is being executed, the scene change logic can identify certain video frames or situations in the game represented by corresponding video frames as a scene change. The scene change logic is configured to predict the scene change based on tracking the game state collected while generating the plurality of video frames. For example, the scene change logic can predict when a scene change is about to occur, such as when a character moves from one scene to another scene in a virtualized game environment, or when a character ends one level and is transitioning to another level in a video game, etc. The scene change can be represented by a video frame that includes a large and complex scene of a virtualized game world or environment. For example, the scene change can occur when the executed video game transitions from one scene to another scene between two video frames (e.g., a scene change or scene cut in a movie sequence, or after a series of menus to start interactive game play). Thus, the video game (e.g., the game logic) is able to determine or anticipate when a scene change is going to occur while generating the video frames.

[0088] At 630, the method includes identifying a range of video frames in the plurality of video frames predicted to include the scene change. That is, the scene change logic is executed to predict the likelihood of a scene change in an upcoming range of video frames or within an identifiable video frame. In one embodiment, the scene change logic is able to predict a scene change for an identifiable video frame, such as the first video frame. In this case, the range of video frames is limited to the first video frame.

[0089] At 640, the method includes generating a scene change hint using scene change logic. In particular, the scene change hint identifies a range of video frames, where the range of video frames includes the first video frame. Scene change prediction by the scene change logic can inform other components of the cloud game server that are performing operations on one or more corresponding video frames of the scene change.

[0090] At 650, the method includes delivering the first video frame to an encoder, where the first video frame is within the range of video frames identified as having a scene change. The first video frame is scanned into the encoder for compression, such as when streaming content from a video game executing at the cloud game server to a client across a network, in preparation for streaming the first video frame to the client.

[0091] At 660, the method includes sending the scene change hint (e.g., notification) from the CPU to the encoder. Scene change awareness by the scene change logic and / or game logic provides downstream notification that one or more of the identified video frames or range of video frames includes a scene change, allowing downstream components to act accordingly in an effort to reduce processing time in processing the one or more identified video frames by the cloud game server. For example, the scene change hint can be delivered from the scene change logic to the encoder through an API, where the API can be used to communicate between components or between applications running on components of the cloud game server.

[0092] At 670, the method includes encoding a corresponding video frame (e.g., the first video frame) received by the encoder as an I-frame based on the scene change hint, where the encoder is configured to compress video frames for streaming content from a video game executing on the cloud game server to a client device across a network. As previously described, the behavior of the encoder differs depending on whether a video frame is identified as a scene change by receiving a scene change hint. That is, the encoder is configured to determine whether any scene change hints for a video frame are received at the encoder. In particular, when a scene change hint is received, the encoder compresses the corresponding video frame as an I-frame (e.g., key frame).

[0093] In general, according to one embodiment of the present disclosure, when streaming content from a cloud game server, video frames are compressed as P-frames until it is no longer feasible to compress a video frame as a P-frame (e.g., unable to reference a previous I-frame or key frame), and unless otherwise indicated by a scene change hint. According to one embodiment of the present disclosure, because GOP mode is disabled and scene detection is also disabled, the encoder relies heavily on scene change hints to identify video frames as a scene change. That is, when a scene change hint is not received, the encoder assumes that the video frame is not a scene change and normally encodes the video frame (e.g., encodes it as a P-frame). On the other hand, when a scene change hint is received, the corresponding video frame is encoded as an I-frame regardless of whether the video frame would have been normally encoded as a P-frame or an I-frame. In this way, when a scene change hint is received for a corresponding video frame, the encoder then compresses that video frame as an I-frame.

[0094] In embodiments, the scene change hint supplements the encoder compression determination. In particular, the game provides a scene change hint and the encoder can be configured with a predetermined or dynamically determined threshold (e.g., a high threshold) that both serve to determine whether a video frame will be encoded as an I-frame or a P-frame. In one scenario, the game can not provide a scene change hint for a video frame that displays an explosion (e.g., during a camera cut) and it is not recommended to encode the video frame as an I-frame, but it can be more efficient to encode the frame as an I-frame determined by the encoder. Specifically, the encoder also determines whether the explosion is large enough to compress as an I-frame. That is, the encoder determines that there is a scene change (e.g., an explosion) that reaches or exceeds a threshold within the range of the corresponding video frame independent of any detection of a scene change by the scene change logic. For example, the encoder determines that the delta between frames is large enough to warrant a scene change. In this way, the corresponding video frame is encoded as an I-frame based on the scene change hint or the encoder determining that the scene change reaches or exceeds the threshold. In another scenario, the game can provide a scene change hint for a video frame that displays an explosion (e.g., during a camera cut) and it is therefore recommended to encode the video frame as an I-frame, but it can be more efficient to encode the frame as a P-frame determined by the encoder. Specifically, the encoder also determines whether the explosion is large enough to compress as an I-frame. That is, the encoder determines that the scene change identified by the scene change logic in the range of the corresponding video frame does not reach or exceed a threshold. For example, the encoder determines that the delta between frames is below the threshold, indicating that compressing as a P-frame will be more efficient. In this way, the corresponding video frame can be terminated from being encoded as an I-frame and the corresponding video frame is encoded as a P-frame.

[0095] Figure 7 Components of an example device 700 that can be used to perform aspects of the various embodiments of the present disclosure are shown. For example, Figure 7An exemplary hardware system suitable for use with streaming media content and / or receiving streamed media content, including providing scene change cues to an encoder for video frames generated during execution of a video game in order to reduce latency between a cloud gaming server and a client and provide more consistent latency, as well as improve smoothness of client display of video, is shown in accordance with embodiments of the present disclosure. This block diagram shows a device 700, which can incorporate or be a personal computer, a server computer, a game console, a mobile device, or other digital device, each of which is suitable for practicing embodiments of the present invention. The device 700 includes a central processing unit (CPU) 702 for running software applications and optionally an operating system. The CPU 702 can be constituted by one or more homogeneous or heterogeneous processing cores.

[0096] According to various embodiments, the CPU 702 is one or more general-purpose microprocessors having one or more processing cores. Additional embodiments can be implemented using one or more CPUs, where the microprocessor architecture is particularly suited to highly parallel and compute-intensive applications, such as media and interactive entertainment applications, configured for graphics processing during game execution.

[0097] The memory 704 stores applications and data for use by the CPU 702 and GPU 716. The storage 706 provides non-volatile storage for applications and data and other computer-readable media, and can include fixed or removable magnetic disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray Disc, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. The user input devices 708 communicate user inputs from one or more users to the device 700, examples of which can include keyboards, mice, joysticks, touch pads, touch screens, still or video recorders / cameras, and / or microphones. The network interface 709 allows the device 700 to communicate with other computer systems via an electronic communications network, and can include wired or wireless communication over local- and wide-area networks, such as the Internet. The audio processor 712 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 702, memory 704, and / or storage 706. The components of the device 700, including the CPU 702, graphics subsystem including GPU 716, memory 704, data storage 706, user input devices 708, network interface 709, and audio processor 712, are connected by one or more data buses 722.

[0098] Graphics subsystem 714 is further connected with data bus 722 and components of device 700. Graphics subsystem 714 includes a graphics processing unit (GPU) 716 and graphics memory 718. Graphics memory 718 includes a display memory (e.g., a frame buffer) for storing pixel data for each of the pixels that make up an output image. Graphics memory 718 can be integrated in the same device as GPU 716, connected as a separate device with GPU 716, and / or implemented within memory 704. Pixel data can be provided to graphics memory 718 directly from CPU 702. Alternatively, CPU 702 provides GPU 716 with data and / or instructions defining the desired output images, and GPU 716 generates pixel data for the output image(s) in accordance with the data and / or instructions. The data and / or instructions defining the desired output images can be stored in memory 704 and / or graphics memory 718. In embodiments, GPU 716 includes 3D rendering capability to generate pixel data for output images from instructions and data defining geometry, lighting, shading, texturing, motion, and / or camera parameters for a scene. GPU 716 can also include one or more programmable execution units capable of executing shader programs.

[0099] Graphics subsystem 714 periodically outputs pixel data for an image from graphics memory 718 to be displayed on display device 710 or to be projected by a projection system (not shown). Display device 710 can be any device capable of displaying visual information in response to a signal from device 700, including CRT displays, LCD displays, plasma displays, and OLED displays. Device 700 can provide the signal to display device 710 as, for example, an analog or digital signal.

[0100] Other embodiments for optimizing graphics subsystem 714 can include multi-tenant GPU operations, where GPU instances are shared between multiple applications and distributed GPUs supporting a single game. Graphics subsystem 714 can be configured as one or more processing devices.

[0101] For example, in one embodiment, graphics subsystem 714 can be configured to perform multi-tenant GPU functions, where one graphics subsystem can implement graphics and / or rendering pipelines for multiple games. That is, graphics subsystem 714 is shared between multiple games that are being executed.

[0102] In other embodiments, graphics subsystem 714 includes multiple GPU devices that are combined to perform graphics processing for a single application executing on a corresponding CPU. For example, multiple GPUs can perform alternate forms of frame rendering, where in successive frame periods, GPU 1 renders a first frame, and GPU 2 renders a second frame, and so on until the last GPU, then the initial GPU renders the next video frame (e.g., if there are only two GPUs, then GPU 1 renders a third frame). That is, the GPUs rotate in rendering frames. The rendering operations can overlap, where GPU 2 can begin rendering a second frame before GPU 1 completes rendering a first frame. In another implementation, different shader operations can be assigned to multiple GPU devices in the rendering and / or graphics pipeline. A master GPU is performing master rendering and compositing. For example, in a group including three GPUs, master GPU 1 can perform master rendering (e.g., first shader operations) and compositing from outputs of slave GPU 2 and slave GPU 3, where slave GPU 2 can perform second shader (e.g., fluid effects, such as rivers) operations, slave GPU 3 can perform third shader (e.g., particle smoke) operations, where master GPU 1 composites results from each of GPU 1, GPU 2, and GPU 3. In this way, different GPUs can be assigned to perform different shader operations (e.g., waving flags, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs can be assigned to different objects and / or portions of a scene corresponding to a video frame. In the above embodiments and implementations, these operations can be performed in the same frame period (parallel simultaneously) or in different frame periods (parallel sequentially).

[0103] Accordingly, the present disclosure describes methods and systems configured for streaming media content and / or receiving streamed media content, including providing scene change hints to an encoder for video frames generated during execution of a video game in order to reduce latency between a cloud gaming server and a client and provide more consistent latency, as well as improve smoothness of client display of the video.

[0104] It should be appreciated that various embodiments defined herein can be combined or assembled into specific implementations using various features disclosed herein. Accordingly, the examples provided are only some of the possible examples and are not limited to the possible various implementations by way of combination of various elements in order to define more implementations. In some examples, some implementations can include fewer elements, without departing from the spirit of the disclosed or equivalent implementations.

[0105] Embodiments of the present disclosure can be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. Embodiments of the present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire- or wireless-based network.

[0106] With the above embodiments in mind, it should be understood that the embodiments of the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The embodiments of the present disclosure also relate to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purposes, or it can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it can be more convenient to construct a more specialized apparatus to perform the required operations.

[0107] The present disclosure can also be embodied as computer readable code on a computer readable medium. Any of the data storage devices described herein qualify as computer readable media. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

[0108] Although the method operations were described in a specific order, it should be understood that other housekeeping operations can be performed in between operations, or operations can be adjusted so that they occur at slightly different times, or can be distributed in different systems, which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations occur as desired and that the desired processing of the overlay operations are performed in the desired order.

[0109] While the foregoing disclosure has described that which is presently considered to be specific embodiments of the present disclosure, it is understood that various modifications can be made to the embodiments disclosed without departing from the scope of the present disclosure. Accordingly, the present embodiments are to be considered illustrative and not restrictive, and the present disclosure is not to be limited to the details given herein, but can be modified within the scope and equivalents of the appended claims.

Claims

1. A method for encoding, comprising: Execute game logic built on the game engine of a video game at the cloud gaming server to generate multiple video frames; Execute scene change logic to predict scene changes in the plurality of video frames, wherein the prediction is based on game states collected during the execution of the game logic; The scene change logic is used to identify the range of video frames predicted to include the scene change from among the multiple video frames. The scene change logic is used to generate a scene change prompt, wherein the scene change prompt identifies the video frame range, wherein the video frame range includes a first video frame. The first video frame is delivered to the encoder; The scene change prompt is sent from the scene change logic to the encoder; The first video frame is encoded into an I-frame based on the scene change cues. At the encoder, it is determined that the scene change within the range of the video frames, corresponding to the increment between video frames and identified by the scene change logic, is below a threshold; Terminating the encoding of the first video frame into an I-frame; and The first video frame is encoded as a P-frame.

2. The method of claim 1, further comprising: The first video frame is received at the encoder. Determine whether any scene change notifications for the first video frame are received at the encoder; The first video frame is encoded normally when no scene change notification is received; as well as After confirming that the scene change notification has been received, the first video frame is encoded as an I-frame.

3. The method of claim 1, further comprising: At the encoder, it is determined that the scene change predicted within the range of the video frames, corresponding to the increment between video frames, reaches or exceeds a threshold. as well as The first video frame is encoded as an I-frame based on the scene change cues or the encoder determining that the scene change has reached or exceeded the threshold.

4. The method of claim 1, wherein the scene change logic is integrated into the game logic or provided as an add-on to the game logic, wherein the scene change logic is configured to predict scene changes based on tracking game states collected during the generation of the plurality of video frames.

5. The method of claim 1, further comprising: Disable image group mode at the encoder; as well as Scene change detection is disabled at the encoder to ensure that a given video frame is encoded as an I-frame only when a corresponding scene change cue is received at the encoder.

6. The method of claim 1, further comprising: Predict scene changes in the first video frame. The scene change prompts the identification of the first video frame. The range of video frames is limited to the first video frame.

7. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 6.

8. A computer system comprising: processor; as well as A memory coupled to the processor and having instructions stored therein, which, when executed by the computer system, cause the computer system to: Execute game logic built on the game engine of a video game at the cloud gaming server to generate multiple video frames; Execute scene change logic to predict scene changes in the plurality of video frames, wherein the prediction is based on game states collected during the execution of the game logic; The scene change logic is used to identify the range of video frames predicted to include the scene change from among the multiple video frames. The scene change logic is used to generate a scene change prompt, wherein the scene change prompt identifies the video frame range, wherein the video frame range includes a first video frame. The first video frame is delivered to the encoder; The scene change prompt is sent from the scene change logic to the encoder; The first video frame is encoded into an I-frame based on the scene change cues. At the encoder, it is determined that the scene change within the range of the video frames, corresponding to the increment between video frames and identified by the scene change logic, is below a threshold; Terminating the encoding of the first video frame into an I-frame; and The first video frame is encoded as a P-frame.

9. The computer system of claim 8, wherein, The stored instructions, when executed by the computer system, cause the computer system to: The first video frame is received at the encoder. Determine whether any scene change notifications for the first video frame are received at the encoder; The first video frame is encoded normally when no scene change notification is received; as well as After confirming that the scene change notification has been received, the first video frame is encoded as an I-frame.

10. The computer system of claim 8, wherein, The stored instructions, when executed by the computer system, cause the computer system to: At the encoder, it is determined that the predicted scene change within the range of the video frames, corresponding to the increment between video frames, reaches or exceeds a threshold; and The first video frame is encoded as an I-frame based on the scene change cues or the encoder determining that the scene change has reached or exceeded the threshold.

11. The computer system of claim 8, wherein, The scene change logic is integrated into the game logic or provided as an add-on to the game logic, wherein the scene change logic is configured to predict scene changes based on tracking the game state collected when the plurality of video frames are generated.

12. The computer system of claim 8, wherein, The stored instructions, when executed by the computer system, cause the computer system to: Disable image group mode at the encoder; and Scene change detection is disabled at the encoder to ensure that a given video frame is encoded as an I-frame only when a corresponding scene change cue is received at the encoder.

13. The computer system of claim 8, wherein, The stored instructions, when executed by the computer system, cause the computer system to: Predict scene changes in the first video frame. The scene change prompts the identification of the first video frame. The range of video frames is limited to the first video frame.

Images