Cloud gaming method, computer-readable medium, and computer system

Abstract

To substantially reduce round-trip and one-way latency between a server and a client by improvements in operations, such as advanced strategies related to frame decoding and display.MEANS FOR SOLVING THE PROBLEM: There is provided a method including generating a video frame when executing a video game at a server, the video frame being stored in a frame buffer. The method includes determining a maximum pixel clock for a chip-set including a scan-out block. The method includes determining a frame rate setting based on the maximum pixel clock and an image size of a target display of a client. The method includes determining a speed setting value for the chip-set, and scanning the video frame into the scan-out block from the frame buffer. The method includes scanning-out the video frame from the scan-out block to the encoder at the speed setting value.SELECTED DRAWING: Figure 5A-1

Description

[Technical field]

[0001] The present disclosure relates to a streaming system configured to stream content over a network, and more specifically to performing a fast scanout operation on a cloud gaming server and / or performing an early scanout operation on a cloud gaming server to reduce latency between a cloud gaming server and a client, where smoothness of display of video at the client can be improved by transmitting an ideal display time to the client. [Background technology]

[0002] In recent years, online services that enable online or cloud gaming in a streaming format between cloud gaming servers and clients connected via a network have been continuously promoted. The streaming format has become increasingly popular due to the availability of on-demand game titles, networking capabilities between players for multiplayer games, sharing of assets between players, sharing of instant experiences between players and / or spectators, enabling friends to watch one friend play a video game, joining one friend in the friend's ongoing gameplay, etc. Unfortunately, the demands are also increasing against the limitations of network connection capabilities and the processing performed on the client and the server being responsive enough to render high quality images delivered to the client. For example, the results of all game activity performed on the server need to be compressed and sent back to the client with low millisecond latency to achieve the best user experience. Round-trip latency may be defined as the overall time from a user's controller input to the display of a video frame at a client, which may include the processing and transmission of control information from the controller to the client, the processing and transmission of control information from the client to the server, the use of that input at the server to generate a video frame responsive to the input, the processing and forwarding of the video frame to an encoding unit (e.g., scanning out), encoding the video frame, transmitting the encoded video frame back to the client, receiving and decoding the video frame, and any processing or staging of the video frame before displaying the video frame. One-way latency may be defined as a portion of the round-trip latency consisting of the time from the start of the transfer (e.g., scan-out) of a video frame to an encoding unit at the server to the start of the display of the video frame at the client. Part of the round-trip and one-way latency is related to the time it takes for the data stream to be transmitted over the communication network from the client to the server and from the server to the client. Another part is related to the processing at the client and the server. Improvements in these operations, such as advanced strategies for decoding and displaying frames, can significantly reduce the round-trip and one-way latency between the server and the client, providing a high-quality experience to users of the cloud gaming service.

[0003] The embodiments of the present disclosure have been made against the above background. Summary of the Invention

[0004] Embodiments of the present disclosure relate to streaming systems configured to stream content (e.g., games) over a network, and more particularly to performing a fast scanout operation or performing an early scanout, such as before the occurrence of the next system VSYNC signal or at the flip time of the corresponding video frame, to send modified video frames to an encoder.

[0005] An embodiment of the present disclosure discloses a method for cloud gaming. A video frame is generated when a video game is executed on a server. The method performs a scanout process by scanning the video frame scanline by scanline, scanning one or more user interface features scanline by scanline, and synthesizing and blending the video frame and the one or more user interface features into a modified video frame into one or more input frame buffers. The method scans the modified video frame scanline by scanline into an encoder of the server. The method starts scanning the video frame and the one or more user interface features into one or more input frame buffers at a corresponding flip time of the video frame into the one or more input frame buffers.

[0006] In another embodiment, a non-transitory computer readable medium storing a computer program for cloud gaming is disclosed. The computer readable medium includes program instructions for generating video frames when executing a video game on a server. The computer readable medium includes program instructions for performing a scanout process by scanning the video frames scanline by scanline, scanning one or more user interface features scanline by scanline into one or more input frame buffers, and synthesizing and blending the video frames and the one or more user interface features into a modified video frame. The computer readable medium includes program instructions for scanning the modified video frames scanline by scanline into an encoder of the server in the scanout process. The computer readable medium includes program instructions for initiating the scanning of the video frames and the one or more user interface features into one or more input frame buffers at a corresponding flip time of the video frames in the scanout process.

[0007] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor storing instructions that, when executed by the computer system, cause the computer system to perform a method for cloud gaming. The method generates video frames when executing a video game on a server. The method performs a scanout process by scanning the video frames scanline by scanline, scanning one or more user interface features scanline by scanline into one or more input frame buffers, and combining and blending the video frames and the one or more user interface features into a modified video frame. The method performs the scanout process by scanning the modified video frames scanline by scanline into an encoder of the server. The method performs the scanout process by initiating scanning the video frames and the one or more user interface features into one or more input frame buffers at a corresponding flip time of the video frames.

[0008] In another embodiment, a method for cloud gaming is disclosed. The method generates video frames when executing a video game on a server. The method includes performing a scan-out process to send the video frames to an encoder configured to compress the video frames, the scan-out process starting at a flip time of the video frames. The method transmits the compressed video frames to a client. The method includes determining a target display time for the video frames at the client. The method includes scheduling a display time for the video frames at the client based on the target display time.

[0009] In another embodiment, a non-transitory computer readable medium storing a computer program for cloud gaming is disclosed. The computer readable medium includes program instructions for generating video frames when executing a video game on a server. The computer readable medium includes program instructions for performing a scan-out process to send the video frames to an encoder configured to compress the video frames, the scan-out process starting at a flip time of the video frames. The computer readable medium includes program instructions for transmitting the compressed video frames to a client. The computer readable medium includes program instructions for determining target display times for the video frames at the client. The computer readable medium includes program instructions for scheduling display times for the video frames at the client based on the target display times.

[0010] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor storing instructions that, when executed by the computer system, cause the computer system to perform a method for cloud gaming. The method includes generating video frames when executing a video game on a server. The method includes performing a scan-out process to send the video frames to an encoder configured to compress the video frames, the scan-out process starting at a flip time of the video frames. The method includes transmitting the compressed video frames to a client. The method includes determining a target display time for the video frames at the client. The method includes scheduling display times of the video frames at the client based on the target display time.

[0011] In another embodiment, a method for cloud gaming is disclosed. The method generates video frames when executing a video game on a server. The method includes performing a scan-out process to send the video frames to an encoder configured to compress the video frames, the scan-out process scans the video frames scanline by scanline into one or more input frame buffers, scans one or more user interface features scanline by scanline, and synthesizes and blends the video frames and the one or more user interface features into a modified video frame, the scan-out process being initiated at a flip time of the video frame. The method transmits the compressed modified video frames to a client. The method determines a target display time for the modified video frames at the client. The method schedules a display time for the modified video frames at the client based on the target display time.

[0012] In another embodiment, a non-transitory computer readable medium storing a computer program for cloud gaming is disclosed. The computer readable medium includes program instructions for generating video frames when executing a video game on a server. The computer readable medium includes program instructions for executing a scanout process for sending the video frames to an encoder configured to compress the video frames, the scanout process scans the video frames scanline by scanline into one or more input frame buffers, scans one or more user interface features scanline by scanline, and composites and blends the video frames and the one or more user interface features into a modified video frame, the scanout process starting at a flip time of the video frames. The computer readable medium includes program instructions for transmitting the compressed modified video frames to a client. The computer readable medium includes program instructions for determining a target display time at the client for the modified video frames. The computer readable medium includes program instructions for scheduling a display time at the client for the modified video frames based on the target display time.

[0013] In yet another embodiment, a computer system includes a processor and a memory coupled to the processor storing instructions, the instructions, when executed by the computer system, cause the computer system to perform a method for cloud gaming. The method includes generating video frames when executing a video game on a server. The method includes performing a scan-out process to send the video frames to an encoder configured to compress the video frames, the scan-out process scanning the video frames scanline by scanline into one or more input frame buffers, scanning one or more user interface features scanline by scanline, and synthesizing and blending the video frames and the one or more user interface features into a modified video frame, the scan-out process starting at a flip time of the video frame. The method includes sending the compressed modified video frame to a client. The method includes determining a target display time for the modified video frame at the client. The method includes scheduling a display time for the modified video frame at the client based on the target display time.

[0014] In another embodiment, a method for cloud gaming is disclosed. The method includes generating video frames when executing a video game on a server, the video frames being stored in a frame buffer. The method includes determining a maximum pixel clock for a chipset including a scanout block. The method includes determining a frame rate setting based on the maximum pixel clock and an image size of a target display of a client. The method includes determining a speed setting for the chipset. The method includes scanning the video frames from the frame buffer to a scanout block. The method includes scanning out the video frames from the scanout block to an encoder at the speed setting.

[0015] In another embodiment, a non-transitory computer readable medium storing a computer program for cloud gaming is disclosed. The computer readable medium includes program instructions for generating video frames when executing a video game on a server, the video frames being stored in a frame buffer. The computer readable medium includes program instructions for determining a maximum pixel clock of a chipset including a scanout block. The computer readable medium includes program instructions for determining a frame rate setting based on a maximum pixel clock and an image size of a client's target display. The computer readable medium includes program instructions for determining a speed setting for a chipset. The computer readable medium includes program instructions for scanning video frames from a frame buffer to a scanout block. The computer readable medium includes program instructions for scanning out video frames from the scanout block to an encoder at the speed setting.

[0016] In yet another embodiment, a computer system includes a processor and a memory, coupled to the processor, storing instructions that, when executed by the computer system, cause the computer system to perform a method for cloud gaming, the method including generating video frames when executing a video game on a server, the video frames being stored in a frame buffer. The method includes determining a maximum pixel clock for a chipset that includes a scanout block. The method includes determining a frame rate setting based on the maximum pixel clock and an image size of a client's target display. The method includes determining a speed setting for the chipset. The method includes scanning video frames from a frame buffer to a scanout block. The method includes scanning out video frames from the scanout block to an encoder at the speed setting.

[0017] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure.

[0018] The present disclosure is best understood by reference to the following detailed description taken in conjunction with the accompanying drawing figures. [Brief description of the drawings]

[0019] [Figure 1A] FIG. 2 is a diagram of a VSYNC signal at the start of a frame period according to one embodiment of the present disclosure. [Figure 1B] FIG. 2 is a diagram of a frequency of a VSYNC signal according to one embodiment of the present disclosure. [Figure 2A] FIG. 1 is a diagram of a system for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations according to one embodiment of the disclosure, where VSYNC signals can be synchronized and offset to reduce one-way latency. [Figure 2B]FIG. 1 is a diagram for providing gaming between two or more peer devices according to one embodiment of the present disclosure, where the VSYNC signals can be synchronized and offset to achieve optimal timing of reception of controllers and other information between the devices. [Figure 2C] 1 illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Figure 2D] 1 illustrates a multi-tenancy configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. [Diagram 3] 1 illustrates the variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure. [Figure 4] 1 illustrates a network configuration including a cloud gaming server and clients when streaming video frames generated from a video game running on the server, where the VSYNC signals of the server and clients are synchronized and offset to allow overlap of operations at the server and client and to reduce one-way latency between the server and client. [Figure 5A-1] 1 illustrates an accelerated processing unit (APU) configured to perform high speed scanout operations to send to an encoder when streaming content over a network from a video game running on a cloud gaming server, according to one embodiment of the present disclosure, or a CPU and GPU connected via a bus (e.g., PCI Express). [Figure 5A-2]5 illustrates a chipset 540B configured to perform fast scan-out operations to send to an encoder when streaming content over a network from a video game running on a cloud gaming server, in accordance with one embodiment of the present disclosure, where user interface features are integrated into rendered video frames of the game. [Figure 5B-1] 1 illustrates a scan-out operation performed to generate modified video frames to send to an encoder when streaming content from a video game running on a cloud gaming server to a client over a network according to one embodiment of the present disclosure. [Figure 5B-2] 1 illustrates a scan-out operation performed to generate modified video frames to send to an encoder when streaming content from a video game running on a cloud gaming server to a client over a network according to one embodiment of the present disclosure. [Figure 5B-3] 1 illustrates a scan-out operation performed to generate modified video frames to send to an encoder when streaming content from a video game running on a cloud gaming server to a client over a network according to one embodiment of the present disclosure. [Figure 5C] 1 illustrates an exemplary server configuration having one or more input frame buffers used when performing fast scanout operations to feed an encoder when streaming content over a network from a video game running on a cloud gaming server in accordance with an embodiment of the present disclosure. [Figure 5D] 1 illustrates an exemplary server configuration having one or more input frame buffers used when performing fast scanout operations to feed an encoder when streaming content over a network from a video game running on a cloud gaming server in accordance with an embodiment of the present disclosure. [Figure 6]FIG. 1 is a flow diagram illustrating a method for cloud gaming that performs an early scanout process to start the encoding process early, thereby reducing one-way latency between the server and the client, according to one embodiment of the present disclosure. [Figure 7A] 1 illustrates a process for generating and transmitting video frames on a cloud gaming server according to one embodiment of the present disclosure, the process being optimized to perform fast and / or early scan-out to the encoder to reduce one-way latency between the cloud gaming server and the client. [Figure 7B] 1 illustrates when a scanout process is performed on a cloud gaming server according to one embodiment of the present disclosure, where scanout is performed fast and / or early, whereby video frames can be scanned into the encoder early to reduce one-way latency between the cloud gaming server and the client. [Figure 7C] 1 illustrates a period for performing a fast scanout so that one-way latency between the cloud gaming server and the client can be reduced by scanning video frames into the encoder early, according to one embodiment of the present disclosure. [Figure 8A] FIG. 11 is a flow diagram illustrating a cloud gaming method that may smooth video displayed at a client in a cloud gaming application, according to one embodiment of the present disclosure, and may perform fast and / or early scanout operations at the server to reduce one-way latency between the cloud gaming server and the client. [Figure 8B] 1 illustrates a timing diagram of server and client operations performed during the execution of a video game on a server that generates rendered video frames of the game that are then sent to a client for display, according to one embodiment of the present disclosure. [Figure 9] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Although the following detailed description includes many specific details for purposes of explanation, those skilled in the art will appreciate that many variations and modifications to the following details are within the scope of the disclosure. Accordingly, the aspects of the disclosure described below are set forth without loss of generality to, and without imposing limitations on, the claims that follow this description.

[0021] Generally speaking, various embodiments of the present disclosure describe methods and systems configured to reduce latency and / or latency instability between a source device and a target device when streaming media content (e.g., streaming audio and video from a video game). Latency instability can occur in one-way latency between a server and a client due to the additional time required to generate a complex frame (e.g., a scene change) at the server, the increased time to encode / compress the complex frame at the server, variable communication paths through the network, and the increased time to decode the complex frame at the client. Latency instability can also occur due to differences in the server and client clocks, which causes drift between the server and client VSYNC signals. In an embodiment of the present disclosure, one-way latency between the server and the client can be reduced in a cloud gaming application by performing a fast scanout of the cloud gaming display buffer. In yet another embodiment, one-way latency can be reduced by performing an early scanout of the cloud gaming display buffer. In another embodiment, when addressing latency issues, the smoothness of video display at the client in a cloud gaming application can be improved by sending an ideal display time to the client.

[0022] In particular, in some embodiments of the present disclosure, one-way latency in cloud gaming applications may be reduced by starting the encoding process early. For example, in a particular architecture used for streaming media content from a cloud gaming server to a client (e.g., streaming audio and video from a video game), scan-out of the server display buffer(s) involves performing additional operations on a video frame to generate one or more layers, which are then combined and scanned into a unit that performs video encoding. Performing the scan-out at a high rate (120 Hz or faster) allows the encoding process to start early, thus reducing one-way latency. Also, in some embodiments of the present disclosure, one-way latency in cloud gaming applications may be reduced by performing an early scanout process at the cloud gaming server. In particular, in a particular architecture used for streaming media content from a cloud gaming server to a client (e.g., streaming audio and video from a video game), an application program (e.g., a video game) running on the server requests that a "flip" of the server display buffer occur upon completion of rendering of a video frame. Instead of performing a scanout operation upon the occurrence of a subsequent server VSYNC signal, the scanout operation is initiated at the flip time, and the scanout of the server display buffer(s) performs additional operations on the video frame to generate one or more layers, which are then combined and scanned into a unit that performs video encoding. Scanning out at flip time (rather than the next VSYNC) allows the encoding process to start earlier, reducing one-way latency. Display timing is unaffected because cloud gaming servers do not actually have displays attached to them. In some embodiments of the present disclosure, when performing server scanout of the display buffer at flip time (rather than the following VSYNC), the ideal display timing at the client depends on both the time the scanout occurred and the intent of the game for that particular display buffer (e.g., was it intended for the next VSYNC, or was the game execution delayed and was actually intended for the previous VSYNC). Strategies vary depending on whether the game is fixed frame rate or variable frame rate, and whether the information is implicit (inferred from the scanout timing) or explicit (the game provides ideal timing via a GPU API, which may be VSYNC or fractional time).

[0023] With the above general understanding of the various embodiments, details of example embodiments will now be described with reference to the various drawing figures.

[0024] Throughout this specification, references to a "game" or a "video game" or a "game application" are meant to represent any type of interactive application that is directed through the execution of input commands. For purposes of explanation only, interactive applications include applications for games, word processing, video processing, video game processing, and the like. Moreover, these terms are interchangeable.

[0025] Cloud gaming involves running a video game on a server to generate rendered video frames of the game and transmit them to clients for display. The timing of operations on both the server and the client may be associated with respective vertical synchronization (VSYNC) parameters. When the VSYNC signals are properly synchronized and / or offset between the server and / or client, operations performed on the server (e.g., generating and transmitting video frames over one or more frame periods) are synchronized with operations performed on the client (e.g., displaying video frames on a display at a display frame or refresh rate corresponding to the frame period). In particular, a server VSYNC signal generated at the server and a client VSYNC signal generated at the client may be used to synchronize operation at the server and the client, i.e., when the server and client VSYNC signals are synchronized and / or offset, the server generates and sends video frames such that the client is synchronized with how those video frames are displayed.

[0026] VSYNC signaling and vertical blanking intervals (VBI) are incorporated to generate and display video frames when streaming media content between the server and the client. For example, the server attempts to generate rendered video frames of the game at one or more frame periods defined by the corresponding server VSYNC signal (e.g., if the frame period is 16.7 ms, generating a video frame every frame period results in 60 Hz operation, while generating one video frame every two frame periods results in 30 Hz operation), and then encodes and sends the video frames to the client. At the client, the received encoded video frames are decoded and displayed, and the client displays each rendered video frame for display beginning with the corresponding client VSYNC.

[0027] For purposes of illustration, FIG. 1A illustrates how a VSYNC signal 111 may indicate the start of a frame period, and various operations may be performed during the corresponding frame period at the server and / or client. When streaming media content, the server may use the server VSYNC signal to generate and encode video frames, and the client may use the client VSYNC signal to display the video frames. The VSYNC signal 111 is generated at a defined frequency corresponding to a defined frame period 110, as shown in FIG. 1B. Additionally, the VBI 105 defines the period from when the last raster line was drawn to the display in the previous frame period until the first raster line (e.g., the top) is drawn to the display. As shown, after the VBI 105, the video frame rendered for display is displayed by the raster scan lines 106 (e.g., raster line by raster line, from left to right).

[0028] Further, various embodiments of the present disclosure are disclosed for reducing one-way latency and / or latency volatility between a source device and a target device, such as when streaming media content (e.g., video game content). For illustrative purposes only, various embodiments for reducing one-way latency and / or latency volatility are described in a server and client network configuration. However, it is understood that various techniques disclosed for reducing one-way latency and / or latency volatility may be implemented in other network configurations and / or on peer-to-peer networks, such as those illustrated in Figures 2A-2D. For example, various embodiments disclosed for reducing one-way latency and / or latency volatility may be implemented between one or more of a server device and a client device in various configurations (e.g., server and client, server and server, server and multiple clients, server and multiple servers, client and client, client and multiple clients, etc.).

[0029] FIG. 2A is a diagram of a system 200A for providing games over a network 250 between one or more cloud gaming networks 290 and / or servers 260 and one or more client devices 210 in various configurations according to one embodiment of the disclosure, where the server and client VSYNC signals may be synchronized and offset, and / or dynamic buffering may be performed on the client, and / or encoding and transmitting operations at the server may be overlapped, and / or receiving and decoding operations at the client may be overlapped, and / or decoding and display operations at the client may be overlapped, in order to reduce one-way latency between the server 260 and the client 210. In particular, system 200A provides games over cloud gaming network 290, where the games are executed remotely from corresponding user client devices 210 (e.g., thin clients) playing the games, according to one embodiment of the disclosure. System 200A may provide game control to one or more users playing one or more games through cloud gaming network 290 via network 250 in either single player or multiplayer mode. In some embodiments, the cloud gaming network 290 may include multiple virtual machines (VMs) running on a host machine's hypervisor, where one or more virtual machines are configured to utilize hardware resources available to the host's hypervisor to execute a game processor module. The network 250 may include one or more communication technologies. In some embodiments, the network 250 may include fifth generation (5G) network technologies having advanced wireless communication systems.

[0030] In some embodiments, communication may be facilitated using wireless technology. Such technology may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks in which the service areas covered by providers are divided into smaller geographic areas called cells. Analog signals representing sound and images are digitized in the phone, converted by an analog-to-digital converter, and transmitted as a stream of bits. All 5G wireless devices within a cell communicate over the air with a local antenna array and low-power automatic transceivers (transmitters and receivers) within the cell via frequency channels assigned by the transceivers from a pool of frequencies reused in other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cell networks, mobile devices moving from one cell to another are automatically transferred to the new cell. Of course, 5G networks are just one example of a type of communication network, and embodiments of the present disclosure may utilize previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies following 5G.

[0031] As shown in the figure, cloud gaming network 290 includes game server 260 that provides access to multiple video games. Game server 260 may be any type of server computing device available in the cloud and may be configured as one or more virtual machines running on one or more hosts. For example, game server 260 may manage virtual machines that support game processors that instantiate instances of users' games. Thus, multiple game processors of game server 260 associated with multiple virtual machines are configured to run multiple instances of one or more games associated with gameplay of multiple users. In that way, the backend server supports streaming of gameplay media (e.g., video, audio, etc.) for multiple game applications to a corresponding number of users. That is, the game servers 260 are configured to stream data (e.g., rendered images and / or frames of corresponding gameplay) back to the corresponding client devices 210 over the network 250. In that way, computationally complex game applications can be executed on the backend servers in response to controller inputs received and forwarded by the client devices 210. Each server can render the images and / or frames, which are then encoded (e.g., compressed) and streamed to the corresponding client devices for display.

[0032] For example, multiple users may access the cloud gaming network 290 over the communications network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, the client devices 210 may be configured as thin clients that interface with a backend server (e.g., a game server 260 of the cloud gaming network 290) configured to provide computing functionality (e.g., including a game title processing engine 211). In another embodiment, the client device 210 may be configured with a game title processing engine and game logic for at least some local processing of a video game, and may be further utilized to receive streaming content generated by a video game running on a back-end server, or for other content provided by a back-end server support. In the case of local processing, the game title processing engine includes basic processor-based functionality for running the video game and services related to the video game. The game logic is stored on the local client device 210 and is used to run the video game.

[0033] In particular, a client device 210 of a corresponding user (not shown) is configured to request access to a game via a communications network 250, such as the Internet, and to render display images generated by the video game executed by a game server 260, with the encoded images being delivered to the client device 210 for display in association with the corresponding user. For example, a user may interact with an instance of the video game executing on a game processor of the game server 260 through the client device 210. More specifically, an instance of a video game is executed by a game title processing engine 211. Corresponding game logic (e.g., executable code) 215 that implements the video game is stored and accessible via a data store (not shown) and used to execute the video game. The game title processing engine 211 can support multiple video games using multiple game logics, each video game selectable by a user.

[0034] For example, client device 210 is configured to interact with game title processing engine 211 in connection with the game play of a corresponding user, such as via input commands used to drive game play. In particular, client device 210 may receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 may be any type of computing device having at least a memory and a processor module, and may be connected to game server 260 via network 250. The backend game title processing engine 211 is configured to generate rendered images, which are delivered over the network 250 for display on a corresponding display associated with the client device 210. For example, via a cloud-based service, the rendered images of the game may be delivered by an instance of the corresponding game running on the game execution engine 211 of the game server 260. That is, the client device 210 is configured to receive the encoded images (e.g., encoded from the rendered images of the game generated through execution of the video game) and to display the rendered images on the display 11. In one embodiment, display 11 includes an HMD (e.g., displaying VR content). In some embodiments, rendered images can be streamed wirelessly or wired to a smartphone or tablet directly from a cloud-based service or via a client device 210 (e.g., PlayStation® Remote Play).

[0035] In one embodiment, the game server 260 and / or the game title processing engine 211 include basic processor-based functionality for executing services related to games and game applications. For example, the processor-based functionality may include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. Additionally, the game application services may include memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with social network friends, communication channels, texting, instant messaging, chat support, etc.

[0036] In one embodiment, the cloud gaming network 290 is a distributed game server system and / or architecture. In particular, a distributed game engine that executes game logic is configured as an instance of a corresponding game. In general, a distributed game engine takes each function of the game engine and distributes them for execution by multiple processing entities. Individual functions may be further distributed across one or more processing entities. The processing entities may be configured in various configurations, including physical hardware and / or as virtual components or virtual machines and / or as virtual containers, which are distinct from virtual machines because containers virtualize instances of game applications running on virtualized operating systems. The processing entities may utilize and / or rely on the underlying hardware of the cloud gaming network 290 servers and one or more servers (computing nodes), which may be located in one or more racks. Coordination, allocation, and management of the execution of those functions for the various processing entities is performed by a distributed synchronization layer. In such a way, the execution of those functions is controlled by the distributed synchronization layer to generate the media (e.g., video frames, audio, etc.) of the game application in response to the controller input by the player. The distributed synchronization layer can efficiently execute those functions (e.g., through load balancing) across the distributed processing entities, whereby critical game engine components / functions are distributed and restructured for more efficient processing.

[0037] The game title processing engine 211 includes a group of central processing units (CPUs) and graphics processing units (GPUs) that can be configured to perform multi-tenancy GPU functions. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0038] 2B is a diagram for providing gaming between two or more peer devices, where VSYNC signals can be synchronized and offset to achieve optimal timing of receipt of controller and other information between the devices, according to one embodiment of the disclosure. For example, head-to-head gaming may be conducted using two or more peer devices connected directly via a network 250 or via peer-to-peer communications (e.g., Bluetooth, local area networks, etc.).

[0039] As shown in the figure, a game is running locally on each of the client devices 210 (e.g., game consoles) of corresponding users playing the video game, and the client devices 210 communicate over a peer-to-peer network. For example, an instance of the video game is executed by a game title processing engine 211 of the corresponding client device 210. Game logic 215 (e.g., executable code) that implements the video game is stored on the corresponding client device 210 and used to run the game. For illustrative purposes, the game logic 215 may be distributed to the corresponding client device 210 via portable media (e.g., optical media) or over a network (e.g., downloaded from a game provider over the Internet).

[0040] In one embodiment, the game title processing engine 211 of the corresponding client device 210 includes basic processor-based functionality for executing services related to games and game applications. For example, the processor-based functionality includes 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. Additionally, the game application services include memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with social network friends, communication channels, texting, instant messaging, chat support, etc.

[0041] Client device 210 may receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 may be any type of computing device having at least a memory and a processor module and configured to generate rendered images executed by game title processing engine 211 or to display rendered images on a display (e.g., display 11, or display 11 including a head mounted display HMD, etc.). For example, the rendered images may be associated with an instance of a game running locally on the client device 210 to effectuate the gameplay of the corresponding user, such as via input commands used to drive the gameplay. Some examples of client devices 210 include personal computers (PCs), game consoles, home theater devices, general purpose computers, mobile computing devices, tablets, phones, or any other type of computing device capable of running an instance of a game.

[0042] 2C illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, including the configurations of FIGS. 2A and 2B, in accordance with one embodiment of the present disclosure. In particular, various network configurations benefit from proper alignment of the frequency of the server and client VSYNC signals and timing offset of the server and client VSYNC signals to reduce one-way latency and / or latency variation between the server and client. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration. In one embodiment, the client may include a web RTC client configured to provide audio and video communication within a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Another network device configuration includes a client (e.g., source) to client (target) configuration, where each client may be, for example, a game console for providing head-to-head gaming.

[0043] In particular, VSYNC signal alignment may include synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to eliminate drift and / or maintain an ideal relationship between the server and client VSYNC signals in order to reduce one-way latency and / or latency variation. To achieve proper alignment, in one embodiment, the server's VSYNC signal may be tuned to enforce proper alignment between the server 260 and client 210 pair. In another embodiment, the client VSYNC signals may be tuned to implement proper alignment between a server 260 and client 210 pair. When the client and server VSYNC signals are aligned, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency and are offset from one another by a timing offset that may be adjusted from time to time. In another embodiment, alignment of the VSYNC signals may include synchronizing the frequency of the VSYNC of the two clients, adjusting the timing offset between these VSYNC signals to eliminate drift, and / or achieve optimal timing of receipt of controller and other information. Either VSYNC signal may be tuned to achieve this alignment. In yet another embodiment, alignment may include synchronizing the frequency of VSYNC of multiple servers, and may also include synchronizing the frequency of the server and client VSYNC signals and adjusting the timing offset between the client and server VSYNC signals, for example, for head-to-head cloud gaming. In server-to-client and client-to-client configurations, alignment may include both synchronizing the frequency between the server and client VSYNC signals, as well as providing an appropriate timing offset between the server and client VSYNC signals. In a server-to-server configuration, alignment may include synchronizing the frequency between the server and client VSYNC signals without setting a timing offset.

[0044] 2D illustrates a multi-tenancy configuration between a cloud gaming server 260 and one or more clients 210 that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the disclosure. In a server-to-client configuration, alignment may include both synchronizing the frequency between the server VSYNC signal and the client VSYNC signal, and providing a proper timing offset between the server VSYNC signal and the client VSYNC signal. In a multi-tenancy configuration, in one embodiment, the client VSYNC signal is tuned at each client 210 to implement proper alignment between the server 260 and client 210 pair.

[0045] For example, the graphics subsystem, in one embodiment, may be configured to perform multi-tenancy GPU functionality, where one graphics subsystem may implement a graphics and / or rendering pipeline for multiple games. That is, the graphics subsystem is shared between multiple games being executed. In particular, the game title processing engine, in one embodiment, may include a CPU and GPU group that may be configured to perform multi-tenancy GPU functionality, where one CPU and GPU group may implement a graphics and / or rendering pipeline for multiple games. That is, the CPU group and GPU group are shared between multiple games being executed. The CPU group and GPU group may be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

[0046] 3 shows a general process of running a video game on a server to generate rendered video frames of the game and send those video frames to a client for display. Conventionally, some operations on the game server 260 and the client 210 are performed within a frame period defined by each VSYNC signal. For example, the server 260 attempts to generate rendered video frames of the game in one or more frame periods defined by a corresponding server VSYNC signal 311 at 301. The video frames are generated by the game in response to either control information (e.g., a user's input command) delivered from an input device at operation 350 or game logic that is not driven by control information. Transmission jitter 351 may be present when transmitting control information to the server 260, and jitter 351 measures the variation in network latency from the client to the server (e.g., when sending an input command). As shown, the thick arrow indicates the current delay in sending the control information to the server 260, but due to jitter, the arrival time of the control information at the server 260 may have a range (e.g., the range surrounded by the dotted arrow). At flip time 309, the GPU arrives at a flip command indicating that the corresponding video frame has been completely generated and placed into the frame buffer of the server 260. The server 260 then performs a scan-out / scan-in (operation 302), where the scan-out may be aligned with the VSYNC signal 311 of that video frame over the subsequent frame period defined by the server VSYNC signal 311 (VBI omitted for clarity). The video frame is then encoded (operation 303) (e.g., encoding may begin after the occurrence of the VSYNC signal 311 and the end of encoding may not be aligned with the VSYNC signal) and transmitted to the client 210 (operation 304, transmission may not be aligned with the VSYNC signal 311). At the client 210, the encoded video frames are received (operation 305, where reception may not be aligned with the client VSYNC signal 312), decoded (operation 306, where decoding may not be aligned with the client VSYNC signal 312), buffered, and displayed (operation 307, where start of display may be aligned with the client VSYNC signal 312). In particular, the client 210 displays each video frame rendered for display beginning upon the occurrence of the corresponding client VSYNC signal 312.

[0047] One-way latency 315 may be defined as the latency from the start of the transfer of a video frame to an encoding unit at the server (e.g., scanout 302) to the start of display of the video frame at the client 307. That is, one-way latency is the time from server scanout to client display, taking into account client buffering. Individual frames have a latency from the start of scanout 302 to the completion of decode 306, and the latency may vary from frame to frame due to server operations such as encode 303 and transmit 304, network transmission with jitter 352 between the server 260 and client 210, and high variability in receive 305 at the client. As shown in the figure, the straight thick arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but due to jitter 352, the arrival time of the video frame at the client 210 may have a certain range (e.g., the range surrounded by the dotted arrow). Since a good playing experience requires that the one-way latency is relatively stable (e.g., fairly constant), buffering 320 is conventionally performed, so that each frame with low latency (e.g., from the start of scanout 302 to the completion of decode 306) is delayed for several frame periods. That is, when the network is unstable or has unpredictable encoding / decoding times, additional buffering is required to keep the one-way latency constant.

[0048] According to an embodiment of the present disclosure, one-way latency between a cloud gaming server and a client may vary due to clock drift when streaming video frames generated from a video game running on the server. That is, due to differences in the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312, the client VSYNC signal may drift with respect to the frames arriving from the server 260. The drift may be due to slight differences in the crystal oscillators used in the server and client clocks. Furthermore, embodiments of the present disclosure reduce one-way latency by performing one or more of synchronization and offsetting of the VSYNC signals to align between the server and the client, by providing dynamic buffering at the client, by overlapping the encoding and transmission of video frames at the server, by overlapping the receiving and decoding of video frames at the client, and by overlapping the decoding and display of video frames at the client.

[0049] 4 illustrates the flow of data through a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210 when streaming video frames generated from a video game running on the server, according to an embodiment of the present disclosure, where overlapping server and client operations reduces one-way latency, synchronization and offset between the server and client VSYNC signals reduces one-way latency, and reduces variability in one-way latency between the server and client. In particular, FIG. 4 illustrates the desired alignment between the server and client VSYNC signals. In one embodiment, synchronization of the server VSYNC signal 311 is performed to obtain proper alignment between the server and client VSYNC signals, such as in a server and client network configuration. In another embodiment, tuning of the client VSYNC signal 312 is performed to obtain proper alignment between the server and client VSYNC signals, such as in a multi-tenant server to multi-client network configuration. For illustrative purposes, tuning of the server VSYNC signal 311 to synchronize the frequency of the server and client VSYNC signals and / or to adjust the timing offset between the corresponding client and server VSYNC signals is illustrated in FIG. 4, although it should be understood that the client VSYNC signal 312 may also be used for tuning. In the context of this patent application, "synchronization" should be interpreted as tuning the signals so that the frequencies match, but the phases may differ, and "offset" should be interpreted as the time delay between the signals, e.g., the time from when one signal reaches its maximum to when the other signal reaches its maximum.

[0050] As shown, Figure 4 illustrates an improved process for running a video game on a server to generate rendered video frames and send those video frames to a client for display in an embodiment of the present disclosure. The process is illustrated with respect to the generation and display of a single video frame on the server and the client. In particular, the server generates rendered video frames of the game at 401. For example, the server 260 includes a CPU (e.g., game title processing engine 211) configured to run the game. The CPU generates one or more draw calls for a video frame, the draw calls including commands that are placed in a command buffer for execution by a corresponding GPU of server 260 in a graphics pipeline. The graphics pipeline may include one or more shader programs that operate on vertices of objects in the scene to generate rendered texture values ​​for the video frame for display, with operations performed in parallel via the GPU for efficiency. At flip time 409, the GPU reaches a flip command in the command buffer, which indicates that the corresponding video frame has been completely generated and / or rendered and placed in the frame buffer of server 260.

[0051] At 402, the server performs scanout of the rendered video frames of the game to the encoder. In particular, scanout is performed scanline by scanline or groups of consecutive scanlines, where a scanline refers to, for example, a single horizontal line of the display from edge of the screen to edge of the screen. These scanlines or groups of consecutive scanlines may also be referred to as slices, and are referred to herein as screen slices. In particular, scanout 402 may include several processes of modifying the rendered frames of the game, including overlaying the rendered frames with another frame buffer, or shrinking the frames to surround the rendered frames with information from another frame buffer. During scanout 402, the modified video frames are then scanned into the encoder for compression. In one embodiment, scanout 402 is performed upon occurrence 311a of the VSYNC signal 311. In other embodiments, scanout 402 may be performed before the occurrence of the VSYNC signal 311, such as at flip time 409.

[0052] At 403, the rendered video frames of the game (possibly modified) are encoded in an encoder slice-by-encoder slice to generate one or more encoded slices, where the encoded slices are independent of scanlines or screen slices, such that the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scanout 402 process of the corresponding video frame is fully completed. Furthermore, the start and / or end of encoding 403 may or may not be aligned with the server VSYNC signal 311. The boundaries of the encoded slices are not limited to a single scanline, but may consist of a single scanline or multiple scanlines. Furthermore, the end of an encoded slice and / or the start of the next encoder slice do not necessarily occur at the edge of the display screen (e.g., may occur in the middle of the screen or the middle of a scanline), and thus the encoded slices do not have to traverse completely from one end of the display screen to the other. As shown in the figure, one or more encoded slices, including the compressed "encoded slice A" having hash marks, may be compressed and / or encoded.

[0053] At 404, the encoded video frame is transmitted from the server to the client, where the transmission may occur encoded slice by encoded slice, with each encoded slice being a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 of the corresponding video frame is fully completed. Furthermore, the start and / or end of the transmission 404 may or may not be aligned with the server VSYNC signal 311. As shown, compressed encoded slice A is transmitted to the client independently of the other compressed encoder slices of the rendered video frame. The encoder slices may be transmitted one at a time or in parallel.

[0054] At 405, the client again receives the compressed video frames for each encoded slice. Furthermore, the start and / or end of receiving 405 may or may not be aligned with the client VSYNC signal 312. As shown, compressed encoded slice A is received by the client. Transmission jitter 452 may exist between the server 260 and the client 210, and jitter 452 measures the variation in network latency from the server 260 to the client 210. A lower jitter value indicates a more stable connection. As shown, the straight thick arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but due to jitter, there may be a range (e.g., the range surrounded by the dotted arrow) in the arrival time of the video frame at the client 210. The variation in latency may be due to one or more operations at the server, such as encoding 403 and transmitting 404, as well as network issues that introduce latency in transmitting the video frame to the client 210.

[0055] At 406, the client decodes the compressed video frame, again encoded slice by slice, to produce decoded slice A (shown without hash marks), which is now ready for display. In one embodiment, the decoding process 406 begins before the corresponding video frame receiving process 405 is fully completed. Furthermore, the start and / or end of the decoding 406 may or may not be aligned with the client VSYNC signal 312. At 407, the client displays the decoded rendered video frame on the client's display. That is, the decoded video frame is put into a display buffer and streamed out to a display device, e.g., scanline by scanline. In one embodiment, the display process 407 (i.e., streaming out to the display device) begins after the decode process 406 is fully completed for the corresponding video frame, i.e., after the decoded video frame is fully resident in the display buffer. In one embodiment, the display process 407 begins before the decode process 406 is fully completed for the corresponding video frame. That is, the streaming out to the display device begins from an address in the display buffer at a time when only a portion of the decoded frame buffer is resident in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frame in time for display, whereby the display buffer updates are performed before those portions are streamed out to the display. Additionally, the start and / or end of display 407 is aligned with the client VSYNC signal 312.

[0056] In one embodiment, one-way latency 416 between server 260 and client 210 may be defined as the elapsed time from when scanout 402 begins to when display 407 begins. Embodiments of the present disclosure may align (e.g., synchronize frequency and adjust offset) VSYNC signals between server and client to reduce one-way latency between server and client and reduce variation in one-way latency between server and client. For example, embodiments of the present disclosure can calculate an optimal adjustment to the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312 so that even if server operations such as encode 403 and transmit 404 require near worst-case times, even if the network latency between the server 260 and the client 210 is near worst-case, and even if client operations such as receive 405 and decode 406 are near worst-case, the decoded rendered video frame will still be available in time for the display process 407. In other words, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC, it is sufficient to adjust the offset so that the decoded rendered video frame is available in time for the display process.

[0057] In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 may be aligned by synchronization. The synchronization is achieved by synchronizing the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, the synchronization is described with respect to the server VSYNC signal 311, but it should be understood that the synchronization can be performed on the client VSYNC signal 312 instead. For example, as shown in FIG. 4, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), which indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal.

[0058] To maintain frequency synchronization of the server and client VSYNC signals, the timing of the server VSYNC signal 311 may be manipulated. For example, the vertical blanking interval (VBI) of the server VSYNC signal 311 may be increased or decreased over a period of time to account for drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulation of vertical blanking (VBLANK) lines in the VBI adjusts the number of scan lines used for VBLANK for one or more frame periods of the server VSYNC signal 311. Reducing the number of scanlines in VBLANK reduces the corresponding frame period (e.g., time interval) between occurrences of two server VSYNC signals 311. Conversely, increasing the number of scanlines in VBLANK increases the corresponding frame period (e.g., time interval) between occurrences of two VSYNC signals 311. In this manner, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies of the client and server VSYNC signals 311 and 312 to be substantially the same frequency. Additionally, the offset between the server and client VSYNC signals can be adjusted by increasing or decreasing the VBI for a short period of time before returning it to its original value. In one embodiment, the server VBI is coordinated. In another embodiment, the client VBI is coordinated. In yet another embodiment, instead of two devices (server and client), there may be multiple connected devices, each of which may have a corresponding VBI that is coordinated. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices may include one or more server devices and / or one or more client devices arranged in one or more server / client architectures, a multi-tenant server / client(s) architecture, or some combination thereof.

[0059] Alternatively, the server's pixel clock (e.g., in the southbridge of the server's northbridge / southbridge core logic chipset, or in the case of a discrete GPU, generating the pixel clock solely using its own hardware) may, in one embodiment, be manipulated to perform coarse and / or fine tuning of the frequency of the server VSYNC signal 311 over a period of time to bring the frequency synchronization between the server VSYNC signal 311 and the client VSYNC signal 312 back into alignment. Specifically, the pixel clock of the server's southbridge may be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311. In this manner, the frequency of the server VSYNC signal 311 is adjusted to align the frequency between the client VSYNC signal 312 and the server VSYNC signal 311 to substantially the same frequency. The offset between the server VSYNC and client VSYNC can be adjusted by briefly increasing or decreasing the client server pixel clock before restoring the pixel clock back to its original value. In one embodiment, the server pixel clock is adjusted. In another embodiment, the pixel clocks of the clients are adjusted. In yet another embodiment, instead of two devices (a server and a client), there may be multiple connected devices, each of which has a corresponding pixel clock that is adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices may include one or more server devices and one or more client devices arranged in one or more server / client architectures, a multi-tenant server / client(s) architecture, or some combination thereof.

[0060] FIG. 5A-1 illustrates a chipset 540 configured to perform a fast scanout operation to send to an encoder when streaming content from a video game running on a cloud gaming server over a network, according to one embodiment of the present disclosure. Additionally, the chipset 540 may be configured to perform an early scanout operation, such as before the next system VSYNC signal occurs or at the flip time of the corresponding video frame. In particular, FIG. 5A-1 illustrates how the speed of the scanout block 550 is determined for a client's target display in one embodiment.

[0061] Chipset 540 is configured to operate at a maximum pixel clock 515. The pixel clock defines the rate at which the chipset can process pixels, such as via scanout block 550. The rate of the pixel clock is typically expressed in megahertz, which represents the number of pixels that can be processed. In particular, pixel clock calculator 510 is configured to determine maximum pixel clock 515 based on chip calculation settings 501 and / or self-tests 505. For example, chipset 540 may be designed with a particular maximum pixel clock included in chip calculation settings 501. However, after construction, chipset 540 may be capable of operating at a higher pixel clock or may not actually operate at the designed pixel clock as determined from chip calculation settings 501. Thus, a test 505 may be performed to determine a self-diagnostic pixel clock 505. Pixel clock calculator 510 may be configured to define a maximum pixel clock 515 for chipset 540 based on the higher of the designed pixel clock, as determined from chip calculation settings 501 or self-diagnostic pixel clock 505. For illustrative purposes, an exemplary maximum pixel clock may be 300 Megapixels per second (Mpps).

[0062] Scanout block 550 operates at a speed corresponding to the target display of client 210. In particular, frame rate calculator 520 determines a frame rate setting 525 based on various inputs including maximum pixel clock 515 of chipset 540 and requested image size 521, which may be obtained from values ​​522 including conventional display values ​​(e.g., 480p, 720p, 1080p, 4K, 8K, etc.) and other defined values. For the same maximum pixel clock, there may be different frame rate settings depending on the client's target display, where the frame rate setting is determined by dividing the maximum pixel clock 515 by the number of pixels of the target display. For example, with a maximum pixel clock of 300 megapixels per second, the frame rate setting of a 480p display (e.g., about 300k pixels as used on a mobile phone) is about 1000Hz. And with the same maximum pixel clock of 300 megapixels per second, the frame rate setting of a 1080p display (about 2 megapixels) is about 150Hz. And for purposes of illustration, with a maximum pixel clock of 300 megapixels per second, the frame rate setting of a 4k display is about 38Hz.

[0063] The frame rate setting 525 is input to a scanout setting converter 530 configured to determine a speed setting 535 formatted for the chipset 540. For example, the chipset 540 may be operating at a bit rate. In one embodiment, the speed setting 535 may be the frame rate setting 525 (e.g., frames per second). In some embodiments, the speed setting 535 may be determined to be a multiple of the base frame rate. For example, the speed setting 535 may be set to a multiple of 30 frames per second (e.g., 30 Hz), such as 30 Hz, 60 Hz, 90 Hz, 120 Hz, 150 Hz, etc. The speed setting 535 is input to a cache 545 of the chipset 540 to be accessed by a corresponding scanout block 550 in the chipset to determine the operating speed of the target display of the client 210.

[0064] Chipset 540 includes a game title processing engine 211, which is configured to execute video game logic 215 of a video game to generate rendered video frames of the game for streaming back to client 210. As shown in the figure, game title processing engine 211 includes a CPU 501 and a GPU 502 (e.g., configured to implement a graphics pipeline). In one embodiment, the CPU 501 and GPU 502 are configured as an Accelerated Processing Unit (APU) configured to integrate the CPU and GPU on the same chip or die using the same bus for faster communication and processing. In another embodiment, the CPU 501 and GPU 502 may be connected via a bus such as PCI-Express, Gen-Z, etc. Rendered video frames of a video game are generated and placed into buffers 555 (e.g., display buffers or frame buffers), including one or more game buffers such as game buffer 0 and game buffer 1. Game buffer 0 and game buffer 1 are driven by a flip control signal to determine which game buffer stores which video frames output from the game title processing engine 211. The game title processing engine runs at a particular speed defined by the video game. For example, video frames may be output by the game title processing engine 211 at 30 Hz or 60 Hz, etc.

[0065] Additional information may optionally be generated for inclusion in the rendered video frames of the game. In particular, feature generation block 560 includes one or more feature generation units, each configured to generate features. Each feature generation unit includes a feature processing engine and a buffer. For example, feature generation unit 560-A includes feature processing engine 503. In one embodiment, feature processing engine 503 runs on CPU 501 and GPU 502 of game title processing engine 211 (e.g., on another thread). Feature processing engine 503 may be configured to generate a number of user interface (UX) features, such as user interface, messaging, and the like. The UX features may be presented as an overlay in one embodiment. A number of UX features generated for a video game are placed into a buffer (e.g., a display buffer or a frame buffer) including one or more UX buffers, such as UX buffer 0 and UX buffer 1. UX buffer 0 and UX buffer 1 are driven by corresponding flip control signals to determine which UX buffer stores which features output from the feature processing engine 503. The feature processing engine 503 is also operating at a particular speed that may be defined by the video game. For example, video frames may be output by the feature processing engine 503 at 30 Hz or 60 Hz, etc. The feature processing engine 503 may also operate at a speed independent of the speed at which video frames may be output by the game title processing engine 211 (i.e., at a rate other than 30 Hz or 60 Hz, etc.).

[0066] The rendered video frames of the game scanned out of buffer 555, and optional features scanned out of buffers of feature generation units (e.g., unit 560-A), are scanned into scanout block 550 at a rate X. The rate X for scanning the game buffer 555 holding the rendered video frames of the game and / or the UX buffer holding the features, in one embodiment, may not correspond to the rate setpoint 535, such that information is scanned out of the buffer as quickly as possible. In another embodiment, the rate X corresponds to the rate setpoint 535.

[0067] As mentioned above, scanout block 550 operates at a speed corresponding to the target display of client 210. If there may be multiple clients with multiple target displays (e.g., mobile phones, television displays, computer monitors, etc.), then there may be multiple scanout blocks, each supporting a corresponding display and each operating at a different speed setting. For example, scanout block A (550-A) receives rendered video frames of the game from buffer 555 and feature overlays from feature generation block 560. Scanout block A (550-A) operates at the corresponding speed setting of cache A (545-A), such as the corresponding frame rate setting. Thus, for the target display, scanout block 550 outputs modified video frames to encoder 570 at a rate defined by the speed setting (e.g., 120 Hz). That is, the rate at which modified video frames are output to encoder 570 is higher than the rate at which video frames are being generated and / or encoded, and this rate is based on the maximum pixel clock of chipset 540, which includes scanout block 550, and the image size of the target display.

[0068] In one implementation, the encoder 570 may be part of the chipset 540. In another implementation, the encoder 570 is separate from the chipset 540. The encoder 570 is configured in part to compress the modified video frames for streaming to the client 210. For example, the modified video frames are encoded slice by slice in the encoder to generate one or more encoded slices for the corresponding modified video frames. The one or more encoded slices of the corresponding modified video frames, including the additional feature overlays, are then streamed over the network to the target display of the client 210. The encoder may output the one or more encoded slices at a rate that may be associated with the synchronized and offset server and client VSYNC signals, independent of the speed setting, as described above. For example, the one or more encoded slices may be output at 60 Hz.

[0069] FIG. 5A-2 illustrates a chipset 540B configured to perform a fast scanout operation to send to an encoder when streaming content over a network from a video game running on a cloud gaming server, according to one embodiment of the disclosure, where optional user interface features may be integrated into the rendered video frames of the game. Additionally, the chipset 540 may be configured to perform an early scanout operation, such as before the next system VSYNC signal occurs or at the flip time of the corresponding video frame. Some components illustrated in FIG. 5A-2 are similar to those in FIG. 5A-1, and similar features have similar functions. Differences between FIG. 5A-2 and FIG. 5A-1 are indicated in the respective chipsets. In particular, the configuration of chipset 540B of FIG. 5A-2 differs from that of FIG. 5A-1 in that there is no separate feature generation block. Thus, one or more optional UX features may be generated by CPU 501 and / or GPU 502 and integrated into the rendered video frames of the game that are buffered 555, as described above. That is, the features are integrated into the rendered video frames and do not need to be provided as an overlay. The rendered video frames of the game may optionally be scanned from buffer 555 to scanout block 550, which includes one or more scanout blocks 550-B for one or more target displays of the client(s). As mentioned above, the corresponding scanout block 550-B operates at the speed of the target display. Thus, for a target display, the corresponding scanout block 550-B outputs video frames to the encoder 570 at a rate defined by the speed setting. In some embodiments, only the buffer 555 is needed because features are integrated into the rendered video frames of the game, so that the rendered video frames may be scanned directly into the encoder, bypassing the scanout block 550. In that case, the additional operations performed during scanout may be performed by, for example, the CPU 501 and / or the GPU 502.

[0070] FIG. 5B-1 illustrates a scanout operation performed to send rendered video frames of a game, which may optionally include one or more additional features (e.g., layers), to an encoder when streaming content from a video game running on a cloud gaming server to a client over a network, according to one embodiment of the disclosure. For example, FIG. 5B-1 illustrates the operation of scanout block 550-A of FIG. 5A-1. Scanout block A (550-A) receives rendered video frames of a game scanline by scanline.

[0071] In particular, scanout block A (550-A) receives rendered video frames of the game from buffer 555 and feature overlays from feature generation block 560, which are provided to input buffer 580. As previously described, scanout block A (550-A) operates according to corresponding speed settings of cache A (545-A), such as corresponding frame rate settings of the target display of client 210. For example, rendered video frames of the game are output from game buffer 0 and game buffer 1 to input frame buffer 580-A of scanout block A (550-A) as controlled by the flip control signal.

[0072] Further, scanout block A (550-A) may optionally receive one or more UX features (e.g., as an overlay). For example, the UX features are output from buffer 560-A, including UX buffer 0 and UX buffer 1, as controlled by corresponding flip control signals. The UX features are scanned into input frame buffer 580-B of scanout block A (550-A). Other feature overlays may be provided, and exemplary UX features may include user interface, system user interface, texting, messaging, menus, communications, additional game perspectives, e-sports information, and the like. For example, additional UX features may be output from buffers 560A-560N, each including UX buffer 0 and UX buffer 1, as controlled by corresponding flip control signals. For illustrative purposes, the UX features are output from buffer 560-N to input frame buffer 580-N.

[0073] The information in input frame buffer 580 is output to combiner 585, which is configured to combine the information. For example, for each corresponding video frame generated by a video game, combiner 585 combines the game's rendered video frame from input frame buffer 580-A with each of the optional UX features provided in input frame buffers 580-B through 580-N.

[0074] The rendered video frames of the game combined with one or more optional UX features are then provided to block 590, where additional operations may be performed to generate modified video frames suitable for display. The additional operations performed during the scan-out process of block 590 may include one or more operations such as DCC surface decompression, resolution scaling to the target display, color space conversion, de-gamma, HDR expansion, gamut remapping, LUT formation, tone mapping, gamma blending, blending, etc.

[0075] In other implementations, additional operations, as broadly described in block 590, are performed on each of the input frame buffers 580 to generate a corresponding layer of a modified video frame. For example, the input frame buffers may be used to store and / or generate the rendered video frames of a game of a video game, as well as one or more optional UX features (e.g., as an overlay), such as a user interface (UI), system UI, text, messaging, etc. The additional operations may include DCC compression plane decompression, resolution scaling, color space conversion, de-gamma, HDR expansion, gamut remapping, LUT creation, tone mapping, gamma blending, etc. After these operations are performed, the one or more layers of the input frame buffer 580 are composited and blended, optionally put into a display buffer, and then scanned into (e.g., scanned out of) the encoder.

[0076] Thus, for the target display, scanout block 550-A outputs a number of modified video frames to encoder 570 at a rate defined by the rate setting (e.g., 120 Hz). That is, the rate at which the modified video frames are output to encoder 570 is higher than the rate at which the video frames are being generated and / or encoded, and this rate is based on the maximum pixel clock of chipset 540 that includes scanout block 550 and the image size of the target display. As described above, encoder 570 compresses each of the modified video frames. For example, the corresponding modified video frames may be compressed into one or more encoded slices (compressed encoder slices), which may be further packetized for network streaming. The modified video frames compressed and / or packetized into encoded slices are then stored in a buffer 580 (e.g., a first-in-first-out or FIFO buffer). The streamer 575 is configured to transmit the encoded slices to the client 210 over the network 250. As previously mentioned, the streamer device may be configured to operate at the application layer of the Transmission Control Protocol / Internet Protocol (TCP / IP) computer networking model. In embodiments, assuming an IP-based network (e.g., home / internet), TCP / IP or UDP may be used. For example, cloud gaming services may use UDP. TCP / IP guarantees that all data will arrive. However, the "guaranteed arrival" comes at the cost of retransmissions, which introduce additional latency. On the other hand, UDP-based protocols offer the best latency performance, but at the cost of packet loss, which incurs data loss.

[0077] FIG. 5B-2 illustrates a scanout operation performed to send rendered video frames of a game, which may optionally include one or more additional features (e.g., layers), to an encoder when streaming content from a video game running on a cloud gaming server to a client over a network, according to one embodiment of the present disclosure. For example, FIG. 5B-2 illustrates the operation of scanout block 550-A2. Scanout block A2 (550-A2) receives rendered video frames of a game scanline by scanline. The configuration of the scan block 550-A2 in Fig. 5B-2 is similar to the scan block 550-A in Fig. 5B-1, and similar features have similar functions. The scanout block 550-A2 in Fig. 5B-2 differs from the scanout block 550-A in Fig. 5B-1 because it does not have a combiner 585. As a result, the rendered video frames of the game and the UX feature overlays can be composited and blended on the client side.

[0078] As shown, the information in each of the input frame buffers 580 is sent to a corresponding block 590 where additional operations are performed. That is, the additional operations outlined in block 590 are performed on each of the input frame buffers 580 to generate a corresponding layer. The additional operations may include DCC plane decompression, resolution scaling, color space conversion, de-gamma, HDR expansion, gamut remapping, LUT creation, tone mapping, gamma blending, etc. After these operations are performed, the modified layer or layers are sent separately to the encoder 570. The encoder delivers each layer separately to the client, which may composite and blend the layers to generate a modified video frame for display.

[0079] FIG. 5B-3 illustrates a scanout operation performed on a rendered video frame of a game to send to an encoder when streaming content from a video game running on a cloud gaming server to a client over a network, according to one embodiment of the present disclosure. For example, FIG. 5B-3 illustrates the operation of the scanout block 550B of FIG. 5A-2, where the scanout block 550-B does not have a combiner function. Some components of the scanout block 550-B are similar to the scanout block 550-A of FIG. 5B-1, and similar features have similar functions. The scanout block 550-B of FIG. 5B-3 differs from the scanout block 550-A of FIG. 5B-1 because it is a single input frame buffer without a combiner (e.g., to perform compositing and blending) and without separate feature generation. In particular, scanout block B (550-B) receives rendered video frames of the game from buffer 555 scanline by scanline. Optionally, user interface features may be integrated into the rendered video frames of the game generated by the CPU and / or GPU. For example, multiple rendered video frames of the game are output from game buffer 0 and game buffer 1 to input frame buffer 580 of scanout block B (550-B) as controlled by the flip control signal. The rendered video frames of the game are then provided to block 590, which may perform additional operations (e.g., decompression of DCC compressed planes, resolution scaling to the target display, color space conversion, etc.) to generate modified video frames suitable for display, as described above. Because optional UX features are already integrated into the rendered video frames of the game, additional operations may not be required to perform compositing and / or blending. In some implementations, additional operations outlined in block 590 may be performed in input frame buffer 580. Thus, for the target display, scanout block 550-B outputs a plurality of modified video frames to encoder 570 (e.g., at a rate defined by a corresponding speed setting). As described above, encoder 570 compresses each of the modified video frames into one or more encoded slices (compressed encoder slices), etc., which may be further packetized for network streaming. The modified video frames compressed and / or packetized into encoded slices are then stored in buffer 580. Streamer 575 is configured to transmit the encoded slices over the network to client 210, as described above.

[0080] 5C and 5D show an example server configuration including a scanout block having one or more input frame buffers used when performing fast scanout operations to send to an encoder when streaming content over a network from a video game running on a cloud gaming server, according to an embodiment of the present disclosure. In particular, FIGs. 5C and 5D show an example configuration of scanout block 550-A of FIG. 5A-1 and / or scanout block 550B of FIG. 5A-2 including one or more input frame buffers used to generate composite video frames to be displayed on a high-definition display or a virtual reality (VR) display (e.g., a head-mounted display). In one implementation, the input frame buffers may be implemented in hardware.

[0081] FIG. 5C illustrates a scanout block 550-A' that includes four input frame buffers that can be used to generate composite video frames for high-resolution displays. Purely by way of example, three input frame buffers (e.g., FB0, FB1, and FB2) may be dedicated to a video game and used to store and / or generate corresponding layers including at least one of a video frame, UI, eSports UI, and text layers. The video game input frame buffers may generate rendered video frames of the game from one or more viewpoints within the game environment. Another input frame buffer FB3 may be dedicated to the system and used to generate system overlays (e.g., UI), including friend notifications, etc.

[0082] FIG. 5D illustrates a scanout block 550-A″ that includes four input frame buffers that can be used to generate composite video frames for a VR display. Purely by way of example, two input frame buffers (e.g., FB0 and FB1) are dedicated to the video game and may be used to store and / or generate corresponding layers that include at least one of video frames taken from various perspectives of the game environment, UI, eSports UI, and text layers. Another two input frame buffers (FB2 and FB3) are dedicated to the system and may be used to generate system overlays (e.g., UI), including friend notifications, eSports UI, etc.

[0083] In an embodiment of the present disclosure, the server may perform fast and / or early scan-out / scan-in without considering display requirements and / or parameters since no physical display is attached to the server. In particular, the server may perform scan-out / scan-in to a target virtual display, which may be user defined to operate at a selected frequency (e.g., 93Hz, 120Hz).

[0084] In conjunction with detailed descriptions of the various client devices 210 and / or cloud gaming network 290 (e.g., within game server 260) in Figures 2A-2D, flow diagram 600 in Figure 6 illustrates a cloud gaming method that may perform fast and / or early scan-out operations to reduce one-way latency between the cloud gaming server and clients, according to one embodiment of the present disclosure.

[0085] At 610, the method generates video frames when executing the video game on the server. For example, the server may be running the video game in a streaming mode, whereby a CPU of the server executes the video game in part in response to input commands from a user or game logic that is not driven by control information from a user, to generate rendered video frames of the game using a graphics pipeline that is enabled for streaming. In particular, a CPU executing a video game in cooperation with a GPU graphics pipeline is configured to generate a plurality of video frames. In cloud gaming, the generated video frames of the game are typically rendered for display on a virtual display. The server may perform additional operations in a scan-out process of the generated video frames of the game. For example, one or more overlays may be added to the corresponding generated video frames of the game, such as during the scan-out process.

[0086] At 620, the method performs a scanout process by scanning multiple screen slices of a video frame into one or more input frame buffers scanline by scanline and performing one or more operations to modify the multiple screen slices. As described above, UX features (e.g., overlays) may be scanned into one or more input frame buffers. Thus, the one or more input frame buffers may be used to store and / or generate rendered video frames of a game of a video game and one or more optional UX features, such as user interface (UI), system UI, text, messaging, etc. (e.g., as an overlay). The scanout process generates modified video frames that are composited and blended to include one or more optional UX features, such as those implemented via an overlay. In one implementation, the UX features are first composited (e.g., as an overlay) and then additional operations are performed, as described above, including, for example, DCC surface decompression, resolution scaling, color space conversion, de-gamma, HDR expansion, gamut remapping, LUT creation, tone mapping, gamma blending, etc. In another implementation, additional operations are performed on each UX feature before compositing and blending, as described above.

[0087] At 630, after the modified video frames are generated, multiple screen slices of the modified video frames are scanned scanline by scanline into an encoder in a scanout process such that the generated modified video frames of the game (e.g., modified with an optional UX feature layer) are scanned into an encoder for compression in preparation for streaming the modified video frames to clients, such as when streaming a video game running on a cloud gaming server to the clients over a network.

[0088] In particular, at 640, the method starts the scanout process early. In one embodiment, multiple screen slices of the generated video frames of the game are scanned into one or more input frame buffers at corresponding flip times of the video frames. That is, instead of waiting for the next occurrence of a server VSYNC signal to start the scanout process, the modified video frames are scanned into the corresponding input frame buffers early (i.e., before the next server VSYNC signal). The flip time may be included in a command in the command buffer that, when executed by the GPU in the graphics pipeline, indicates that the GPU has finished executing commands in the command buffer and that a rendered video frame of the game has been fully loaded into the server's display buffer. The rendered video frame of the game is then scanned into a corresponding input frame buffer during the scanout process. Additionally, one or more optional UX features (e.g., overlays) are also scanned into the flip time corresponding to the one or more input frame buffers generated for the UX features.

[0089] In another embodiment, when streaming content over a network from a video game running on a cloud gaming server according to an embodiment of the present disclosure, the scanout process is performed at high speed. For example, the scanout process operates at a speed / rate corresponding to the client's target display, and is based on the server's maximum pixel clock and the target display's requested image size, as described above. For example, the scanout process includes receiving the rendered video frames of the game and feature overlays that are subsequently composited, and additional operations such as scaling, color scaling, blending, etc. may be performed on the composited video frames. As mentioned above, the scanout process outputs the modified video frames at a scanout rate based on a speed setting (e.g., 120 Hz), where the speed setting is based on the maximum pixel clock of the server and the required image size of the target display. In one embodiment, the speed setting is the frame rate. Thus, the scanout rate at which the modified video frames are output to the encoder can be higher than the rate at which the video frames are being generated and / or encoded.

[0090] Each modified video frame may be divided into one or more encoder slices and then compressed as one or more encoded slices. In particular, an encoder receives the modified video frames and encodes the modified video frames slice by slice at the encoder to generate one or more encoded slices. As mentioned above, the boundaries of the encoded slices are not limited to a single scan line and may consist of a single scan line, or multiple scan lines. Furthermore, the end of an encoded slice and / or the start of the next encoded slice do not necessarily occur at the edge of the display screen (e.g., they may occur in the center of the screen or the center of a scan line). In one embodiment, the VSYNC signals of the server and the client are synchronized and offset so that operations at the encoder may overlap. In particular, the encoder is configured to generate a first encoded slice of a modified video frame, where the modified video frame may include multiple encoded slices. The encoder may be configured to start compressing the first encoded slice before the modified video frame is completely received. That is, the first encoded slice may be encoded (e.g., compressed) before fully receiving the multiple screen slices of the modified video frame, the screen slices being delivered scanline by scanline. In some embodiments, depending on the number of processors or hardware, multiple slices may be encoded simultaneously (e.g., in parallel). For example, some game consoles may generate four encoded slices in parallel. More specifically, a hardware encoder may be configured to compress multiple encoder slices in parallel (e.g., to generate one or more encoded slices) for hardware pipelining.

[0091] 7A illustrates a process for generating and transmitting modified video frames at a cloud gaming server, where the process is optimized to perform fast and / or early scanout to the encoder to reduce one-way latency between the cloud gaming server and the client, according to one embodiment of the disclosure. The process is illustrated for generating and transmitting a single modified video frame modified at the server with additional UX features (e.g., overlays). The operations at the server include generating rendered video frames 490 of the game in operation 401. The scanout process 402 sends the rendered video frames 490 of the game to one or more input frame buffers of the scanout block to generate overlays to be composited. That is, the rendered video frames 490 of the game are composited with optional UX features (e.g., overlays). Additional operations (e.g., blending, resolution scaling, color space conversion, etc.) are performed on the composited video frames to generate modified video frames (e.g., modifying the rendered video frames of the game with additional UX features overlays). In the scanout process, the modified video frames are scanned into an encoder. The modified video frames are encoded (e.g., performing compression) slice-by-slice in the encoder into encoded video frames in operation 403. The compressed encoded video frames are transmitted from the server to the client in operation 404.

[0092] As mentioned above, the scanout process 402 is shown to be performed early, before the occurrence of the server VSYNC signal 311. Typically, scanout begins the next time the server VSYNC signal occurs. In one embodiment, early scanout is performed at flip time 701, which, as mentioned above, occurs when the GPU finishes generating the rendered frame 490.

[0093] By performing an early scanout process, the remaining server operations (e.g., encoding, transmitting, etc.) may also start early and / or overlap, thereby reducing one-way latency between the server and the client. In particular, additional time 725 is gained by performing an early scanout, the additional time being defined as between flip time 701 and the next occurrence of the server's VSYNC signal. The additional time 725 may offset adverse latency variations incurred during other operations, such as encoding 403 and transmitting 404. For example, if the encoding process 403 takes longer than a frame period, and the encoding process 403 starts early (e.g., is not synchronized to start with a VSYNC signal), the additional time gained may be sufficient for a video frame to be encoded before the next server VSYNC signal. Similarly, the additional time gained by performing an early scanout operation may be provided to reduce latency variations (e.g., increased delivery time over the network) in delivering a video frame to a client.

[0094] 7B illustrates timing of a scanout process performed on a cloud gaming server according to one embodiment of the disclosure, where the scanout is performed fast and / or early, so that video frames can be scanned into the encoder early at the end of the scanout process to reduce one-way latency between the cloud gaming server and the client. Typically, an application program (e.g., a video game) running on the server requests that a "flip" of the display buffer occur when rendering is complete. The flip occurs during execution of a flip command at flip time 701 during frame period 410, which is executed by a graphics processing unit (GPU). The flip command is one of multiple commands that are placed in the command buffer by the central processing unit (CPU) when the application is executed, and the command in the command buffer is used by the GPU to render the corresponding video frame. Thus, the flip indicates that the GPU has finished executing the commands in the command buffer to generate the rendered video frame, and the rendered video frame is fully loaded into the server's display buffer. There is a waiting period 725, after which the scanout process 402a is executed upon the occurrence of the subsequent server VSYNC signal 311f. That is, in a typical process, the scanout 402a is executed after the waiting period 725, and the modified video frame in the display buffer (e.g., the rendered video frame of the game composited and blended with optional UX feature overlays) is scanned into the encoder to perform video encoding. That is, the scanout process usually occurs upon the next VSYNC signal after the waiting period. This is true even if the display buffer is full beforehand.

[0095] An embodiment of the present disclosure provides early scanout 402b of the display buffer to the encoder, such as in cloud gaming applications. As shown in FIG. 7B, the scanout process 402b is triggered early at flip time 701 instead of when the next server VSYNC signal 311f occurs. This allows the encoder to start encoding early in overlapping operations, instead of waiting for the next server VSYNC signal to perform a scanout to send to the encoder for encoding / compression. Display timing is not affected because there is no actual display attached to the server. As mentioned above, early encoding reduces one-way latency between the server and the client, since the processing of complex video frames is less likely to miss one or more VSYNCs that are destined for delivery to and / or display at the client.

[0096] FIG. 7C illustrates a period for performing a scanout at a high speed so that one-way latency between the cloud gaming server and the client can be reduced by scanning video frames into the encoder early, according to one embodiment of the present disclosure. In particular, the scanout process may be performed at a high speed when streaming content over a network from a video game running on the cloud gaming server, and the scanout process operates at a speed / rate corresponding to the client's target display, which is based on the server's maximum pixel clock and the target display's requested image size, as described above. Thus, the scanout rate at which modified video frames are output to the encoder may be higher than the rate at which the video frames are being generated and / or encoded. That is, the scanout rate may not correspond to the rate at which the video game is generating video frames. For example, the scanout rate (e.g., frame rate setting) is higher than the frequency of the server VSYNC signal used to generate video frames when running the video game on the server.

[0097] In another embodiment, the scan-out rate may not correspond to the refresh rate of the client's display device (e.g., 60 Hz, etc.). That is, the display rate of the client's display device and the scan-out rate may not be the same rate. For example, the display rate of the client's display device may be 60 Hz, or a variable refresh rate, etc., and the scan-out rate is a different rate (e.g., 120 Hz, etc.).

[0098] Typically, the video frame scan-out process takes place over an entire frame period (e.g., 16.6 ms at 60 Hz). For example, one representative frame period 410 is shown between two server VSYNC signals 311c and 311d. In an embodiment of the present disclosure, instead of performing the scanout process on the rendered video frames for the entire frame period, the scanout is performed at a higher rate. By performing the scanout process (e.g., including the scan into the encoder) at a rate (e.g., 120 Hz or higher) higher than the rate of the processed frames (e.g., 60 Hz), it is possible to start the encoding process earlier, such as when waiting for the scanout process 402 to finish before starting the encoding 403, or when the scanout 402 and the encoding 403 overlap. For example, the scanout process 402 can be performed for a period 730 (e.g., about 8 ms) that is shorter than the entire frame period 410 (e.g., 16.6 ms at 60 Hz).

[0099] In some cases, encoding can begin earlier, such as before the next server VSYNC signal occurs. In particular, the encoder may begin processing as soon as a minimum amount of data (e.g., 16 or 64 scanlines) from a corresponding modified video frame (e.g., a rendered video frame of a game that has been modified using one or more optional UX features as an overlay) has been sent to the encoder, and then processes additional data as soon as it arrives at the encoder. Processing of complex video frames may reduce one-way latency due to a lower likelihood of missing one or more VSYNCs for delivery to and / or display at the client. One-way latency may be due to network jitter and / or increased processing time at the server. For example, a modified video frame containing a large amount of data (e.g., a scene change) may take multiple frame periods to encode. A faster scan-out process leaves more time for encoding, and a modified video frame containing a large amount of data is more likely to complete the encoding process before the server VSYNC signal for delivery to the client.

[0100] In another embodiment, the encoding process may be further optimized to ensure a minimum amount of time for encoding by limiting the encoding resolution to that required by the client display, thereby not wasting time encoding higher resolution video frames than the client display can process or request at a particular point in time.

[0101] In conjunction with detailed descriptions of the various client devices 210 and / or cloud gaming network 290 (e.g., at the game server 260) of FIGS. 2A-2D, flow diagram 800A of FIG. 8A illustrates a method for cloud gaming in a cloud gaming application according to one embodiment of the present disclosure that may smooth video displayed at the client and may perform fast and / or early scanout operations at the server to reduce one-way latency between the cloud gaming server and the client.

[0102] At 810, the method generates video frames when executing the video game on the server. For example, the cloud gaming server may be running the video game in a streaming mode, whereby a CPU executes the video game in response to input commands from a user to generate rendered video frames of the game using a graphics pipeline.

[0103] The server may perform additional operations on the generated video frames of the game in the scanout process. For example, one or more overlays may be added to the generated video frames of the corresponding game, such as during the scanout process. In particular, the method includes performing a scanout process at 820 to generate modified video frames and send the modified video frames to an encoder configured to compress the video frames. The scanout process includes scanning the video frames scanline-by-scanline into one or more input frame buffers, scanning one or more user interface features scanline-by-scanline, and compositing and blending the video frames and one or more user interface (UX) features (e.g., as an overlay including user interface (UI), system UI, text, messaging, etc.) into the modified video frames, the scanout process beginning at the flip time of the video frames. Thus, the scanout process generates modified video frames that are composited and blended to include one or more optional UX features, such as those implemented via an overlay.

[0104] At 830, the method transmits the compressed modified video frames to the client. In particular, each modified video frame may be divided into one or more encoder slices and then compressed by the encoder as one or more encoded slices. That is, the encoder receives the modified video frames and encodes the modified video frames slice by slice in the encoder to generate one or more encoded slices. These slices are then packetized and delivered to the client over the network.

[0105] At 840, the method determines a target display time for the modified video frame at the client. In particular, if the scanout of the server display buffer occurs at the flip time, rather than at the occurrence of the next server VSYNC signal, then the client-side ideal display timing can be performed based on the time the scanout occurred at the server and the intent of the game with respect to the particular display buffer (e.g., the target display buffer VSYNC). The intent of the game determines whether the frame was intended for the next client VSYNC, or was actually intended for the previous client VSYNC because the game was late in processing that frame.

[0106] At 850, the method schedules display times for the modified video frames at the client based on the target display times. The client-side strategy for selecting when to display the frames may depend on whether the game is designed for a fixed or variable frame rate and whether the VSYNC timing information is implicit or explicit, as will be further described below in connection with FIG. 8B.

[0107] 8B illustrates a timing diagram of server and client operations performed during execution of a video game on the server 260 to generate rendered video frames that are then sent to the client 210 for display, according to one embodiment of the present disclosure. Since the client is aware of various timing parameters associated with each of the server-generated rendered video frames that can be used to indicate and / or determine ideal display times, the client can determine when to display those video frames based on one or more strategies. In particular, the ideal display times of the corresponding server-generated rendered video frames indicate when the game application executing on the server intends to display the rendered video frames, with reference to the occurrence of a target server VSYNC signal. The target server VSYNC signal can be converted to a target client VSYNC signal, particularly when the server and client VSYNC signals are synchronized (e.g., frequency and timing) and aligned using an appropriate offset.

[0108] Desirable synchronization and alignment between the server and client VSYNC signals is shown in FIG. 8B. In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are synchronized such that they have the same frequency and corresponding frame period. For example, the frame period 410 of the server VSYNC signal 311 is substantially equal to the frame period 415 of the client VSYNC signal 312. Additionally, the server and client VSYNC signals may be aligned with an offset 430. The timing offset may be determined such that a predetermined number (e.g., 99.99 percent) of the received video frames arrive at the client and are displayed at the next proper occurrence of the client VSYNC signal. More particularly, the offset is set such that the video frame received within the predetermined number and with the greatest variation in one-way latency between the server and the client arrives just before the next proper occurrence of the client VSYNC signal for display purposes. Proper synchronization and alignment allows for the use of ideal display times for server-generated video frames that can be translated between the server and the client.

[0109] In one embodiment, the timing parameters include an ideal display time at which the corresponding video frame is intended to be displayed. The ideal display time may refer to the occurrence of a targeted server VSYNC signal, i.e., the ideal display time is explicitly provided in the timing parameters. In one embodiment, the timing parameters can be delivered from the server to the client via some mechanism in one of the packets used to deliver the encoded video frame. For example, the timing parameters can be added to a packet header, or the timing parameters can be part of the encoded frame data of the packet. In another embodiment, the timing parameters can be delivered from the server to the client by sending a data control packet using a GPU API. The GPU API can be configured to send the data control packet from the server to the client over the same data channel used to send the compressed rendered video frame. The data control packet is formatted so that the client understands what type of information is included and the appropriate reference to the corresponding rendered video frame. In one embodiment, the communication protocol used for the GPU API, the format of the data control packets may be defined in the corresponding software development kit (SDK) of the video game, and the signaling information provided such as client notification of the data control packets (e.g., provided in a header, provided in a data packet with a marker, etc.) In one embodiment, the data control packets are of a minimum size and therefore bypass the encoding process.

[0110] In another embodiment, the timing parameters include a flip time and a simulation time that are delivered from the server to the client, as described above. The flip time and the simulation time can be used by the client to determine an ideal display time. That is, the ideal display time is implicitly provided in the timing parameters. The timing parameters may include other information that can be used to infer the ideal display time. In particular, the flip time indicates when a display buffer flip occurs, indicating that the corresponding rendered video frame is ready for transmission and / or display. In one embodiment, the scan-out / scan-in process also occurs early in the flip time. Simulation time refers to the time it takes to render a video frame through the CPU and GPU pipelines. Determining the ideal display time of a corresponding video frame depends on whether the game is running at a fixed frame rate or a variable frame rate.

[0111] For fixed frame rate games, the client may implicitly determine the target VSYNC timing information from the scan out / scan in timing (e.g., flip time timestamps) and the corresponding simulation time. For example, the server records and sends the scan out / scan in time of the corresponding video frame to the client. The client may infer the scan out / scan in timing and the corresponding simulation time, which is the occurrence time of the target server VSYNC signal, which can be converted to the occurrence time of the target client VSYNC signal. The client may explicitly determine the target timing information when the game provides ideal display timing (e.g., via a GPU API), which may be integral VSYNC timing or fractional VSYNC timing (also referred to as fractions of VSYNC timing). Fractional VSYNC timing may be implemented when the processing time of a frame exceeds the frame period, and the ideal display timing may specify or be based on the simulation time.

[0112] For variable frame rate games, the client may implicitly determine the ideal target VSYNC timing information from the scan out / scan in timings and the simulation times of the corresponding video frames. For example, the server records and sends to the client the scan out time and the simulation time of the corresponding frames. The client may infer from the scan out / scan in timings and the simulation times which are the target server VSYNC signal occurrences for displaying the corresponding video frames, where the target VSYNC signal can be converted to the corresponding target client VSYNC signal occurrences. Furthermore, when the game provides ideal timings via the GPU API, the client may explicitly determine the target VSYNC timing information. In this case, fractional VSYNC timings may be specified by the game, such as providing simulation times and display times.

[0113] As shown in Figure 8B, server VSYNC signal 311 and client VSYNC signal 312 occur at a 60 Hz timing. Server VSYNC signal 311 is synchronized (e.g., substantially equal frequency) and aligned (e.g., with an offset) with client VSYNC signal 312. For example, the occurrence of the server VSYNC signal may be aligned with the occurrence of the client VSYNC signal. In particular, the occurrence of server VSYNC signal 311a corresponds to the occurrence of client VSYNC signal 312a, server VSYNC signal 311c corresponds to client VSYNC signal 312c, server VSYNC signal 311d corresponds to client VSYNC signal 312d, server VSYNC signal 311e corresponds to client VSYNC signal 312e, etc.

[0114] For purposes of illustration, server 260 is executing a video game running at 30 Hz, whereby rendered video frames are generated during a frame period (33.33 milliseconds) at 30 Hz (e.g., corresponding to a frame period of 30 frames per second). Thus, the video game may render up to 30 frames per second. Ideal display timings for the corresponding video frames are also shown. The ideal display timing may reflect the game's intention to display the video frames. As mentioned above, the ideal display timing may be determined from the flip time of each frame, which is also shown. The ideal display time can be used by the client to determine when to display the video frames depending on the strategy adopted, as described below. For example, video frame A is rendered and ready to be displayed with a flip time of 0.6 (e.g., 0.6 / 60 at 60 Hz). Also, the ideal display timing of video frame A is targeted for display at the occurrence of the server VSYNC signal 311a, which translates to targeted display at the client at the client VSYNC signal 312a. Similarly, video frame B is rendered and ready to be displayed with a flip time of 2.1 (e.g., 2.1 / 60 at 60 Hz). The ideal display timing of video frame B is targeted for display at the time of server VSYNC signal 311c, which translates to targeted display at the client with client VSYNC signal 312c. Also, video frame C is rendered and ready for display with a flip time of 4.1 (e.g., 4.1 / 60 at 60Hz). The ideal display timing of video frame C is targeted for display at the time of server VSYNC signal 311e, which translates to targeted display at the client with client VSYNC signal 312e. Also, video frame D is rendered and ready for display with a flip time of 7.3 (e.g., 7.3 / 60 at 60Hz). The ideal display timing of video frame D is targeted for display at the time of server VSYNC signal 311g, which translates to targeted display at the client with client VSYNC signal 312g.

[0115] 8B, video frame D took longer than expected to generate, so the flip time for video frame D occurs at 7.3, which is after the targeted occurrence of server VSYNC signal 311g. That is, server 260 should have completed rendering video frame D before server VSYNC signal 311g occurs. However, because the ideal display time for video frame D is known or can be determined, even if the server misses the timing to generate the video frame, the client can still display video frame D upon the occurrence of client VSYNC signal 312g that aligns with the ideal display time (e.g., server VSYNC signal 311g).

[0116] Another problem illustrated in FIG. 8B is that video frames B and C are generated at the server 260 at the appropriate time (e.g., targeted for display at different server VSYNC signals), but due to the additional latency experienced during transmission, video frames B and C are received at the client within the same frame period, and therefore appear to both be targeted for display at the client upon the occurrence of the same client VSYNC signal 312d. For example, delayed transmission will cause video frames B and C to arrive in the same frame period. However, with proper buffering and knowledge of the ideal display timings for both video frames B and C, the client can decide when and how to display these video frames depending on the strategy implemented, including following the game intent, prioritizing latency, prioritizing smoothness, or adjusting client-side VBI settings for variable refresh rate display.

[0117] For example, one strategy is to follow the game intent determined during execution on the server. The intent may be inferred from the timing of the flip times of corresponding video frames, such that video frames A, B, and C are intended for display at the next server VSYNC signal. The intent may be explicitly known as communicated by the video game, such that video frame D is intended for display at the previous server VSYNC signal 311e, even though it finished rendering after that VSYNC signal. Furthermore, the ambiguity of video frames B and C arriving similarly (e.g., within the same frame period) at the client is resolved by following the game intent. So, with appropriate buffering, the client may display video frames at 60Hz (each frame is displayed for 16.66ms) in an order such as AAABCCDD.

[0118] The second strategy is to prioritize latency over smoothness of frame presentation, whereby the goal is to reduce latency as much as possible using a minimal amount of buffering. That is, video frames are displayed with rapid latency by displaying the most recently received video frame at the next client VSYNC signal. In this way, ambiguity between video frames B and C that arrive similarly at the client (e.g., within the same frame period) may be resolved by dropping video frame B and displaying only video frame C at the next client VSYNC signal. This will skip video frame B in the sequence of displayed video frames, resulting in a less smooth frame display that may be noticeable to the viewer. So, with appropriate buffering, a client may display video frames at 60Hz (each frame is displayed for 16.66ms) in an order such as AAACCCDD.

[0119] A third strategy is to prioritize smoothness of frame presentation over latency. In this case, the additional latency is not a factor and can be addressed by appropriate buffering. That is, video frames are presented in a way that provides the best viewing experience for the viewer. The client uses the time between target VSYNCs as a guide. For example, the time between B target 312c and C target 312e is two VSYNCs, so B should be displayed for two frames regardless of the arrival times of B and C to the client. The time between C target 312e and D target 312g is two VSYNCs, so C should be displayed for two frames regardless of the arrival times of C and D to the client. Thus, with appropriate buffering, the client may present video frames in an order such as AAABBCCDD at 60Hz (each frame is displayed for 16.66ms).

[0120] A fourth strategy provides for adjusting the client-side VBI timing for a display that supports variable refresh rate. That is, a variable refresh rate display can increase or decrease the VBI interval when displaying video frames to implement an instantaneous frame rate for displaying video frames rendered at the client for display. For example, instead of displaying a rendered video frame for display at the client at each client VSYNC signal, which may require displaying the video frame twice when waiting for a delayed video frame, the refresh rate of the display can be dynamically adjusted for each video frame rendered for display. Thus, the video frames displayed may be adjusted to account for variations in latency in receiving, decoding, and rendering the display video frames at the client. In the example shown in Figure 8B, video frames B and C are generated at the server 260 at the appropriate times (e.g., intended for display on different server VSYNC signals), but due to additional latency incurred during transmission, video frames B and C are received at the client within the same frame period. In this case, video frame B may be displayed for a shorter period than intended (e.g., less than a frame period), allowing video frame C to be rendered with a client-determined, intended client VSYNC signal. For example, video frame C may have an occurrence of an intended server VSYNC signal, which is then converted to an intended client VSYNC signal, particularly when the server and client VSYNC signals are synchronized (e.g., frequency and timing) and aligned using an appropriate offset.

[0121] FIG. 9 illustrates components of an exemplary device 900 that can be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 9 illustrates an exemplary hardware system suitable for streaming media content and / or receiving streamed media content, including performing a fast scan-out operation or performing an early scan-out, such as before the occurrence of the next system VSYNC signal or at the flip time of the corresponding video frame, to send modified video frames to an encoder when streaming content over a network from a video game running on a cloud gaming server, according to embodiments of the present disclosure. The block diagram illustrates a device 900, which may incorporate a personal computer, a server computer, a game console, a mobile device, or other digital device, each of which is suitable for practicing embodiments of the present invention. The device 900 includes a central processing unit (CPU) 902 for executing software applications and optionally an operating system. The CPU 902 may be comprised of one or more homogeneous or heterogeneous processing cores.

[0122] According to various embodiments, CPU 902 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs having a microprocessor architecture specifically adapted for highly parallel and computationally intensive applications, such as applications configured for graphics processing during game execution, media and interactive entertainment applications.

[0123] Memory 904 stores applications and data used by CPU 902 and GPU 916. Storage 906 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 908 communicate user input from one or more users to device 900, and examples of user input devices 908 may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, and / or microphone. The network interface 909 enables the device 900 to communicate with other computer systems over electronic communications networks, which may include local area networks and wired or wireless communications over wide area networks such as the Internet. The audio processor 912 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 902, the memory 904, and / or the storage 906. The components of the device 900, including the CPU 902, the graphics subsystem 914 including the GPU 916 and GPU cache 918, the memory 904, the data storage 906, the user input devices 908, the network interface 909, and the audio processor 912, are connected via one or more data buses 922.

[0124] The graphics subsystem 914 is further coupled to the data bus 922 and to the components of the device 900. The graphics subsystem 914 includes a graphics processing unit (GPU) 916 and a graphics memory 918. The graphics memory 918 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 918 may be integrated into the same device as the GPU 916, may be coupled as a separate device from the GPU 916, and / or may be implemented within the memory 904. The pixel data may be provided to the graphics memory 918 directly from the CPU 902. Alternatively, the CPU 902 provides data and / or instructions defining a desired output image to the GPU 916, based on which the GPU 916 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 904 and / or the graphics memory 918. In one embodiment, the GPU 916 includes 3D rendering capabilities that generate pixel data for output images from instructions and data defining scene geometry, lighting, shading, texturing, motion, and / or camera parameters. The GPU 916 may further include one or more programmable execution units capable of executing shader programs.

[0125] Graphics subsystem 914 periodically outputs pixel data for images from graphics memory 918 to be displayed on display device 910 or projected by a projection system (not shown). Display device 910 may be any device, including CRT, LCD, plasma, and OLED displays, capable of displaying visual information in response to signals from device 900. Device 900 may provide analog or digital signals, for example, to display device 910.

[0126] Other embodiments for optimizing the graphics subsystem 914 may include multi-tenancy GPU operations where a GPU instance is shared among multiple applications, and distributed GPUs supporting a single game. The graphics subsystem 914 may be configured as one or more processing devices.

[0127] For example, in one embodiment, graphics subsystem 914 may be configured to perform multi-tenancy GPU functions, where one graphics subsystem can implement the graphics and / or rendering pipeline for multiple games, i.e., graphics subsystem 914 is shared between multiple running games.

[0128] In other embodiments, the graphics subsystem 914 includes multiple GPU devices that are combined to perform graphics processing for a single application running on a corresponding CPU. For example, the multiple GPUs may perform alternative forms of frame rendering, where in successive frame cycles, GPU1 renders a first frame, GPU2 renders a second frame, and when the last GPU is reached, the first GPU renders the next video frame (e.g., if there are only two GPUs, GPU1 renders the third frame). That is, the GPUs take turns when rendering frames. Rendering operations may overlap, such that GPU2 may start rendering the second frame before GPU1 has finished rendering the first frame. In another embodiment, multiple GPU devices may be assigned different shader operations in the rendering and / or graphics pipeline. A master GPU performs the main rendering and compositing. For example, in a group including three GPUs, master GPU1 may perform the main rendering (e.g., first shader operations) and compositing the output from slave GPU2 and slave GPU3, slave GPU2 may perform a second shader (e.g., fluid effects such as rivers) operation, slave GPU3 may perform a third shader (e.g., particle smoke) operation, and master GPU1 composites the results from each of GPU1, GPU2, and GPU3. In this way, different GPUs may be assigned to perform different shader operations (e.g., flag waving, wind, smoke generation, fire, etc.) for rendering a video frame. In yet another embodiment, each of the three GPUs can be assigned to a different object and / or portion of the scene corresponding to the video frame. In the above embodiments and implementations, these operations can be performed in the same frame period (concurrently in parallel) or in different frame periods (sequentially, in parallel).

[0129] Accordingly, the present disclosure describes methods and systems configured for streaming media content and / or receiving streamed media content that include performing a fast scanout operation, or performing an early scanout, such as before the occurrence of the next system VSYNC signal or at the flip time of the corresponding video frame, to send modified video frames to an encoder when streaming content over a network from a video game running on a cloud gaming server.

[0130] Of course, the various embodiments defined herein may be combined or assembled into specific implementations using various features disclosed herein. Thus, the examples provided are only some possible examples and are not limited to the various implementations possible by combining various elements to define more implementations. In some examples, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0131] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0132] With the above embodiments in mind, it will be appreciated that embodiments of the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The disclosed embodiments also relate to devices or apparatus for performing these operations. The apparatus can be specially constructed for the required purposes. Alternatively, the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0133] The present disclosure can also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data which can subsequently be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible media distributed across network-coupled computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0134] Although the operations of the method have been described in a particular order, it will be appreciated that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals relative to processing, so long as the processing of the overlay operations is performed in the desired manner.

[0135] Although the above disclosure has been described in some detail for clarity of understanding, it will be understood that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the embodiments of the present disclosure should not be limited to the details provided herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. 1. A cloud gaming method, comprising: generating video frames within a frame period when executing the video game on the server; generating one or more user interface features for the video frames; performing a scanout process, in which the video frames are scanned line by line and the one or more user interface features are scanned line by line into one or more frame buffers, and the video frames and the one or more user interface features are composited and blended into a modified video frame; In the scanout process, scanning the modified video frames scanline by scanline into an encoder of the server; The method of claim 1, wherein the scanout process begins scanning the video frames and begins scanning the one or more user interface features into the one or more input frame buffers at video frame flip times that occur within the video frames.

2. generating a first encoded slice of the video frame in the encoder; beginning compression of the first encoded slice in the encoder before the modified video frame is completely received; The method of claim 1.

3. the flip time is included in a command in a command buffer that, when executed by a graphics processing unit (GPU), indicates that the GPU has finished executing the commands in the command buffer and that the video frame has been completely loaded into the server's display buffer; The method of claim 1.

4. In the scan out process, At least one of the following is performed: DCC compression plane decompression, or resolution scaling, or color space conversion, or de-gamma, or HDR expansion, or color gamut remapping, or LUT creation, or tone mapping, or gamma blending. The method of claim 1.

5. determining a frame rate setting based on a requested image size and a maximum pixel clock for the client display; Determine the speed setting, performing the scanout process at the speed setting; scanning a plurality of screen slices of the modified video frames into the encoder at the frame rate setting; The method of claim 1.

6. the maximum pixel clock is determined from a chip calculation setting or by a self-diagnostic test; The method according to claim 5.

7. The speed setting is the frame rate setting. The method according to claim 5.

8. the frame rate setting is greater than a frequency of a server VSYNC signal used to generate the video frames when executing the video game on the server; The method according to claim 5.

9. performing encoding at a resolution limit set by said client display; The method according to claim 5.

10. A computer-readable medium storing a computer program for cloud gaming, the computer-readable medium comprising: having program instructions for generating video frames within a frame period when executing the video game on the server; having program instructions for generating one or more user interface features for the video frames; a program instruction for executing a scanout process, the scanout process including: scanning the video frames scanline by scanline; scanning the one or more user interface features scanline by scanline into one or more frame buffers; and combining and blending the video frames and the one or more user interface features into a modified video frame; and program instructions for scanning the modified video frames scanline by scanline into an encoder of the server during the scanout process; A computer-readable medium having program instructions for initiating scanning of the video frames in the scanout process and scanning the one or more user interface features into the one or more input frame buffers at video frame flip times that occur within the video frames.

11. program instructions for generating in the encoder a first encoded slice of the video frame; and program instructions for starting compression of the first encoded slice in the encoder before the modified video frame is completely received. The computer-readable medium of claim 10.

12. A computer system comprising: a processor; a memory coupled to the processor and storing instructions that, when executed by the computer system, cause the computer system to perform a method for cloud gaming, the method comprising: generating video frames within a frame period when executing the video game on the server; generating one or more user interface features for the video frames; performing a scanout process, in which the video frames are scanned line by line and the one or more user interface features are scanned line by line into one or more frame buffers, and the video frames and the one or more user interface features are composited and blended into a modified video frame; In the scanout process, scanning the modified video frames scanline by scanline into an encoder of the server; The computer system begins scanning the video frames in the scanout process and begins scanning the one or more user interface features into the one or more input frame buffers at video frame flip times that occur within the video frames.

13. In the method, generating a first encoded slice of the video frame in the encoder; beginning compression of the first encoded slice in the encoder before the modified video frame is completely received; 13. The computer system of claim 12.

14. the method further comprising: including the flip time in a command in a command buffer that, when executed by a graphics processing unit (GPU), indicates that the GPU has finished executing the commands in the command buffer and that the video frame has been completely loaded into the server's display buffer; 13. The computer system of claim 12.

15. In the method, the scan-out process includes: At least one of the following is performed: DCC compression plane decompression, or resolution scaling, or color space conversion, or de-gamma, or HDR expansion, or color gamut remapping, or LUT creation, or tone mapping, or gamma blending.

13. The computer system of claim 12.

16. The method further comprises: determining a frame rate setting based on a requested image size and a maximum pixel clock for the client display; Determine the speed setting, performing the scanout process at the speed setting; scanning a plurality of screen slices of the modified video frames into the encoder at the frame rate setting; 13. The computer system of claim 12.

17. The method further comprises determining the maximum pixel clock from a chip calculation setting or by a self-test.

17. The computer system of claim 16.

18. In the method, the speed setting is the frame rate setting.

17. The computer system of claim 16.

19. the method further comprising: the frame rate setting being greater than a frequency of a server VSYNC signal used to generate the video frames when executing the video game on the server; 17. The computer system of claim 16.

20. In the method, performing encoding at a resolution limit set by said client display; 17. The computer system of claim 16.

Images