Synchronizing remote displays to preserve local displays

Abstract

The remote display synchronization technique maintains the presence of a local display device for a remotely rendered video stream. The server and client devices cooperate to dynamically determine a target frame rate for the stream of rendered frames appropriate to the current capabilities and networking conditions of the server and client devices. From this target frame rate, the server generates a synchronization signal that serves as a timing control for the rendering process. The client device can provide feedback to prompt changes in the target frame rate and, therefore, corresponding changes in the synchronization signal. In this approach, the rendering frame rate and encoding frequency can be "synchronized" in a manner consistent with the capabilities of the server, network, and client device, resulting in the generation, encoding, transmission, decoding, and presentation of a stream of frames that mitigates lost encoding of frames while providing acceptable latency.

Description

[Background Technology]

[0001] A remote or "cloud-based" video game system uses a remote server to receive user game input from a client device over a network, run an instance of a video game based on the user game input, and deliver a stream of rendered video frames and a corresponding audio stream to the client device over the network for presentation on the client device's display and speakers, respectively. Typically, for video content, the server uses a rendering and capture process in which the server renders a stream of video frames from the video game application and an encoder encodes the pixel content of the stream of video frames into an encoded stream of video frames that are simultaneously transmitted to the client device. Typically, the rendering and capture processes are decoupled. The server executes a streaming pipeline process in which the server runs the video game, renders frames, and in parallel runs another application to capture and encode the rendered frames. Due to the decoupling of frame rendering from capture and encoding, the encoding frequency does not match the rendering frame rate, which typically leads to the encoder "missing" frames (i.e., not encoding frames for inclusion in the transmitted encoded stream) and relatively large and inconsistent latency across the stream of frames. Thus, if the rendering frame rate is maintained at a frame rate lower than the encoding frequency, this can reduce latency and avoid missing frames, but may result in a lower display frame rate at the client device than would otherwise be available.Conversely, if the encoding frequency is lower than the rendering frame rate, rendered frames will be "dropped" by the encoder as it works to keep up with the rendering process, which leads to the large and inconsistent latency described above, while also wasting power in generating rendered frames that are missed and therefore ultimately not presented at the client device. Furthermore, the decoupling of the server frame rendering and encoding rate from the client decoding and display rate leads to greater latency and / or dropped frames.

[0002] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of a server-based video game system, according to some embodiments. [Figure 2] 2 is a block diagram illustrating a hardware configuration of a client device of the video game system of FIG. 1 in accordance with some embodiments. [Figure 3] 2 is a block diagram illustrating a hardware configuration of a game server of the video game system of FIG. 1 according to some embodiments. [Figure 4] FIG. 1 is a flow diagram illustrating a method for remote frame rate synchronization between a server and client devices of a server-based video game system, according to some embodiments. [Figure 5] FIG. 1 is a flow diagram illustrating a method for rendering rate control for remote frame rate synchronization based on selective present call blocking, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] Pixel streaming approaches in remote video game systems and other systems that remotely render graphics content for display on a local device rely on frame rendering and frame capture (encoding) processes (i.e., “streaming processes”) that may have disparate capabilities, resulting in relatively high or variable latency (when rendering capabilities exceed encoding capabilities) or an artificially reduced frame rate on the local device (when rendering capabilities are kept below encoding capabilities). To mitigate such capabilities, FIGS. 1-5 illustrate systems and methods for remote display synchronization techniques that preserve the presence and characteristics of the local display. In at least one embodiment, the server and client devices cooperate to determine a target frame rate appropriate for both the server's current capabilities as well as the client device's current capabilities and the network(s) connecting the two. From this target frame rate, the server then generates a series of frame rate synchronization signals (referred to herein as “remote sync” or “RSync”) that serve as timing controls for the current cadence for the rendering and encoding processes. Additionally, on a periodic basis or in response to one or more triggers, the client device can provide update feedback to prompt changes in the target frame rate and, therefore, corresponding changes in the synchronization signal. In this approach, the rendering frame rate and encoding frequency can be "synchronized" in a manner consistent with the capabilities of the server, network, and client device, resulting in the generation, encoding, transmission, decoding, and presentation of a stream of rendered video frames that mitigates missed encoding of rendered frames while maintaining acceptable latency. Additionally, the server and client device can cooperate to communicate various display characteristics implemented by a running video game for implementation on the client device.For example, a video game application may run with vertical synchronization (vsync) either enabled or disabled, and this vsync status may be communicated from the server to the client device for implementation of the indicated vsync status at the client device.

[0005] It should be noted that for ease of explanation, the systems and techniques of this disclosure are described in the exemplary context of a server-based video game system, in which an instance of a video game application runs remotely and its rendered video and audio output is encoded and streamed to a client device, which then decodes the video and audio content and presents the resulting decoded content to a user. However, these systems and techniques are not limited to the context of a server-based game, but instead can be used in any of a variety of scenarios in which real-time video content is rendered remotely, and the resulting pixel content is then remotely encoded and delivered to one or more client devices as an encoded video stream. Accordingly, references to a "video game" apply equally to such equivalent scenarios, unless otherwise indicated.

[0006] 1 illustrates a remote video game system 100 for providing server-based video games, according to some embodiments. The system 100 includes a video game server 102 (hereinafter, for brevity, "game server 102") remotely connected to at least one client device 104 via one or more networks 106. The client device 104 may include, for example, a smartphone, a computer-enabled vehicle entertainment system, a computer-enabled appliance, a tablet computer, a laptop computer, a desktop computer, a video game console, a television, etc. The one or more networks 106 may include a personal area network (PAN), one or more wireless networks such as a cellular network or a wireless local area network (WLAN), one or more wired networks such as a local area network (LAN) or a wide area network (WAN), the Internet, or a combination thereof.

[0007] 2 and 3, exemplary hardware configurations for a client device 104 and a game server 102 are shown, respectively. As shown in FIG. 2, the illustrated hardware configuration 200 for a client device 104 includes one or more I / O devices 202, including a network interface 202-1 for interfacing with one or more networks 106, one or more central processing units (CPUs) 204, one or more graphics processing units (GPUs) 206, one or more memories 208, and a display 210 that is integrated with or otherwise connected to the client device 104. Other well-known hardware components typically implemented in client devices, such as speakers, microphones, power supplies, buses, power managers, etc., are omitted for clarity. The one or more memories 208 may include one or more types of memory, such as random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive, register file, etc., and store one or more sets of executable instructions that, when executed by one or more CPUs 204 and / or one or more GPUs 206, cause the hardware of the client device 104 to perform the functions attributed to the client device 104 herein. In particular, the executable instructions may execute an operating system (OS) 214 for overall control and coordination of the hardware components of the client device 104, a set of graphics (GFX) drivers 216, such as user mode drivers (UMD) and kernel mode drivers (KMD), for coordination and control of the one or more GPUs 206 by the one or more CPUs 204, and a client streaming application 218. This client streaming application 218, in turn, may be executed by the game server 102, as described below. The game server 102 includes a frame rate control module 220 for adjusting the rendering frame rate, as well as a decoder 222 that operates to decode encoded video frames received from the game server 102 and buffered, for example, in an input queue 224.

[0008] 3, the illustrated hardware configuration 300 for a game server 102 includes one or more I / O devices 302, including a network interface 302-1 for interfacing with one or more networks 106, one or more CPUs 304, one or more GPUs 306, and one or more memories 308 (e.g., RAM, ROM, flash memory, hard disk drives, or combinations thereof). Other well-known hardware components typically implemented in a game server, such as power supplies, buses, power managers, etc., are omitted for clarity. The one or more memories 308 store one or more sets of executable instructions that, when executed by the one or more CPUs 304 and / or one or more GPUs 306, cause the hardware of the game server 102 to perform the functions attributed to the game server 102 herein. In particular, the executable instructions may execute an OS 314 (and a hypervisor for multi-tenancy implementations) for overall control and coordination of the hardware components of the game server 102, a set of graphics (GFX) drivers 316, such as KMD 316-2 and UMD 316-1, for coordination and control of one or more GPUs 306 by one or more CPUs 304, and a server streaming application 318. The server streaming application 318, in turn, includes a frame rate control module 320 for coordinating with a frame rate control module 220 of a client device 104 to set the rendering frame rate by the game server 102, as described below, and an encoder 322 that operates to encode video frames rendered by a running instance of a video game application 324.

[0009] Referring to Figure 4, a method 400 for remote display synchronization for the video game system 100 of Figures 1-3 is shown, according to at least one embodiment. Although method 400 is described in the exemplary context of the video game system 100 of Figure 1 and the corresponding hardware configurations of the client device 104 and game server 102 of Figures 2 and 3, respectively, method 400 can be performed in other scenarios in which video content is rendered remotely and corresponding pixel content is encoded and transmitted in real time using guidelines provided herein. The illustrated method 400 includes three simultaneously executed sub-processes: a video game streaming sub-process 402 executed in combination by the game server 102 and client device 104; a client reporting sub-process 404 executed by the client device 104; and a frame rate configuration sub-process 406 executed by the game server 102.

[0010] Referring initially to the video game streaming sub-process 402, at block 410, the game server 102 executes an instance 108 ( FIG. 1 ) of a video game application 324 (hereinafter, “video game instance 108”) on behalf of the client device 104. In at least one embodiment, the execution of the video game instance 108 is responsive to user game inputs 110 ( FIG. 1 ) received at one or more I / O devices 202 of the client device 104, such as a keyboard, mouse, gamepad, game controller, or touchscreen, and transmitted to the game server 102 by the client streaming application 218. These user game inputs 110 are provided as inputs to the running video game instance 108, which adjusts gameplay in response to the user game inputs 110. According to the commanded frame rate (described below), at block 412, the video game instance 108 commands the rendering of a stream 114 of video frames 116 ( FIG. 1 ), each of which represents the visual or graphical content of a current aspect of the running instance of the video game instance 108 at a corresponding time, such as the current state of game play from a corresponding viewpoint, a menu screen, etc. At block 414, the encoder 322 of the server streaming application 318 encodes each rendered video frame 112 to generate an encoded video frame 116, which the server streaming application 318 then transmits to the client device 104 over one or more networks 106 as part of a transmitted data stream 118. A stream of audio data (not shown) representing corresponding audio content is also generated by the game server 102 and then encoded for inclusion as an encoded audio stream as part of the transmitted data stream 118. This subprocess then repeats for the generation, encoding, and transmission of the next video frame 116 at the commanded frame rate.

[0011] For each transmitted encoded video frame 116, in block 416, the client streaming application 218 at the client device 104 receives data representing the encoded video frame 116 in the data stream 118, decodes the encoded video frame 116 to generate a decoded video frame 120 (FIG. 1), and provides the decoded video frame 120 to a display 210 associated with the client device 104 for display. Similarly, the corresponding encoded audio content is decoded and then provided to one or more speakers, with the audio being output in synchronization with the display of the corresponding rendered video content.

[0012] Ideally, the capabilities of the game server 102 (as hardware) to encode video frames 112, the capabilities of the game server 102 to render video frames 112, the capabilities of the network(s) 106 to transmit the encoded video frames 116, and the capabilities of the client device 104 to decode and present the decoded video frames 120 for display are matched or at least compatible. However, in practical embodiments, a mismatch exists between these capabilities, and the mismatch between the capabilities for frame rendering and frame encoding at the game server 102 often has the greatest impact in the form of dropped frames for encoding and / or significant and variable latency in the encoding and networking transmission process. Thus, in at least one embodiment, the client device 104 and the game server 102 together perform a remote synchronization process in which an appropriate frame rendering rate is determined at the game server 102 based on input from the client device 104, which serves to better balance rendering performance and latency. This remote synchronization process is represented by subprocesses 404 and 406 of method 400 .

[0013] Starting with sub-process 404, at block 420, the frame rate control module 220 of the client streaming application 218 determines one or more current parameters of the client device 104 that may affect the rate at which the client device 104 can receive, decode, and display video frames from the transmitted data stream 118. Such parameters may include, for example, the frame rate display range of the display 210, the decoding capabilities of the client device 104 (e.g., hardware and other resources available for decoding), the current power state or current thermal state of the client device 104, the fullness of one or more buffers used to buffer frame data for decoding or display, and current parameters for one or more networks 106, such as the current bandwidth and / or latency between the game server 102 and the client device 104 provided by the one or more networks 106. At block 422, the frame rate control module 220 uses the current parameters determined at block 420 to determine a proposed maximum frame rate 130 ( FIG. 1 ) that can reasonably be supported by the client device 104 given the identified current parameters. The client streaming application 218 may rely on any of a variety of parameters in determining the proposed maximum frame rate 130. For example, the frame rate control module 220 may consider one or more of the frame rate display range of the display 210 and the decoding capabilities of the client device 104 (e.g., hardware and other resources available for decoding), as well as current parameters for one or more networks 106, such as the current bandwidth and / or latency between the game server 102 and the client device 104 provided by the one or more networks 106.For example, the frame rate control module 220 may monitor the input buffer used to buffer incoming frame data for decoding and may determine the proposed maximum frame rate 130 at least in part from the current buffer fullness (or from the current rate of change in buffer fullness). As another example, the current power state (indicating currently available decoding hardware resources) and / or the current temperature state may be considered. The frame rate control module 220 may determine the proposed maximum frame rate 130 from these considered parameters using any of a variety of techniques. For example, the frame rate control module 220 may use a weighted sum equation for these parameters, an algorithm that determines the frame rate from these parameters, a stored look-up table (LUT) that uses some or all of these parameters as inputs, a trained deep neural network (DNN) or other neural network, etc. At block 424, the client streaming application 218 transmits the determined proposed maximum frame rate 130 to the game server 102 over one or more networks 106. Sub-process 404 may then be repeated for another iteration to update the proposed maximum frame rate 130 based on the updated current parameters of the client device 104, and the next iteration of sub-process 404 may be performed on a periodic basis (e.g., in response to a repetition timer), in response to an aperiodic trigger (e.g., a parameter being considered changes by exceeding a fullness threshold amount), etc.

[0014] Sub-process 406 begins with the transmission by client device 104 of a proposed maximum frame rate 130. Accordingly, at block 426, frame rate control module 320 of server streaming application 318 receives the proposed maximum frame rate 130 and, in response, at block 428, determines one or more current parameters of game server 102 that may affect the ability of game server 102 to render, encode, or transmit video frames from the running video game instance 108. Accordingly, these parameters may include, for example, network performance parameters (e.g., bandwidth or latency) as observed by game server 102, hardware capabilities of game server 102, frame rate limits or other client-oriented policies implemented by the operator of game server 102, server occupancy / utilization (e.g., the number of instances of video game application 324 or other video game applications running for other client devices), etc. At block 430, the frame rate control module 320 uses the proposed maximum frame rate 130 and the determined server-side current parameters to determine a target frame rate 132 ( FIG. 1 ) that is consistent with the server's current capabilities (e.g., hardware capabilities) as represented by the proposed maximum frame rate 130 and the determined current server parameters. Similar to the client-side determination of the proposed maximum frame rate 130, the frame rate control module 320 can determine the target frame rate 132 in any of a variety of ways or combinations of ways, such as using a weighting equation, an algorithm, one or more LUTs, a DNN or other neural network, or a combination thereof. In other embodiments, the frame rate control module 320 can select the target frame rate 132 from a set of options.For example, the frame rate control module 320 may have three options: a proposed maximum frame rate 130 from the client device 104, a maximum allowed frame rate defined by server configuration or policy, and a measured current frame rate based on the server's current performance (e.g., from game rendering time, encoder time, time slices for the server's virtual machine (VM), etc.). The frame rate control module 320 may then select a target frame rate 132 from these three options based on which option is best suited to providing a sustainable frame rate.

[0015] The game server 102 uses the determined target frame rate 132 to synchronize operations of the game server 102 related to rendering and capturing video frames 116 to the target frame rate 132. Thus, in at least one embodiment, in block 432, the target frame rate 132 is provided to the KMD 316-2, which in turn generates a frame rate synchronization (RSync) “signal” 136 representing the target frame rate 132. While the RSync signal 136 is shown as a periodic square wave in FIG. 1 for purposes of illustration, in at least one embodiment it is not a “signal” per se, but rather a representation of synchronization between the KMD 316-2 and the UMD 316-1 in selectively blocking (or delaying) the rendering of the next video frame through control of synchronization objects between the KMD 316-2 and the UMD 316-1. For example, at the start of a video game instance 108, UMD 316-1 and KMD 316-2 may perform a handshake procedure in which UMD 316-1 signals that it can support the RSync process described herein, and to facilitate this capability, UMD 316-1 registers a synchronization object with KMD 316-2 for use in implementing RSync signals 136 during the execution of the video game instance 108. Thereafter, in response to determining a target frame rate 132, KMD 316-2 signals the synchronization object at a frequency corresponding to the target frame rate 132. KMD 316-2 events that signal or activate the synchronization object are referred to herein as "kernel events" and may serve as the basis for frame rate control, as described below.For example, KMD316-2 may activate a synchronization object (i.e., trigger a "kernel event") at a frequency equal to the frame frequency represented by target frame rate 132 (e.g., every 16.667 milliseconds (ms) for a target frame rate of 60 FPS), and this sequence of kernel events involving the synchronization object between KMD316-2 and UMD316-1 serves as an RSync signal 136 for the frame rate control process described herein.

[0016] In this exemplary embodiment, RSync signal 136 is used to control rendering operations for the running video game instance 108 at block 434. For example, in one embodiment, KMD 316-2 issues a kernel event to UMD 316-1, and in response to the start of a new frame period as represented in the kernel event, UMD 316-1 controls the running video game instance 108 to render the corresponding video frame 116 for that frame period. As described in more detail below with reference to FIG. 5 , the process by which UMD 316-1 controls the rendering behavior of video game instance 108 may include UMD 316-1 blocking the return of a current call for video game instance 108 until the kernel event is unblocked by KMD 316-2 based on RSync signal 136, which allows the current call to proceed toward video game instance 108, resulting in rendered video frame 112 (block 414 of sub-process 402). The server streaming application 318 may then encode the resulting video frame 116 to generate a corresponding encoded video frame 116, as described above with reference to block 416 of sub-process 402.

[0017] The client device 104 can then send an updated proposed maximum frame rate 130 to reflect the maximum frame rate supportable by the client device 104 under changed conditions (e.g., a change in allocated GPU bandwidth or a change in network latency), and another iteration of subprocess 406 is performed to determine an updated target frame rate 132, which then updates the periodicity / frequency of the RSync signal 136 accordingly. For example, in at least one embodiment, the frame rate control module 220 of the client streaming application 218 utilizes the input queue 224 of the decoder 222 to adjust the target frame rate 132. A filling input queue 224 may indicate that the game server 102 is rendering video frames faster than the client device 104's current rate. In some embodiments, the decoder frequency (i.e., the decoder job rate) is not adjusted; therefore, the decoder 222 typically injects decoded frames 120 into the display pipeline of the display 210 for presentation, relying on the display pipe to backpressurize the decoder, which eventually manifests as the input queue 224 filling up. Thus, in this approach, the frame rate control module 220 monitors the fullness of the input queue 224, and if the fullness exceeds a specified high threshold (which may be fixed or may vary to reflect changes in other conditions), the frame rate control module 220 can send an updated proposed maximum frame rate 130 reflecting a lower maximum frame rate to prompt the game server 102 to reduce the target frame rate 132 accordingly.Conversely, in some embodiments, if the fullness of the input queue 224 falls below a lower threshold, indicating that the decoder 222 is decoding frames for presentation faster than the game server 102 can render them, the frame rate control module 220 may send an updated proposed maximum frame rate representing an increased maximum frame rate to prompt the game server 102 to increase the target frame rate 132 accordingly. Similarly, the frame rate control module 320 of the server streaming application 318 may modify the target frame rate 132 independently of client feedback, such as in response to changes in hardware resources allocated to the video game instance 108, in response to changes in network bandwidth observed on the server side, etc.

[0018] In this approach, the client device 104 proposes a maximum frame rate that can be supported by the client device 104 under current conditions (proposed maximum frame rate 130), and the game server 102 uses this proposed maximum frame rate as an upper limit to set its own rendering frame rate that can be supported by the game server 102 under its current conditions, and then synchronizes the frame rendering process via RSync signal 136. The result is the rendering of video frames 116 at a rate that should be encodeable by the encoder 322 of the server streaming application 318, and the resulting encoded video frames 116 are transmitted at a sustainable rate given the associated current conditions of the game server 102. This alignment of server rendering and client decoding and display rates thus facilitates avoiding missed frames due to encoding and improving latency through better synchronization between user game input and the presentation of the resulting video frame(s) at the client device 104.

[0019] In embodiments utilizing an application programming interface (API) to control frame rendering and / or capture, such as with the Microsoft DirectX API, the video game instance 108 renders a video frame, makes a current call to the API, and then proceeds to render the next video frame. Thus, holding or blocking the current call delays the start of the next video frame, and thus selectively holding the current call provides a mechanism for frame rate control. FIG. 5 illustrates an embodiment of the frame rate control process of block 434 using RSync signals 136, which utilizes holding or blocking the issuance and return of the current call to synchronize the rendering and capture of video frames. As described above, initiation of the video game instance 108 may include the UMD 316-1 registering a synchronization object with the KMD 316-2 for frame rate control purposes (as represented by initialization block 502). The KMD 316-2 then subsequently signals (i.e., triggers a kernel event) this synchronization object at a frequency representing the current target frame rate 132. Thus, in block 504, KMD 316-2 determines whether the current frame period represented by RSync signal 136 has expired. If not, in block 506, KMD 316-2 blocks the next kernel event and remains blocked until RSync signal 136 indicates that the current frame period has expired, at which point KMD 316-2 unblocks the next kernel event in block 508, thus allowing the kernel event to be signaled to UMD 316-1 through the registered synchronization object.

[0020] In a parallel process, in block 510, the video game instance 108 issues a current call to the UMD 316-1 to trigger the presentation of a rendered video frame 120. However, the UMD 316-1 does not respond in the form of a return from the current call to immediately initiate frame presentation, but instead determines the current status of a kernel event in the KMD 316-2 in block 512. If a kernel event is not blocked in the KMD 316-2, the UMD 316-1 blocks on the current call (i.e., blocks the return of the current call to the video game instance) in block 514 and maintains the block on the return of the current call until the kernel event is no longer blocked in the KMD 316-2, at which point the UMD 316-1 unblocks the return of the current call in block 516 and then triggers the video game instance 108 to begin rendering the next video frame 116 for the video game instance 108 in block 518.

[0021] One aspect of remote synchronization between game server 102 and client device 104 is the implementation of a vertical synchronization (vsync) feature. In traditional local rendering and display systems, such as game consoles that render video frames for local display, vsync typically utilizes double or triple buffering and page flipping on frame pixel data to ensure that one video frame is fully displayed before the next video frame is displayed, thus avoiding screen tearing. Thus, to provide synchronization between the display intent of video game instance 108 and the resulting local presentation of rendered video content, in at least one embodiment, the current vsync status used by video game instance 108 is communicated from game server 102 to client device 104, for example, as metadata that is part of data stream 118 and received by client streaming application 218, which then configures the display pipeline to activate or deactivate vsync in the display pipeline of client device 104 to match the vsync status of video game instance 108. To this end, the UMD 316-1 can communicate the vsync status of the video game instance 108 to the KMD 316-2 (as well as the OS 314), which then provides a representation of the vsync status to the server streaming application 318 for inclusion as metadata in the data stream 118. Furthermore, in particular embodiments, the vsync function results in the issuance of a flip call to instruct the KMD 316-2 to flip between paged buffers. However, because the remote render-capture-transmit process of the game server 102 does not utilize buffer flipping, the KMD 316-2 can emulate a page flip to satisfy the flip call by reporting flip completion upon receiving the flip call.

[0022] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s). The executable instructions stored on the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0023] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, that some of the particular activities or devices may not be required, that one or more additional activities may be performed, and that one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and variations can be made without departing from the scope of the invention as set forth in the claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention.

[0024] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and features from which any benefit, advantage, or solution may arise or be manifested are not construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, since the disclosed invention may be modified and practiced in different, but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and that all such variations are considered within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

Claims

1. 1. A method executed on a server, comprising: determining a target frame rate based on a first proposed maximum frame rate received from a client device and based on one or more current parameters of the server; rendering video frames of the stream of video frames based on the target frame rate; encoding video frames of the stream to generate a stream of encoded video frames; transmitting the stream of encoded video frames to the client device over at least one network. method.

2. modifying the target frame rate based on a second proposed maximum frame rate received from the client device after receiving the first proposed maximum frame rate; rendering video frames of the stream of video frames based on the modified target frame rate.

10. The method of claim 1.

3. modifying the target frame rate based on one or more updated current parameters of the server; and rendering the stream of video frames based on the modified target frame rate.

10. The method of claim 1.

4. the one or more current parameters of the server include at least one of a network latency between the server and the client device, a network bandwidth between the server and the client device, a capability of the server's hardware resources allocated to the client device, or a frame rate policy set by an operator of the server; 10. The method of claim 1.

5. the target frame rate is determined from the first proposed maximum frame rate and the one or more current parameters using at least one of a weighted sum equation, an algorithm, a lookup table, or a trained neural network.

10. The method of claim 1.

6. the stream of video frames is generated for an instance of a video game application executing on the server on behalf of the client device; 10. The method of claim 1.

7. determining a status of a vertical synchronization (vsync) feature of the instance of the video game application; transmitting a representation of the status of the vsync functionality to the client device over the at least one network. The method of claim 6.

8. Rendering video frames of the stream of video frames based on the target frame rate includes: blocking kernel events in a kernel mode driver until a start of a next frame period based on the target frame rate; receiving a current call in a user mode driver and blocking a return of the current call in the user mode driver until the kernel event is unblocked; the server begins rendering video frames of the stream upon unblocking the return of the current call.

10. The method of claim 1.

9. a server, a network interface coupleable to at least one network; at least one processor coupled to the network interface; at least one memory coupled to the at least one processor; the at least one memory storing executable instructions; The executable instructions include: determining a target frame rate based on a first proposed maximum frame rate received from a client device via the at least one network and based on one or more current parameters of the server; rendering video frames of the stream of video frames based on the target frame rate; encoding video frames of the stream to generate a stream of encoded video frames; providing the stream of encoded video frames to the network interface for transmission over at least one network to the client device; causing the at least one processor to perform server.

10. The executable instructions include: modifying the target frame rate based on a second proposed maximum frame rate received from the client device after receiving the first proposed maximum frame rate; rendering video frames of the stream of video frames based on the modified target frame rate; causing the at least one processor to perform The server of claim 9.

11. The executable instructions include: modifying the target frame rate based on one or more updated current parameters of the server; Rendering the stream of video frames based on the modified target frame rate; and causing the at least one processor to perform The server of claim 9.

12. the one or more current parameters of the server include at least one of a network latency between the server and the client device, a network bandwidth between the server and the client device, a capability of the server's hardware resources allocated to the client device, or a frame rate policy set by an operator of the server; The server of claim 9.

13. the target frame rate is determined from the first proposed maximum frame rate and the one or more current parameters using at least one of a weighted sum equation, an algorithm, a lookup table, or a trained neural network. The server of claim 9.

14. the at least one processor generates the stream of video frames for an instance of a video game application executing on the server on behalf of the client device. The server of claim 9.

15. Rendering video frames of the stream of video frames based on the target frame rate includes: blocking kernel events in a kernel mode driver until a start of a next frame period based on the target frame rate; receiving a current call in a user mode driver and blocking a return of the current call in the user mode driver until the kernel event is unblocked; the server begins rendering video frames of the stream upon unblocking the return of the current call. The server of claim 9.

16. 1. A method executed on a client device, comprising: determining a first proposed maximum frame rate based on one or more current parameters of the client device; transmitting the first proposed maximum frame rate to a server over at least one network; receiving, from the server over the at least one network, a first stream of encoded video frames rendered at a first target frame rate based on the first proposed maximum frame rate; decoding the first stream of encoded video frames to generate a first stream of decoded video frames; presenting a first stream of decoded video frames for display on a display associated with the client device. method.

17. determining a second proposed maximum frame rate based on one or more updated current parameters of the client device; transmitting the second proposed maximum frame rate to the server over at least one network; receiving, from the server via the at least one network, a second stream of encoded video frames rendered at a second target frame rate based on the second proposed maximum frame rate; decoding the second stream of encoded video frames to generate a second stream of decoded video frames; presenting a second stream of decoded video frames for display on the display.

17. The method of claim 16.

18. the one or more updated current parameters of the client device include an indication that an input queue of a decoder of the client device for decoding an encoded video frame has exceeded a fullness threshold; the second proposed maximum frame rate is less than the first proposed maximum frame rate; the second target frame rate is less than the first target frame rate; 18. The method of claim 17.

19. the one or more current parameters of the client device include at least one of a maximum display frame rate of the display, a network latency between the client device and the server, a network bandwidth between the client device and the server, or a hardware capability of the client device for decoding and displaying video frames; 17. The method of claim 16.

20. the first stream of encoded video frames is generated for an instance of a video game application executing on the server on behalf of the client device; 17. The method of claim 16.

21. a client device, a network interface coupleable to at least one network; at least one processor coupled to the network interface; at least one memory coupled to the at least one processor; the at least one memory storing executable instructions; The executable instructions include: determining a first proposed maximum frame rate based on one or more current parameters of the client device; transmitting the first proposed maximum frame rate to a server over at least one network; receiving, from the server over the at least one network, a first stream of encoded video frames rendered at a first target frame rate based on the first proposed maximum frame rate; decoding the first stream of encoded video frames to generate a first stream of decoded video frames; presenting a first stream of decoded video frames for display on a display associated with the client device; causing the at least one processor to perform Client device.

22. The executable instructions include: determining a second proposed maximum frame rate based on one or more updated current parameters of the client device; transmitting the second proposed maximum frame rate to the server over at least one network; receiving, from the server via the at least one network, a second stream of encoded video frames rendered at a second target frame rate based on the second proposed maximum frame rate; decoding the second stream of encoded video frames to generate a second stream of decoded video frames; presenting a second stream of decoded video frames for display on the display; and causing the at least one processor to perform 22. The client device of claim 21.

23. the one or more current parameters of the client device include at least one of a maximum display frame rate of the display, a network latency between the client device and the server, a network bandwidth between the client device and the server, or a hardware capability of the client device for decoding and displaying video frames; 23. The client device of claim 22.

24. the first stream of encoded video frames is generated for an instance of a video game application executing on the server on behalf of the client device; 22. The client device of claim 21.

Images