Remote display synchronization for preserving local display

Abstract

A remote display synchronization technique preserves the presence of a local display device for a remotely rendered video stream. A server cooperates with a client device to dynamically determine a target frame rate for a rendered frame stream suitable for current capacities of the server and the client device and networking conditions. And the server generates a synchronization signal according to the target frame rate, and the synchronization signal is used for timing control of a rendering process. The client device may provide feedback to cause a change in the target frame rate, and thus a corresponding change in the synchronization signal. In the method, the rendering frame rate and encoding frequency may be "synchronized" in a manner consistent with the capabilities of the server, network and client device such that a stream of frames is generated, encoded, transmitted, decoded and rendered, which mitigates encoding loss of frames while providing acceptable latency.

Description

Background Art

[0001] A remote or "cloud-based" video game system employs a remote server to receive user game input from a client device via a network, execute an instance of a video game based on the user game input, and deliver a rendered video frame stream and a corresponding audio stream to the client device via the network to be presented at a display and a speaker of the client device, respectively. Typically, for video content, the server employs a rendering and capture process in which the server renders a video frame stream from a video game application, and an encoder encodes the pixel content of the video frame stream into an encoded video frame stream, which is sent concurrently to the client device. Typically, the rendering process is separate from the capture process. The server implements a streaming pipeline process in which the server executes the video game and renders the frame, and executes another application in parallel to capture the rendered frame and encode the rendered frame. The separation of frame rendering and capturing from encoding results in an encoding frequency that is inconsistent with the rendering frame rate, which typically results in the encoder "losing" frames (i.e., not encoding the frame to be included in the transmitted encoded stream) and results in a relatively high and inconsistent delay on the frame stream. Therefore, if the rendering frame rate is maintained at a frame rate lower than the encoding frequency, this can reduce latency and avoid dropped frames, but may result in a lower display frame rate at the client device than would otherwise be available. Conversely, if the encoding frequency is less than the rendering frame rate, the encoder will "drop" rendered frames as it tries to catch up with the rendering process, which can result in the aforementioned high and inconsistent latency, while also wasting power generating rendered frames that are dropped and therefore ultimately not presented at the client device. In addition, the separation of server frame rendering from encoding rates and client decoding from display rates can also result in longer delays and / or dropped frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The present disclosure may be better understood by referring to the accompanying drawings, and its numerous features and advantages will be apparent to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical items.

[0003] Figure 1 is a block diagram of a server-based video game system according to some specific implementations.

[0004] Figure 2 It is an example according to some specific implementations Figure 1 A block diagram of the hardware configuration of a client device of a video gaming system.

[0005] Figure 3 It is an example according to some specific implementations Figure 1 A block diagram of the hardware configuration of a game server of a video game system.

[0006] Figure 4 is a flow chart illustrating a method for remote frame rate synchronization between a server and a client device of a server-based video game system according to some implementations.

[0007] Figure 5 is a flow chart illustrating a method for rendering rate control based on remote frame rate synchronization with selective existing call blocking according to some specific implementations. DETAILED DESCRIPTION

[0008] Pixel streaming methods in remote video game systems and other systems that render graphical content remotely for display at a local device rely on a frame rendering process and a frame capture (encoding) process (i.e., a "streaming process") that may have different capabilities, resulting in relatively high or variable latency (when rendering capabilities exceed encoding capabilities) or artificially reduced frame rates at the local device (when rendering capabilities remain below encoding capabilities). To mitigate this difference in capabilities, Figures 1 to 5 A system and method for remote display synchronization technology is illustrated, which maintains the presence and characteristics of a local display. In at least one specific implementation, a server collaborates with a client device to determine a target frame rate suitable for both the current capacity of the server and the current capacity of the client device and the network connecting the two. The server then generates a series of frame rate synchronization signals (referred to herein as "remote synchronization" or "RSync") based on the target frame rate, which are used as timing control of the existing rhythm of the rendering process and the encoding process. In addition, on a periodic basis or in response to one or more triggers, the client device can provide update feedback to trigger a change in the target frame rate, and thus cause a corresponding change in the synchronization signal. In this method, the rendering frame rate and the encoding frequency can be "synchronized" in a manner consistent with the capabilities of the server, the network, and the client device, so that a rendering frame stream is generated, encoded, sent, decoded, and presented, which mitigates the encoding loss of the rendering frame while retaining an acceptable delay. In addition, the server and the client device can collaborate to convey various display characteristics achieved by executing a video game to be implemented at the client device. For example, a video game application may be executed with vertical synchronization (vsync) enabled or disabled, and the vsync state may be communicated from a server to a client device to implement the indicated vsync state at the client device.

[0009] It should be noted that for ease of illustration, the systems and techniques of the present disclosure are described in the exemplary context of a server-based video game system, in which an instance of a video game application is remotely executed and its rendered video and audio output is encoded and streamed to a client device, whereby the client device decodes the video and audio content and presents the resulting decoded content to the user. However, these systems and techniques are not limited to the server-based gaming context, but can be used in any of a variety of scenarios in which real-time video content is remotely rendered and the resulting pixel content is then remotely encoded for delivery as an encoded video stream to one or more client devices. Therefore, unless otherwise specified, references to "video games" also apply to such equivalent scenarios.

[0010] Figure 1 A remote video game system 100 for providing server-based video games according to some implementations is illustrated. The system 100 includes a video game server 102 (hereinafter referred to as "game server 102") remotely connected to at least one client device 104 via one or more networks 106. The client device 104 may include, for example, a smartphone, a computing-enabled vehicle entertainment system, a computing-enabled appliance, a tablet computer, a laptop computer, a desktop computer, a video game console, a television, etc. The one or more networks 106 may include one or more wireless networks (such as a personal area network (PAN), a cellular network, or a wireless local area network (WLAN)), one or more wired networks (such as a local area network (LAN) or a wide area network (WAN)), the Internet, or a combination thereof.

[0011] Brief turn Figure 2 and Figure 3 , respectively illustrate exemplary hardware configurations of the client device 104 and the game server 102. Figure 2As shown in FIG. 1 , the illustrated hardware configuration 200 of the client device 104 includes one or more I / O devices 202 (the one or more I / O devices include a network interface 202-1 for interfacing with one or more networks 106), one or more central processing units (CPUs) 204, one or more graphics processing units (GPUs) 206, one or more memories 208, and a display 210 integrated with or otherwise connected to the client device 104. Other well-known hardware components typically implemented at a client device, such as speakers, microphones, power supplies, buses, power managers, etc., are omitted for clarity. The one or more memories 208 include one or more types of memory (such as random access memory (RAM), read-only memory (ROM), flash memory, hard drive, register files, etc.) and store one or more sets of executable instructions that, when executed by the one or more CPUs 204 and / or one or more GPUs 206, manipulate the hardware of the client device 104 to perform the functions attributed to the client device 104 herein. Specifically, the executable instructions may implement an operating system (OS) 214 for overall control and coordination of the hardware components of the client device 104, a set of graphics (GFX) drivers 216 (such as user mode drivers (UMDs) and kernel mode drivers (KMDs)) for coordinating and controlling one or more GPUs 206 by one or more CPUs 204, and a client streaming application 218. The client streaming application 218 further includes a frame rate control module 220 for coordination of the rendering frame rate of the game server 102 and a decoder 222, as described below, the frame rate control module for coordination of the rendering frame rate of the game server 102, and the decoder for decoding the encoded video frames received from the game server 102 and buffering them in, for example, an input queue 224.

[0012] like Figure 3As shown in , the illustrated hardware configuration 300 of the game server 102 includes one or more I / O devices 302 (the one or more I / O devices include a network interface 302-1 for interfacing with one or more networks 106), one or more CPUs 304, one or more GPUs 306, and one or more memories 308 (e.g., RAM, ROM, flash memory, hard drive, or a combination thereof). Other well-known hardware components that are typically implemented at a game server, such as power supplies, buses, power managers, etc., are omitted for clarity. The one or more memories 308 store one or more sets of executable instructions that, when executed by one or more CPUs 304 and / or one or more GPUs 306, manipulate the hardware of the game server 102 to perform functions attributed to the game server 102. Specifically, the executable instructions may implement an OS 314 (and a hypervisor for multi-tenant specific implementations) for overall control and coordination of the hardware components of the game server 102, a set of graphics (GFX) drivers 316 (such as KMD 316-2 and UMD 316-1) for coordinating and controlling one or more GPUs 306 by one or more CPUs 304, and a server streaming application 318. The server streaming application 318 in turn includes a frame rate control module 320 for coordinating with the frame rate control module 220 of the client device 104 for setting the rendering frame rate of the game server 102, and an encoder 322, as described below, the frame rate control module for coordinating with the frame rate control module 220 of the client device 104 for setting the rendering frame rate of the game server 102, and the encoder for encoding the video frames rendered by the executing instance of the video game application 324.

[0013] refer to Figure 4 According to at least one specific embodiment, a method for Figures 1 to 3 The method 400 of remote display synchronization of the video game system 100. Although the method 400 is in Figure 1 The video game system 100 and the Figure 2 and Figure 3 The method 400 is described in the exemplary context of the corresponding hardware configuration of the client device 104 and the game server 102, but the method 400 can be implemented in other scenarios in which the video content is remotely rendered and the corresponding pixel content is encoded and sent in real time using the guidelines provided herein. The illustrated method 400 includes three sub-processes that are executed simultaneously: a video game streaming sub-process 402 executed by the game server 102 in combination with the client device 104, a client reporting sub-process 404 executed by the client device 104, and a frame rate configuration sub-process 406 executed by the game server 102.

[0014] Turning first to the video game streaming subprocess 402, at block 410, the game server 102 executes an instance 108 of the video game application 324 on behalf of the client device 104 ( Figure 1 ) (hereinafter referred to as "video game instance 108"). In at least one specific implementation, execution of video game instance 108 is responsive to user game input 110 (received at one or more I / O devices 202 of client device 104, such as a keyboard, mouse, game pad, game controller, or touch screen) received. Figure 1 ) and is sent by the client streaming application 218 to the game server 102. These user game inputs 110 are provided as input to the executing video game instance 108, which adjusts the game play in response to the user game inputs 110. Based on the indicated frame rate (described below), at block 412, the video game instance 108 directs the video frame 116 ( Figure 1 ), each of the video frames representing visual or graphical content of a current aspect of an execution instance of a video game instance 108 at a corresponding point in time, such as a current state of a game from a corresponding perspective, a menu screen, etc. At block 414, the encoder 322 of the server streaming application 318 encodes each rendered video frame 112 to generate an encoded video frame 116, and the server streaming application 318 then transmits the encoded video frame 116 to the client device 104 via the one or more networks 106 as part of the transmitted data stream 118. An audio data stream (not shown) representing the corresponding audio content is also generated by the game server 102 and is then encoded to be included as an encoded audio stream as part of the transmitted data stream 118. The sub-process then repeats with the generation, encoding, and transmission of the next video frame 116 at the indicated frame rate.

[0015] For each transmitted encoded video frame 116, at block 416, the client streaming application 218 at the client device 104 receives data representing the encoded video frame 116 in the data stream 118 and decodes the encoded video frame 116 to generate a decoded video frame 120 ( Figure 1 ) and provides the decoded video frames 120 to a display 210 associated with the client device 104 for display. Similarly, the corresponding encoded audio content is decoded and then provided to one or more speakers for audio output synchronized with the display of the corresponding rendered video content.

[0016] Ideally, the game server 102 (as hardware) ability to encode video frames 112, the game server 102 ability to render video frames 112, the network 106 ability to send encoded video frames 116, and the client device 104 ability to decode and present decoded video frames 120 for display match or are at least compatible. However, in actual implementations, there is a mismatch between these capabilities, and the mismatch between the frame rendering capability at the game server 102 and the frame encoding capability usually has the greatest impact in the form of encoding lost frames and / or significant and variable delays in the encoding and network transmission process. Therefore, in at least one implementation, the client device 104 and the game server 102 implement a remote synchronization process together, in which the appropriate frame rendering rate is determined at the game server 102 based on input from the client device 104, and the process is used to more appropriately balance rendering performance and delay. The remote synchronization process is represented by sub-processes 404 and 406 of method 400.

[0017] Beginning with subprocess 404, at block 420, a frame rate control module 220 of a client streaming application 218 determines one or more current parameters of a client device 104 that may affect the rate at which the client device 104 may receive, decode, and display video frames from the transmitted data stream 118. Such parameters may include, for example, a frame rate display range of the display 210, decoding capabilities of the client device 104 (e.g., hardware and other resources available for decoding), a current power state or a current temperature state of the client device 104, fullness of one or more buffers used to buffer frame data for decoding or display, and current parameters of one or more networks 106, such as current bandwidth and / or latency between the game server 102 and the client device 104 provided by the one or more networks 106. At block 422, the frame rate control module 220 uses the current parameters determined at block 420 to determine a proposed maximum frame rate 130 ( Figure 1), the proposed maximum frame rate that can be reasonably supported by the client device 104 given the identified current parameters. The client streaming application 218 may rely on any of a variety of parameters to determine the proposed maximum frame rate 130. For example, the frame rate control module 220 may consider one or more of the frame rate display range of the display 210 and the decoding capabilities of the client device 104 (e.g., the hardware and other resources available for decoding) and the current parameters of the one or more networks 106 (such as the current bandwidth and / or latency between the game server 102 and the client device 104 provided by the one or more networks 106). For example, the frame rate control module 220 may monitor an input buffer used to buffer incoming frame data for decoding, and may determine the proposed maximum frame rate 130 based at least in part on the current buffer fullness (or based on the current rate of change of the buffer fullness). As another example, the current power state and / or the current temperature state (indicative of the currently available decoding hardware resources) may be considered. The frame rate control module 220 may determine the proposed maximum frame rate 130 based on these considered parameters using any of a variety of techniques. For example, the frame rate control module 220 may employ a weighted sum equation of these parameters, an algorithm that determines the frame rate based on these parameters, a populated lookup table (LUT) that uses some or all of these parameters as input, a trained deep neural network (DNN) or other neural network, etc. At block 424, the client streaming application 218 sends the determined proposed maximum frame rate 130 to the game server 102 via the one or more networks 106. The sub-process 404 may then be repeated for another iteration to update the proposed maximum frame rate 130 based on the updated current parameters of the client device 104, where the next iteration of the sub-process 404 may be performed periodically (e.g., in response to a repeating timer), in response to a non-periodic trigger (e.g., a parameter under consideration changes by more than a fullness threshold amount), etc.

[0018] Subprocess 406 is initiated by client device 104 transmitting a proposed maximum frame rate 130. Thus, at block 426, frame rate control module 320 of server streaming application 318 receives the proposed maximum frame rate 130 and, in response, determines, at block 428, one or more current parameters of game server 102 that may affect the ability of game server 102 to render, encode, or send video frames from the executing video game instance 108. As such, these parameters may include, for example, network performance parameters observed by game server 102 (e.g., bandwidth or latency), hardware capabilities of game server 102, frame rate limits set by an operator of game server 102 or other client-directed policies, server occupancy / utilization (e.g., the number of instances of video game application 324 or other video game applications executing for other client devices), etc. At block 430, frame rate control module 320 uses the proposed maximum frame rate 130 and the determined server-side current parameters to determine a target frame rate 132 ( Figure 1 ), the target frame rate being consistent with the proposed maximum frame rate 130 and the current capacity (e.g., hardware capacity) of the server as represented by the determined current server parameters. As with the client determination of the proposed maximum frame rate 130, the frame rate control module 320 may determine the target frame rate 132 in any of a variety of ways or a combination of ways, such as using a weighted equation, an algorithm, one or more LUTs, a DNN, or other neural network, or a combination thereof. In other specific implementations, the frame rate control module 320 may select the target frame rate 132 from a set of options. For example, the frame rate control module 320 may have three options: the proposed maximum frame rate 130 from the client device 104; the maximum allowed frame rate defined by the server configuration or policy; and the measured current frame rate based on the current performance of the server (e.g., from game rendering time, encoder time, time slices of the server's virtual machine (VM), etc.). The frame rate control module 320 may then select the target frame rate 132 from the three options based on which option is best suited to provide a sustainable frame rate.

[0019] The game server 102 uses the determined target frame rate 132 to synchronize the game server 102 operations related to rendering and capturing of the video frames 116 to the target frame rate 132. Thus, in at least one specific implementation, the target frame rate 132 is provided to the KMD 316-2 at block 432, which in turn generates a frame rate synchronization (RSync) “signal” 136 that is representative of the target frame rate 132. Although for purposes of illustration, the target frame rate 132 is not provided to the KMD 316-2. Figure 11 is illustrated as a periodic square wave, but in at least one specific implementation, the RSync signal 136 itself is not a "signal", but rather a representation of synchronization between KMD 316-2 and UMD 316-1 when selectively blocking (or delaying) the rendering of the next video frame by controlling a synchronization object between KMD 316-2 and UMD 316-1. For illustration, upon initiating a video game instance 108, UMD 316-1 and KMD 316-2 may perform a handshake process in which UMD 316-1 signals that it is capable of supporting the RSync process described herein, and to facilitate such capability, UMD 316-1 registers a synchronization object with KMD 316-2 for implementing the RSync signal 136 during execution of the video game instance 108. Thereafter and in response to the determination of the target frame rate 132, KMD 316-2 signals the synchronization object at a frequency corresponding to the target frame rate 132. The events that KMD 316-2 signals or asserts synchronization objects are referred to herein as "kernel events" and may be used as the basis for frame rate control as described below. For example, KMD 316-2 may assert synchronization objects (i.e., trigger "kernel events") at a frequency equal to the frame frequency represented by target frame rate 132 (e.g., every 16.667 milliseconds (ms) for a target frame rate of 60 FPS), and the sequence of kernel events involving synchronization objects between KMD 316-2 and UMD 316-1 serves as RSync signal 136 for the frame rate control process described herein.

[0020] In this exemplary implementation, the RSync signal 136 is used at block 434 to control the rendering operation of the executing video game instance 108. For purposes of illustration, in one implementation, the KMD 316-2 issues a kernel event to the UMD 316-1, and in response to the start of a new frame period indicated in the kernel event, the UMD 316-1 controls the executing video game instance 108 to render the corresponding video frame 116 for the frame period. Figure 5 Described in more detail, the process by which the UMD 316-1 controls the rendering behavior of the video game instance 108 can include the UMD 316-1 blocking the return of an existing call of the video game instance 108 until a kernel event is unblocked by the KMD 316-2 based on the RSync signal 136, which unblocks the existing call to allow the video game instance 108 to proceed, thereby obtaining the rendered video frame 112 (block 414 of the subprocess 402). The server streaming application 318 can then encode the resulting video frame 116 to generate a corresponding encoded video frame 116, as described above with reference to block 416 of the subprocess 402.

[0021] Thereafter, the client device 104 may send an updated proposed maximum frame rate 130 to reflect the maximum frame rate that the client device 104 can support under changed circumstances (e.g., a change in allocated GPU bandwidth or a change in network latency), in response to which another iteration of the subprocess 406 is performed to determine an updated target frame rate 132, which in turn updates the periodicity / frequency of the RSync signal 136 accordingly. For purposes of illustration, in at least one implementation, the frame rate control module 220 of the client streaming application 218 utilizes the input queue 224 of the decoder 222 to adjust the target frame rate 132. A full input queue 224 may indicate that the game server 102 is rendering video frames faster than the existing rate of the client device 104. In some implementations, the decoder frequency (i.e., decoder operating rate) is not adjusted, and therefore the decoder 222 typically injects decoded frames 120 into the display pipeline of the display 210 for presentation and relies on the display pipeline to backpressure the decoder, which ultimately manifests as the input queue 224 becoming full. Thus, in this approach, the frame rate control module 220 monitors the fullness of the input queue 224, and when the fullness exceeds a specified high threshold (which may be fixed or variable to reflect changes in other conditions), the frame rate control module 220 may send an updated proposed maximum frame rate 130 reflecting a lower maximum frame rate, in order to prompt the game server 102 to correspondingly lower the target frame rate 132. Conversely, in some implementations, if the fullness of the input queue 224 is below a lower threshold, indicating that the decoder 222 is decoding frames for presentation faster than the game server 102 is rendering frames, the frame rate control module 220 may send an updated proposed maximum frame rate representing an increased maximum frame rate, in order to prompt the game server 102 to increase the target frame rate 132 in response. Similarly, the frame rate control module 320 of the server streaming application 318 may modify the target frame rate 132 independently of client feedback, such as in response to changes in hardware resources allocated to the video game instance 108, in response to changes in network bandwidth observed at the server side, and the like.

[0022] In this method, the client device 104 proposes a maximum frame rate that the client device 104 can support under the current environment (proposed maximum frame rate 130), and the game server 102 uses the proposed maximum frame rate as an upper limit, by which the game server 102 sets its own rendering frame rate that the game server 102 can support under the current environment, and then synchronizes the frame rendering process via the RSync signal 136. The result is that the video frames 116 are rendered at a rate that the encoder 322 of the server streaming application 318 should be able to encode, and the resulting encoded video frames 116 are sent at a supportable rate given the associated current environment of the game server 102. Therefore, this server rendering and client decoding and display rate matching helps avoid missing frames for encoding, as well as improve latency through better synchronization between user game input and the presentation of the resulting video frames at the client device 104.

[0023] In a specific implementation that utilizes an application programming interface (API) to control frame rendering and / or capture, such as the Microsoft DirectX API, the video game instance 108 renders a video frame, makes an existing call to the API, and then continues rendering of the next video frame. Thus, holding or blocking the existing call delays the start of the next video frame, and thus selectively holding the existing call provides a frame rate control mechanism. Figure 5 A specific implementation of the frame rate control process of block 434 illustrating the use of the RSync signal 136, and which utilizes the issuance of an existing call and the holding or blocking of a return from the existing call to synchronize the rendering and capture of video frames. As described above, for the purpose of frame rate control, the initiation of the video game instance 108 may include the UMD 316-1 registering a synchronization object with the KMD 316-2 (as represented by the initialization block 502). Thereafter, the KMD 316-2 then signals the synchronization object (i.e., triggers a kernel event) at a frequency representing the current target frame rate 132. Therefore, at block 504, the KMD 316-2 determines whether the current frame period represented by the RSync signal 136 has expired. If not, then at box 506, KMD 316-2 blocks the next kernel event and remains blocked until the RSync signal 136 indicates that the current frame period has expired, at which time, at box 508, KMD 316-2 unblocks the next kernel event, thereby allowing the kernel event to be signaled to UMD 316-1 through the registered synchronization object.

[0024] In a parallel process, at block 510, the video game instance 108 issues an existing call to the UMD 316-1 to trigger presentation of the rendered video frame 120. However, rather than immediately initiating frame display in response in the form of a return from the existing call, the UMD 316-1 determines the current state of kernel events at the KMD 316-2 at block 512. If the kernel event is not blocked at the KMD 316-2, then at block 514, the UMD 316-1 blocks the existing call (i.e., blocks the existing call from returning to the video game instance), and remains blocked when the existing call returns until the kernel event is no longer blocked at the KMD 316-2, at which point the UMD 316-1 unblocks the return of the current call at block 516, which in turn triggers the video game instance 108 to begin rendering the next video frame 116 of the video game instance 108 at block 518.

[0025] One aspect of remote synchronization between the game server 102 and the client device 104 is the specific implementation of the vertical synchronization (vsync) feature. In traditional local rendering and display systems (such as game consoles that render video frames for local display), vsync typically uses double or triple buffering and page flipping of frame pixel data to ensure that one video frame is displayed in its entirety before the next video frame is displayed, and thus screen tearing is avoided. Therefore, in order to provide synchronization between the display intent of the video game instance 108 and the resulting local presentation of the rendered video content, in at least one specific implementation, the current vsync state used by the video game instance 108 is communicated from the game server 102 to the client device 104 as metadata, for example, as part of the data stream 118, and is received by the client streaming application 218, which in turn configures the display pipeline to enable vsync or disable vsync of the display pipeline of the client device 104, which is consistent with the vsync state of the video game instance 108. To this end, UMD 316-1 can communicate the vsync state of the video game instance 108 to KMD 316-2 (and OS 314), which in turn provides a representation of the vsync state to the server streaming application 318 for inclusion as metadata in the data stream 118. In addition, in some implementations, the vsync feature causes a flip call to be issued to instruct KMD 316-2 to flip between page buffers. However, since the remote rendering-capture-send process of game server 102 does not utilize buffer flipping, KMD 316-2 can simulate a page flip in order to satisfy the flip call by reporting the flip completion in response to receiving the flip call.

[0026] In some embodiments, certain aspects of the above-mentioned technology can be implemented by one or more processors of a processing system that executes software. Software includes one or more sets of executable instructions that are stored or otherwise tangibly embodied on a non-transient computer-readable storage medium. Software may include instructions and certain data that manipulate one or more processors to perform one or more aspects of the above-mentioned technology when executed by one or more processors. Non-transient computer-readable storage media may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM) or other one or more non-volatile memory devices, etc. The executable instructions stored on a non-transient computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0027] It should be noted that not all activities or elements described above in the general description are required, a portion of a particular activity or device may not be required, and one or more additional activities may be performed, or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. In addition, these concepts have been described with reference to specific embodiments. However, it is understood by those of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0028] Benefits, other advantages and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any features that may cause any benefit, advantage or solution to appear or become more significant should not be interpreted as key, essential or basic features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in different but equivalent ways that are obvious to those skilled in the art who benefit from the teachings herein. Except as described in the following claims, it is not intended to limit the details of the construction or design shown herein. Therefore, it is apparent that the specific embodiments disclosed above may be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the following claims.

Claims

1. A method executed at a server, the method include: determining a target frame rate based on a first proposed maximum frame rate received from the client device and further based on one or more current parameters of the server; rendering video frames of a video frame stream based on the target frame rate; encoding video frames of the stream to generate a stream of encoded video frames; as well as The encoded video frame stream is sent to the client device via at least one network.

2. The method according to claim 1, further comprising: include: modifying the target frame rate based on a second proposed maximum frame rate received from the client device after receiving the first proposed maximum frame rate; as well as Video frames of the video frame stream are rendered based on the modified target frame rate.

3. The method according to claim 1, further comprising: include: modifying the target frame rate based on one or more updated current parameters of the server; as well as The video frame stream is rendered based on the modified target frame rate.

4. The method of any preceding claim, wherein the one or more current parameters of the server include at least one of: a network delay between the server and the client device; a network bandwidth between the server and the client device; a capacity of hardware resources of the server allocated to the client device; Or a policy regarding frame rate set by the operator of the server.

5. A method according to any preceding claim, wherein the target frame rate is determined based on the first proposed maximum frame rate and the one or more current parameters using at least one of the following: a weighted sum equation; an algorithm; a lookup table or a trained neural network.

6. A method according to any preceding claim, wherein the video frame stream is generated for an instance of a video game application executing at the server on behalf of the client device.

7. The method according to claim 6, further comprising: include: determining a state of a vertical synchronization (vsync) feature of the instance of the video game application; as well as A representation of the status of the vsync feature is sent to the client device via the at least one network.

8. A method according to any preceding claim, wherein video frames of the video frame stream are rendered based on the target frame rate include: blocking kernel events at a kernel mode driver based on the target frame rate until a next frame period begins; as well as An existing call is received at a user mode driver, and a return of the existing call is blocked at the user mode driver until the kernel event is unblocked, wherein when the return of the existing call is unblocked, the server initiates rendering of a video frame of the stream.

9. A server, wherein the server include: a network interface capable of coupling to at least one network; at least one processor coupled to the network interface; and At least one memory, the at least one memory being coupled to the at least one processor, the at least one memory storing executable instructions to operate the at least one processor to perform the following operations: determining a target frame rate based on a first proposed maximum frame rate received from a client device via the at least one network and based on one or more current parameters of the server; rendering video frames of a video frame stream based on the target frame rate; encoding video frames of the stream to generate a stream of encoded video frames; as well as The encoded video frame stream is provided to the network interface for transmission to the client device via at least one network.

10. The server of claim 9, wherein the executable instructions are further configured to operate the at least one processor to: modifying the target frame rate based on a second proposed maximum frame rate received from the client device after receiving the first proposed maximum frame rate; and Video frames of the video frame stream are rendered based on the modified target frame rate.

11. The server of claim 9, wherein the executable instructions are further configured to operate the at least one processor to: modifying the target frame rate based on one or more updated current parameters of the server; and The video frame stream is rendered based on the modified target frame rate.

12. A server according to any one of claims 9 to 11, wherein the one or more current parameters of the server include at least one of the following: network latency between the server and the client device; network bandwidth between the server and the client device; capacity of hardware resources of the server allocated to the client device; or a policy regarding frame rate set by an operator of the server.

13. A server according to any one of claims 9 to 12, wherein the target frame rate is determined based on the first proposed maximum frame rate and the one or more current parameters using at least one of the following: a weighted sum equation; an algorithm; a lookup table or a trained neural network.

14. The server of any one of claims 9 to 13, wherein the at least one processor generates the video frame stream for an instance of a video game application executing at the server on behalf of the client device.

15. The server according to any one of claims 9 to 14, wherein the executable instructions configured to manipulate the at least one processor to render video frames of the video frame stream based on the target frame rate include executable instructions configured to manipulate the at least one processor to: blocking kernel events at the kernel mode driver until a next frame period begins based on the target frame rate; and An existing call is received at a user mode driver, and a return of the existing call is blocked at the user mode driver until the kernel event is unblocked, wherein when the return of the existing call is unblocked, the server initiates rendering of a video frame of the stream.

16. A method performed at a client device, the method include: determining a first proposed maximum frame rate based on one or more current parameters of the client device; sending the first proposed maximum frame rate to a server via at least one network; receiving, from the server via the at least one network, a first stream of encoded video frames that have been rendered at a first target frame rate based on the first proposed maximum frame rate; Decoding the first encoded video frame stream to generate a first decoded video frame stream; as well as The first decoded video frame stream is rendered for display at a display associated with the client device.

17. The method according to claim 16, further comprising: include: determining a second proposed maximum frame rate based on the one or more updated current parameters of the client device; sending the second proposed maximum frame rate to the server via at least one network; receiving, from the server via the at least one network, a second stream of encoded video frames that have been rendered at a second target frame rate based on the second proposed maximum frame rate; Decoding the second encoded video frame stream to generate a second decoded video frame stream; as well as The second decoded video frame stream is rendered for display at the display.

18. The method according to claim 17, in: The one or more updated current parameters of the client device include an indication that an input queue of a decoder of the client device for decoding encoded video frames has risen above a fullness threshold; the second proposed maximum frame rate is less than the first proposed maximum frame rate; and The second target frame rate is less than the first target frame rate.

19. A method according to any one of claims 16 to 18, wherein the one or more current parameters of the client device include at least one of the following: a maximum display frame rate of the display; a network delay between the client device and the server; a network bandwidth between the client device and the server; or a hardware capacity of the client device for decoding and displaying video frames.

20. The method of any one of claims 16 to 19, wherein the first encoded video frame stream is generated for an instance of a video game application executing at the server on behalf of the client device.

21. A client device, the client device include: a network interface capable of coupling to at least one network; at least one processor coupled to the network interface; and At least one memory, the at least one memory being coupled to the at least one processor, the at least one memory storing executable instructions to operate the at least one processor to perform the following operations: determining a first proposed maximum frame rate based on one or more current parameters of the client device; sending the first proposed maximum frame rate to a server via at least one network; receiving, from the server via the at least one network, a first stream of encoded video frames that have been rendered at a first target frame rate based on the first proposed maximum frame rate; Decoding the first encoded video frame stream to generate a first decoded video frame stream; as well as The first decoded video frame stream is rendered for display at a display associated with the client device.

22. The client device of claim 21, wherein the executable instructions are further for manipulating the at least one processor to: determining a second proposed maximum frame rate based on the one or more updated current parameters of the client device; sending the second proposed maximum frame rate to the server via at least one network; receiving, from the server via the at least one network, a second stream of encoded video frames that have been rendered at a second target frame rate based on the second proposed maximum frame rate; Decoding the second encoded video frame stream to generate a second decoded video frame stream; as well as The second decoded video frame stream is rendered for display at the display.

23. A client device according to claim 22, wherein the one or more current parameters of the client device include at least one of the following: a maximum display frame rate of the display; a network delay between the client device and the server; a network bandwidth between the client device and the server; or a hardware capacity of the client device for decoding and displaying video frames.

24. The client device of any one of claims 21 to 23, wherein the first encoded video frame stream is generated for an instance of a video game application executing at the server on behalf of the client device.

Images