SELECTIVE MULTIPLE-INSTANCE ENCODING OF VIDEO FRAMES TO PROVIDE TARGET FRAME RATES DURING NETWORK TRANSMISSION - Patent application

The selective video frame duplication process addresses inconsistent frame rates by duplicating frames as needed, maintaining a stable frame rate and reducing network transmission problems for improved video streaming.

JP7594580B2Active Publication Date: 2024-12-04GOOGLE LLC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2022521503
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2019-10-10
Publication Date
2024-12-04
Estimated Expiration
2039-10-10

AI Technical Summary

Technical Problem

Video streams with variable or inconsistent frame rates cause network transmission problems, leading to idle timespans and negatively impact viewer experience due to inconsistent processing loads.

Method used

Implement a selective video frame duplication process that duplicates encoded frames only when necessary to maintain a constant target frame rate, using a countdown timer and frame buffer to ensure consistent network transmission.

Benefits of technology

Maintains a consistent network processing load, reducing transmission issues and improving viewer experience by ensuring a stable frame rate in real-time video streaming applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594580000001
    Figure 0007594580000001
  • Figure 0007594580000002
    Figure 0007594580000002
  • Figure 0007594580000003
    Figure 0007594580000003
Patent Text Reader

Abstract

The system (100) includes a video source device (104) connected to a video destination device (106) via a network (108). The video source device includes a network interface (212) and an encoder (210) coupled to the network interface. The encoder is configured to encode a first stream of rendered video frames (128, 146, 166) having a first frame rate to generate a second stream of encoded video frames (132, 150, 170) for transmission over the network via the network interface, the second stream having a second frame rate greater than the first frame rate. As part of this encoding process, the encoder is configured to selectively encode multiple instances (154, 156, 158) of at least one video frame (152) of the first stream for inclusion in the second stream to compensate for a difference between the first and second frame rates.
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] background A video source device typically renders video frames (also known as "pictures") of a video stream at a nominal frame rate. This nominal frame rate may be fixed with little or no frame rate variation, or there may be some variation in the effective frame rate, for example, due to the processing load or capacity required to render any given video frame in the video stream. Transmission networks utilized to transmit such video streams in real time typically operate more effectively when the processing load of the transmission is consistent and at or above a threshold level. Video streams rendered at a frame rate that is variable or different from the nominal frame rate may lead to frames arriving faster or slower than the nominal frame rate, thus causing relatively small or large idle time spans between video frames in the stream. These idle time spans may subject such video streams to network transmission problems. Such problems may include deprioritizing channels established to transmit the video stream or delaying the recognition of problems in the network. Thus, a video stream that exhibits an inconsistent transmission processing load can adversely affect a viewer's experience of the video stream's playback at a destination device that receives the video stream over a network.

[0002] BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0003] [Figure 1]FIG. 1 is a block diagram illustrating an electronic system that employs selective video frame instance duplication to maintain a constant network transmission processing load for transmitting encoded video streams, according to some embodiments. [Diagram 2] 2 is a block diagram illustrating a server of the electronic system of FIG. 1 according to some embodiments. [Diagram 3] 1 is a flow diagram illustrating a method for selective video frame instance duplication for real-time transmission of encoded video streams according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Detailed Description Ensuring that a video stream presents a relatively constant processing load to the network transmitting the video stream typically results in more reliable or otherwise effective transmission of data representing the video stream. One conventional approach to providing a constant transmission processing load to a network is to buffer video data at a source device so that the video data can be transmitted at a constant rate. However, such an approach introduces latency in the delivery of video content, which is often impractical for remotely served video games and other real-time video streaming applications. For such real-time video streaming applications, one technique used to attempt to provide this constant processing load is to automatically duplicate all encoded frames in the video stream at the source device, thereby reducing the period of time during which the channel is idle for transmitting the video stream. For example, if a frame rate of 60 frames per second (FPS) is deemed appropriate for a particular network and the source device renders video frames at a nominal rate of 30 FPS, then the source device duplicates each video frame once to create a resulting video stream at 60 FPS. However, this approach is based on both the assumption that the frame rate of the video stream is known and the assumption that the frame rate is constant. If the rendering frame rate is unknown or the rendering process results in a variable frame rate, when the frame rate is greater than, or increases to greater than, 50% of the network's target frame rate, the automatic duplication of each and every video frame will generate an excessive number of video frames, which can overload the network and therefore adversely affect the viewing experience.Conversely, if the frame rate is or falls below 50% of the target frame rate, automatic single duplication of each and every video frame will produce video frames below the target frame rate and therefore potentially have the same detrimental effect on the transmitted video stream: insufficient or inconsistent processing load on the network.

[0005] In contrast, the present application describes systems and techniques for selective duplication of instances of encoded video frames within a video stream transmitted over a network to provide a constant and sufficiently frequent transmission processing load on the network for the transmission of the video stream. In at least one embodiment, the selective duplication process selectively duplicates encoded instances of video frames based on whether a current rate of video frame rendering meets a target frame rate for the transmission of the resulting encoded video stream. While the current rendered frame rate is equal to or greater than the target frame rate, the rendered frames are encoded and transmitted as part of the video stream without duplication. However, when the current rendered frame rate effectively falls below the target frame rate, the source device operates to insert one or more duplicate instances of one or more rendered video frames to increase the effective frame rate in the video stream up to the target frame rate.

[0006] In one embodiment, this selective replication process is triggered by a periodic event that is synchronized or otherwise represents a frame period of the target frame rate, such as an emulated vertical synchronization ("vsync") signal that occurs at the transition or boundary between one frame period and the next frame period in the target frame rate. A rendering component of the source device renders video frames at a native frame rate for the video stream, which may be fixed or variable and may be lower than, equal to, or higher than the target frame rate. Each rendered frame is stored in a frame buffer accessible to an encoder of the source device.

[0007] In response to assertion of the periodic event or trigger, the encoder attempts to encode the next video frame for inclusion as an encoded video frame in the video stream. If the next video frame is in the frame buffer and ready for encoding, the encoder proceeds to encode the next video frame and insert the resulting encoded video frame into the video stream for transmission over the network to the destination device. However, if the next video frame is not ready for encoding (e.g., the next video frame is only partially rendered at the time the periodic trigger is asserted), rather than delaying encoding it when it is ready and idling the channel in the network, the encoder instead duplicates the previous video frame by including a first duplicate encoded instance of the previous video frame, thus ensuring that the target frame rate is maintained in the video stream being transmitted. If the next video frame is not yet ready for encoding by the time the periodic trigger is next asserted, the encoder again duplicates the previous video frame by including a second duplicate encoded instance of the previous video frame, again maintaining the target frame rate in the encoded video stream. This process involving another encoded instance of the previous video frame can be repeated any number of times until the next rendered video frame is in the frame buffer and ready to be encoded.

[0008] The duplicate encoded instance of the previous video frame, in one embodiment, may include a second, third, fourth, or further encoding, or a re-encoding, of the previous video frame, and thus result in an encoding of the previous video frame of higher quality than an initial encoding of the previous video frame. In other embodiments, each encoded video frame is temporarily buffered, and rather than re-encoding the previous video frame, the same encoded instance of the previous video frame is instead provided as a copy of the buffered, already encoded instance of the previous video frame.

[0009] 1 illustrates an electronic system 100 employing a selective video frame duplication process for providing an encoded video stream at least at a target frame rate, according to at least one embodiment. The electronic system 100 includes a data center 102 having a video source device 104 connected to a video destination device 106 via a packet-switched network 108. The video source device 104 operates to execute a video source software application 110 that results in real-time rendering of a sequence of video frames that the video source device 104 encodes for inclusion in an encoded video stream 112 that is transmitted to the video destination device 106 over a channel 114 (also commonly referred to as a "pipe" or "session"). The video destination device 106 operates to receive the encoded video stream 112, decode the encoded video frames contained therein, and display the decoded video frames in sequence to provide a video presentation of the rendered graphical content to a user of the video destination device 106. For illustrative purposes, electronic system 100 is described herein in the exemplary context of a remotely served video gaming system in which video source device 104 is a gaming server (and thus referred to herein as “gaming server 104”), video destination device 106 is a user's gaming device (and thus referred to herein as “gaming device 106”), and video source software application 110 is a video gaming application (and thus referred to herein as “video gaming application 110”). However, the techniques described herein are applicable to any of a variety of video system implementations in which a real-time video stream is rendered and encoded at a source device, and the resulting encoded video stream is transmitted over a network from the source device to a destination device, such as in a system for streaming live or pre-recorded television content over the Internet.

[0010] Packet-switched network 108 (hereinafter "network 108") may include one or more wired or wireless personal area networks (PANs), one or more wired or wireless local area networks (LANs), one or more wired or wireless wide area networks (WANs), or combinations thereof. By way of example, network 108 may include a wired LAN (not shown) connecting gaming server 104 to the Internet, which is connected to gaming devices 104 via a wireless LAN (WLAN). Gaming server 104, described in more detail below with reference to FIG. 2, includes computational and storage resources for facilitating execution of video game application 110 to render a sequence of video frames according to user input provided from gaming device 106 via upstream subchannel 116 and encode the resulting video frames. Gaming server 104 further includes a network interface to network 108 and provides data representing the encoded video frames, along with any associated metadata and audio data generated by video game application 110, for transmission to gaming device 106 via channel 114.

[0011] The gaming device 106 includes a network interface (not shown) connected to the network 108 for receiving video data, metadata, and audio data transmitted over the network 108 from the gaming server 104, computation, storage, and display resources (not shown) for decoding and displaying video frames of the transmitted video stream 112 and outputting corresponding audio content, and a game controller 118, keyboard (not shown) or other user input / output (I / O) device for receiving user input for transmission over an upstream subchannel 116 to the gaming server 104 during game play, which user input is then provided to the executed video game application 110 and affects the graphical content included in the rendered video frames and the audio content included in the audio stream.

[0012] In at least one embodiment, execution of the video game application 110 results in the rendering of video frames at a frame rate referred to herein as the "rendering frame rate." By way of illustration, the video game application 110 may be designed or programmed to render video frames at a specified nominal frame rate, such as 30 FPS. Alternatively, computational resources allocated by the gaming server 104 may play a role in setting the rendering frame rate. In some implementations, the rendering frame rate is fixed and constant; i.e., video frames are always rendered at the same rate. In other implementations, the rendering frame rate may vary to an extent based on different rendering loads required by each video frame or changes in computational resource allocation over time. For example, complex scenes typically require more computational power to render than simpler scenes, and the complexity of a given video frame may tax the gaming server 104 such that that video frame is rendered "late"; i.e., the video frame is ready for encoding at a point later than the nominal "ready" point for each frame period, given the nominal frame rate. Thus, in such a case, the video stream 112 may have an effective frame rate of, for example, 30 FPS over a large averaging window, but the instantaneous frame rate may vary above or below 30 FPS depending on the rendering processing load.

[0013] The network 108 implements one or more policies or operational parameters, such as a quality of service (QoS) policy, that in one embodiment favors a constant transmission processing load over a bursty processing load for a given transmission channel, including the channel 114 established between the gaming server 104 and the gaming device 106. For example, these policies or parameters typically stipulate that when a channel is idle for at least a threshold duration, packets of that channel are subsequently de-prioritized (i.e., the channel is de-prioritized) by the network 108, which may then result in a lower QoS for packets of that channel. Furthermore, a bursty processing load may result in a transmission processing load that exceeds the capacity of the transmission channel, which in turn causes queuing or even packet loss. Either situation may cause delays and redundant transmissions (retransmissions), which are inherently inefficient. Thus, in at least one embodiment, a designer, implementer, administrator, or other entity in control of the gaming server 104 identifies a threshold frame rate, referred to herein as a “target frame rate,” for the video stream 112 that is expected or predicted to provide a sufficiently constant transmission processing load on the network 108 to mitigate or eliminate channel de-prioritization or other network degradation effects on the transmission of the video stream 112 over the network 108 that would otherwise occur with transmission of the video stream 112 at a lower frame rate. The target frame rate may be determined through experimentation or modeling of the network 108. For example, the gaming server 104 or other components of the data center 102 may transmit one or more test data streams having different transmission processing load properties over the network 108 to one or more destination devices and measure network statistics or otherwise observe the operation of the network 108. Alternatively, characteristics of the network 108 may be determined and an estimate of the minimum transmission processing load may be inferred from the determined characteristics.In yet other implementations, the target frame rate is fixed based on previous experience or other estimates.

[0014] In some instances, the rendering frame rate of the video game application 110 is sufficient to meet or exceed the target frame rate. For example, if the target frame rate of the network 108 is determined to be 30 FPS and the video game application 110 executed by the gaming server 104 is a non-fluctuating 60 FPS (i.e., video frames are not delayed or dropped), then the stationarity and frequency of the transmission processing load required for the video stream 112 is sufficient to avoid the detrimental effects that a more bursty transmission would incur from the network 108. However, if the target frame rate of the network 108 exceeds the effective rendering frame rate of the video game application 110 executed by the gaming server 104, or if dropped or delayed rendered video frames are likely to introduce relatively substantial idle transmission periods into the channel 114, then the video stream 112 may potentially suffer negative transmission effects due to this bursty transmission processing load. Accordingly, in at least one embodiment, the gaming server 104 employs a selective video frame duplication process that selectively incorporates multiple encoded instances of a particular rendered video frame into the video stream 112 to compensate for the difference between the lower rendering frame rate and the higher target frame rate, including compensating for any delayed or dropped video frames in the rendering process that may result in a lower instantaneous rendering frame rate.

[0015] For this process, the gaming server 104, in one embodiment, uses a countdown timer or other mechanism to represent successive frame periods according to the target frame rate. The countdown timer is set to a count equivalent to the duration of each frame period (hereinafter referred to as a "target frame period") corresponding to the target frame rate, and the expiration (and subsequent reset) of the countdown serves as a period trigger that acts as an emulated vertical sync signal or other boundary of the transition between one target frame period and the next. By way of example, for a target frame rate of 60 FPS, the target frame period is 16 milliseconds (ms), and therefore the countdown timer is set to expire and reset every 16 ms. When the countdown timer expires, the gaming server 104 determines whether the next rendered video frame to be encoded is actually ready to be encoded (e.g., fully rendered and stored in a frame buffer used as a source of video frames for the encoder). If so, gaming server 104 encodes the next video frame and provides the resulting encoded instance of the video frame to the network interface for transmission over network 108 as part of video stream 112. If instead, the next video frame is not ready to be encoded when the countdown timer expires (e.g., the next video frame has not been fully rendered in the frame buffer), then rather than waiting for the next video frame to be ready, gaming server 104 instead provides a second, or duplicate, encoded instance of the previous video frame to the network interface for transmission over network 108 as part of video stream 112. That is, gaming server 104 copies a previously encoded instance of the previous video frame or re-encodes the previous video frame to generate a second encoded instance of the previous video frame for inclusion in video stream 112.Under this approach, the gaming server 104 ensures that the video stream 112 contains encoded video frames at least at the target frame rate, even if two or more of the contained encoded video frames are the same or different instances of the same rendered video frame, and therefore ensures that the video stream 112 presents a sufficiently constant transmission processing load on the network 108.

[0016] Diagrams 120, 122, and 124 of FIG. 1 illustrate three examples of this selective video frame duplication process for corresponding portions of a rendered video stream and corresponding portions of an encoded stream. In the example of diagram 120, timeline 126 represents the rendered frame rate of gaming server 104 as it executes video game application 110 to render a stream of rendered video frames (a portion of which is represented by rendering stream 128). The span between each tick mark on timeline 126 represents a corresponding frame period (numbered frame periods J and J+1, from right to left) at the rendered frame rate. Each box in rendering stream 128 represents a rendered video frame for the corresponding frame period. Additionally, in diagram 120, the left edge of each box in rendering stream 128 represents a point in time when the rendering of the corresponding video frame is complete and therefore ready for encoding. Timeline 130 represents a target frame rate for video stream 112 to provide a sufficiently constant transmission processing load on network 108, with each tick mark on timeline 130 representing a corresponding frame period (numbered from right to left, frame periods K through K+2) at the target frame rate. Encoded stream 132 represents a portion of video stream 112, with each box in encoded stream 132 representing an encoded instance of a video frame from rendering stream 128 that is inserted or otherwise included in the sequence of encoded video frames transmitted over network 108 as video stream 112. Each box without hatched fill in rendering stream 128 represents a first, or initial, encoded instance of a corresponding rendered video frame from rendering stream 128, and each box with hatched fill represents an additional, or duplicate, encoded instance of a rendered video frame from rendering stream 128.In the example represented by diagram 120, the target frame rate is significantly higher (eg, about 170%) than the rendering frame rate; illustratively, the rendering frame rate may be, for example, 30 FPS, while the target frame rate is 50 FPS.

[0017] Each tick mark on timeline 130 represents the end of one frame period and the start of the next frame period at the target frame rate, and therefore also represents the expiration of a countdown timer used as a periodic trigger synchronized to the target frame rate. Thus, in response to each periodic trigger represented by a corresponding tick mark on timeline 130, gaming server 104 determines whether the next rendered video frame is available in rendering stream 128. To illustrate, at the tick mark signaling the start of frame period K, rendered video frame 134 is ready for encoding, and therefore gaming server 104 encodes rendered video frame 134 to generate a first encoded instance 136 of video frame 132 for inclusion in encoded stream 132 for frame period K. However, at the start of frame period K+1, there is no newly rendered video frame available for encoding. Thus, in response to a trigger event represented by the beginning of frame period K+1, gaming server 104 determines that the next video frame is not ready for encoding, and therefore instead generates a second encoded instance 138 of the already once encoded video frame 134 for inclusion in encoded stream 132 for frame period K+1. At the beginning of the next frame period K+2, rendered video frame 140 is ready for encoding, and therefore gaming server 104 encodes video frame 140 to generate a first encoded instance 142 for inclusion in encoded stream 132 for frame period K+2.

[0018] Referring to the example diagram 122, timeline 144 represents the rendered frame rate of gaming server 104 when executing video game application 110 to render a stream of rendered video frames, a portion of which is represented by rendering stream 146. As with the previous example, the span between each tick mark on timeline 144 represents a corresponding frame period (numbered frame periods M and M+1, from right to left) at the rendered frame rate. Each box in rendering stream 146 represents a rendered video frame for the corresponding frame period, with the left edge of each box representing the point in time when the rendering of the corresponding video frame is complete and therefore ready for encoding. Timeline 148 represents a target frame rate for video stream 112 to provide a sufficiently constant transmission processing load on network 108, with each tick mark on timeline 148 representing a corresponding frame period (numbered frame periods N through N+3, from right to left) at the target frame rate. Encoded stream 150 represents a portion of video stream 112, with each box in encoded stream 150 representing an encoded instance of a video frame from rendered stream 146 that is inserted or otherwise included in the sequence of encoded video frames transmitted over network 108 as video stream 112. In the example represented by diagram 122, the target frame rate is significantly higher (e.g., about 2X) than the rendered frame rate; illustratively, the target frame rate may be, for example, 60 FPS, while the rendered frame rate is only 30 FPS.

[0019] In response to each periodic trigger, represented by a corresponding tick mark on timeline 148, gaming server 104 determines whether the next rendered video frame is available in rendered stream 146. Illustratively, at the tick mark signaling the start of frame period N, rendered video frame 152 is ready for encoding, and therefore gaming server 104 encodes rendered video frame 152 to generate a first encoded instance 154 of video frame 152 for inclusion in encoded stream 150 for frame period N. However, at the start of frame period N+1, no newly rendered video frame is available for encoding, and therefore, in response to the trigger event represented by the start of frame period N+1, gaming server 104 determines that the next video frame is not ready for encoding, and instead generates a second encoded instance 156 of video frame 152 already encoded once, for inclusion in encoded stream 150 for frame period N+1. At the start of the next frame period, N+2, a rendered video frame is not yet available for encoding, so gaming server 104 generates a third encoded instance 158 of video frame 152, already encoded twice, for inclusion in encoded stream 150 for frame period N+2. At the start of the next frame period, N+3, a rendered video frame 160 is ready for encoding, so gaming server 104 encodes video frame 160 to generate a first encoded instance 162 for inclusion in encoded stream 150 for frame period N+3.

[0020] In the example represented by diagram 124, timeline 164 represents the rendered frame rate of gaming server 104 when executing video game application 110 to render a stream of rendered video frames, a portion of which is represented by rendering stream 166. As in the previous example, the span between each tick mark on timeline 164 represents a corresponding frame period (numbered from right to left, frame periods X through X+2) at the rendered frame rate. Each box in rendering stream 166 represents a rendered video frame for the corresponding frame period, with the left edge of each box representing the point in time when the rendering of the corresponding video frame is complete and therefore ready for encoding. Timeline 168 represents a target frame rate for video stream 112 to provide a sufficiently constant transmission processing load on network 108, with each tick mark on timeline 168 representing a corresponding frame period (numbered from right to left, frame periods Y through Y+3) at the target frame rate. Encoded stream 150 represents a portion of video stream 112, with each box in encoded stream 150 representing an encoded instance of a video frame from rendering stream 146 that is inserted or otherwise included in the sequence of encoded video frames transmitted over network 108 as video stream 112. In the example represented by diagram 124, the target frame rate is slightly higher (e.g., about 33% higher) than the rendering frame rate; illustratively, the rendering frame rate may be, for example, 60 FPS, while the target frame rate is 80 FPS.

[0021] In response to each periodic trigger represented by a corresponding tick mark on the timeline 168, the gaming server 104 determines whether the next rendered video frame is available in the rendering stream 166. At the start of frame period Y, the rendered video frame 172 is ready for encoding, and thus the gaming server 104 encodes the rendered video frame 172 to generate a first encoded instance 174 of the video frame 172 for inclusion in the encoded stream 170 for frame period Y. In the illustrated example, the rendering of the video frame for frame period X+1 is skipped or delayed until frame period X+2. Thus, in response to a trigger event represented by the start of frame period Y+1, the gaming server 104 generates a second encoded instance 176 of the already once encoded video frame 172 for inclusion in the encoded stream 170 for frame period Y+1. At the start of the next frame period Y+2, a rendered video frame is not yet available for encoding, so gaming server 104 generates a third encoded instance 178 of video frame 172, already encoded twice, for inclusion in encoded stream 170 for frame period Y+2. At the start of the next frame period Y+3, rendered video frame 180 is ready for encoding, so gaming server 104 encodes video frame 180 to generate a first encoded instance 182 for inclusion in encoded stream 170 for frame period Y+3.

[0022] As diagrams 120, 122, and 124 show, the selective inclusion of one or more additional encoded instances of the same video frame results in an encoded video stream 112 having a frame rate at least equal to the target frame rate, and thus provides a transmission processing load for the channel 114 carrying video stream 112 that is expected to be sufficient to maintain effective transmission of the underlying data over network 108. In contrast, encoding the rendered video frames at their rendered frame rate, without any video frame duplication, to compensate for the difference between the rendered frame rate and the target frame rate, may lead to inconsistent or otherwise bursty transmission processing loads, and thus would expose video stream 112 to potential transmission problems. To illustrate, an idle period between the transmission of one encoded video frame and the next at 50 FPS, or even more likely, a relatively long idle period resulting from a lack of transmitted data for a skipped / dropped video frame (such as during frame period X+1 in the example of diagram 124), may cause the network 108 to not prioritize or otherwise assign a lower QoS level to the channel 114, and thus the encoded video data and other data transmitted over the channel 114 potentially suffers additional latency and jitter, increased packet loss, etc. Transmission of encoded video data at a lower rendered frame rate typically results in larger gaps between transmitted data packets compared to transmission of encoded video data with duplicated video frames at a higher target frame rate, and these larger gaps result in delays in identifying problems in the network 108, such as packets being queued or packet loss, and resulting duplicate retransmissions.

[0023] Furthermore, by selectively duplicating the encoded instances of certain video frames, rather than automatically duplicating each and every video frame, excessive transmission processing loads are more likely to be avoided. For illustrative purposes, in the example described above, the effective frame rate of the video stream 112 was increased from 50 FPS as rendered to 60 FPS as transmitted. In contrast, an approach that automatically performs one copy of each and every frame would increase the effective frame rate of the resulting video stream from 50 FPS as rendered to 100 FPS as transmitted, which, rather than improving transmission quality, may instead overload channels in the network 108 and lead to even more ineffective transmission of the video stream.

[0024] 2 illustrates an example implementation of a gaming server 104 as a video source device employing the selective video frame duplication process described herein, according to some embodiments. In the illustrated example, the gaming server 104 includes one or more central processing units (CPUs) 202, one or more graphics processing units (GPUs) 204, at least one system memory 206, a frame buffer 208 (which may be implemented as part of the system memory 206 or a separate memory), an encoder 210, and a network interface 212 that is coupled or couplable to the network 108. One or both of the CPU 202 or GPU 204 executes encoding software 214 stored in the system memory 206, for example, and the encoder 210 may be implemented as a hardware component separate from the CPU 202 and GPU 204, such as an application specific integrated circuit (ASIC) or other hard-coded logic, a field programmable gate array (FPGA) or other programmable logic, or as a combination of hard-coded or programmable logic and a processor running software.

[0025] In one embodiment, the encoder 210 includes or has access to a countdown timer 216 and a configuration storage device 218, which may include, for example, a programmable register, a one-time programmable (OTP) memory, a set of one or more fuses or antifuses, basic input / output (BIOS) settings, etc. The configuration storage device 218 operates to store a representation of a determined target frame rate 220 for the network 108. This representation may include, for example, a number of cycles of a clock (not shown) used by the countdown timer 216 equal to the duration of a target frame period for the target frame rate 220. When the countdown timer 216 is reset, it is configured to reset to an initial countdown value based on the representation of the target frame rate 220 stored in the configuration storage device 218.

[0026] The network interface 212 includes any of a variety of network interfaces suitable for coupling to the network 108, such as a LAN network interface, a WLAN network interface, etc. In implementations in which a video frame is included as more than one encoded instance in the video stream 112 by copying a previously encoded instance of the video frame, the gaming server 104 may further include a buffer 222 operable to store copies of the encoded video frames for subsequent access. The buffer 222 may be part of the frame buffer 208 or may be part of a separate memory.

[0027] Figure 3 illustrates an example method 300 for selective video frame duplication in an encoded video stream to achieve a target frame rate expected given a sufficiently constant transmission processing load, according to some embodiments. Method 300 is described in the example context of electronic system 100 of Figure 1, with particular reference to gaming server 104 of Figure 2. As shown, method 300 includes a rendering sub-process 302 and an encoding / transmission sub-process 304 operating in parallel.

[0028] For the rendering sub-process 302, the CPU 202 executes the video game application 110 stored in the system memory 206 to determine a game scene, determine a sequence of views of the game scene, and then in a corresponding iteration of block 306, directs the GPU 204 to render video frames in the frame buffer 208 (e.g., the rendered video frames 224 of FIG. 2) for the corresponding views of the sequence of views of the game scene. As explained above, the CPU 202 and the GPU 204 together control the rendering of the video frames at an effective rendering frame rate that may be substantially fixed or may vary depending on the complexity of the rendering, the current rendering resources available, etc. Further, in the game system 100 of FIG. 1, a user of the gaming device 106 "plays" the game by providing user input via a game controller 118, keyboard, or other I / O device, the user input indicating a desired movement of an avatar in the game scene, selection of a displayed option, etc. Such inputs are transmitted periodically or asynchronously from the gaming device 106 over the network 108 (e.g., over the upstream subchannel 114), and a set of user inputs for a given time slice is received at the gaming server 104 via the network interface 212 in an iteration of block 308 of the rendering sub-process 302. The CPU 202 receives this slice of user inputs, interprets the user inputs, and controls the rendering process for one or subsequent video frames based on the user inputs.

[0029] With respect to the encoding sub-process 304, the encoder 210 is initialized before beginning the encoding process to generate the encoded video stream 112. This initialization process includes determining a target frame rate 220 for the encoded video stream 112. As represented by block 310, the target frame rate 220 may be automatically set to a default value, as represented in the OTP memory or fuses / antifuses implementing the configuration storage device 218. For example, as represented by block 312, an administrative entity in charge of the data center 102 may analyze or model several different network scenarios and determine a default target frame rate that is expected to be sufficient for most or all of the network scenarios studied, and configure each gaming server in the data center 102 to utilize this default target frame rate. Alternatively, as represented by block 314, the gaming server 104 may allow a user of the gaming device 106 to select or otherwise specify the target frame rate 220, for example, via a graphical user interface (GUI) or other user input mechanism. To illustrate, based on the user's expectations or previous experience, a user may select a target frame rate 220 that is expected to provide a balance between efficient network transmission and display quality suitable for the particular user.

[0030] As represented by block 316, the gaming server 104 can set the target frame rate 220 based on a real-time analysis of the network 108. In this implementation, the gaming server 104 can analyze current network statistics, such as whether there are any other channels competing for bandwidth with the channel 114, the current end-to-end packet latency and jitter, the current available bandwidth, the projected future bandwidth / latency / jitter, the various QoS levels currently available, etc. From this information, the gaming server 104, or other systems in the data center 102 or another remote system, determines a target frame rate 220 that is expected to fit the current network conditions. The gaming server 104 can then periodically perform this analysis again to determine an updated target frame rate 220, i.e., the target frame rate 220 can be dynamically updated over the course of generating the encoded video stream 112.

[0031] Further, as indicated by block 318, in some embodiments, the gaming server 104 queries the gaming device 106 to determine a specified or preferred frame rate supported by the display of the gaming device 106. By way of illustration, the gaming device 106 may be configured to display video from the video stream 112 at a fixed display rate, and thus, to reduce the processing load on the gaming device 106, the gaming server 104 may attempt to provide video frames in the encoded video stream 112 at or near the fixed display rate, such that reliance on frame rate conversion processes at the gaming device 106 may be reduced or eliminated. In such cases, the gaming server 104 may apply a minimum threshold to the queried frame rate such that the resulting target frame rate is expected to provide a sufficiently constant transmission processing load on the network 108. For example, if it is determined that a target frame rate below 40 FPS would impose a transmission processing load that is inconsistent enough to cause network transmission degradation, the gaming server 104 sets the target frame rate 220 to the greater of the requested frame rate and 40 FPS.

[0032] Once the target frame rate 220 has been determined and programmed into the configuration storage 218, the encoding process is commenced to generate and transmit the encoded video stream 112. As explained above, in at least one embodiment, the timing of the encoding process is controlled at least in part with an emulated vsync signal provided by the periodic expiration of a countdown timer 216, with each countdown duration representing the duration of a target frame period according to the target frame rate 220. Thus, at each vsync cycle, the countdown timer 216 is reset to a count value representing the duration of the target frame period, and at block 320 of each vsync cycle, the encoder 210 monitors the expiration of the countdown timer 216, thus signalling the passage of the previous target frame period and the start of the next target frame period.

[0033] In response to signaling the expiration of countdown timer 216, and therefore the start of a new frame period at target frame rate 220, in block 322, encoder 210 checks frame buffer 208 to determine whether a previously unencoded rendered video frame (i.e., the “next” video frame) is ready and available in frame buffer 208 for encoding. If the next video frame is ready for encoding (e.g., fully rendered and all data to be utilized for encoding is present in frame buffer 208 or otherwise currently available to encoder 210), then in block 324, encoder 210 encodes the next video frame (e.g., rendered video frame 224) to generate a corresponding encoded instance of the video frame 226 and provides the encoded instance of the video frame 226 to network interface 212 for transmission as part of encoded video stream 112 over channel 114 in network 108 to gaming device 106. Encoder 210 employs any of a variety of standardized or proprietary video encoding techniques, such as, for example, a Motion Pictures Experts Group (MPEG) H.264- or H.265-compliant encoding process, a VP8- or VP9-compliant encoding process, etc. The resulting encoded data may be incorporated into any of a variety of transmission containers, such as a HyperText Markup Language 5 (HTML5) video container (representing the encoded video stream 112), which is then packetized by network interface 212, and the resulting packets are transmitted over network 108 via channel 114 to gaming device 106.

[0034] In implementations in which duplicate copies of previously encoded instances of video frames are inserted to achieve the target frame rate 220, copies of encoded instances 226 of video frames may also be temporarily buffered in a buffer 222 so as to be available for use as duplicate encoded instances when the next video frame is not ready on time, as described below.

[0035] Returning to block 322, if the next video frame is not ready to be encoded when the countdown timer 216 expires for the current cycle, then in block 326 the encoder 210 does not wait for the next video frame to be ready, but instead provides for insertion of a duplicate encoded instance of the most recently encoded video frame into the video stream 112 for transmission. If the rendered video frame was ready for encoding in the previous target frame period, then this duplicate encoded instance is a second encoded instance of that video frame. However, if the rendered video frame was not ready for encoding in the previous target frame period, then the encoder 210 relies on a rendered video frame from a further previous target frame period as the source video frame, and thus this duplicate encoded instance is at least a third encoded instance of the same previous rendered video frame. Thus, the process represented by blocks 322 and 326 may be repeated any number of times to maintain the target frame rate 220 until the rendered video frame is ready for encoding.

[0036] The encoder 210 provides a duplicate coded instance of a previously coded video frame in any of a variety of ways. As indicated by block 328, in one embodiment, as described above, a copy of an initial coded instance 226 of a video frame is temporarily stored in a buffer 222, and the encoder 210 provides a duplicate coded instance of the video frame by accessing the buffer 222 and retransmitting the copy of the copy of the initial coded instance 226 over the network interface 212 as a coded video frame for a corresponding target frame period. Thus, in this manner, the video stream 112 can have a sequence of two or more exact copies of the same coded video frame to compensate for the difference between a higher target frame rate 220 and a lower effective rendering frame rate.

[0037] In other embodiments, as represented by block 330, the encoder 210 again re-encodes the previously rendered video frame 224 to generate a re-encoded instance of the rendered video frame for inclusion in the encoded video stream 112 for the current target frame period. In at least one embodiment, the encoder 210 utilizes the initial encoding of the video frame for this re-encoding process to generate a re-encoded instance of the video frame that may have a higher encoding quality than the initial encoded instance of the video frame. Illustratively, by using the initial encoded instance as a reference frame, the encoder 210 can utilize the bit budget allocated to the re-encoded instance to provide higher quality motion vector information and higher quality residual data (which in turn provides a higher fidelity decoded video frame at the gaming device 106) as compared to the motion vector information and residual data that may be obtained for the initial encoding of the video frame for the same bit budget because the initial encoding cannot rely on a previous encoding of the same video frame as the reference frame.

[0038] According to one aspect, a method includes rendering, at a first device, a first stream of unencoded video frames at a first frame rate, the method further includes encoding, at the first device, video frames of the first stream to generate a second stream of encoded video frames, the second stream having a second frame rate greater than the first frame rate, encoding the video frames of the first stream includes selectively providing a plurality of encoded instances of at least one video frame of the first stream for inclusion in the second stream to compensate for a difference between the first frame rate and the second frame rate, the method further includes providing the second stream for transmission over a network to a second device.

[0039] Selectively providing a plurality of encoded instances of the at least one video frame may include encoding a first video frame of the first stream to generate a first encoded instance of the first video frame for inclusion in the second stream for a first frame period; determining whether a second video frame of the first stream is ready to be encoded for a second frame period following the first frame period, the first frame period and the second frame period being based on the second frame rate; in response to determining that the second video frame is ready to be encoded, encoding the second video frame to generate a encoded instance of the second video frame for inclusion in the second stream for the second frame period; and in response to determining that the second video frame is not ready to be encoded, re-encoding the first video frame or duplicating the first encoded instance of the first video frame to generate a second encoded instance of the first video frame for inclusion in the second stream for the second frame period. In some embodiments, rendering the first stream includes rendering each video frame of the first stream in sequence in a frame buffer at the first frame rate, and determining whether the second video frame of the first stream is ready to be encoded for the second frame period includes determining whether rendering of the second video frame in the frame buffer is completed in response to expiration of a timer representing an end of the first period.Selectively providing a plurality of encoded instances of at least one video frame may include determining whether a third video frame of the first stream is ready to be encoded for a third frame period following the second period; in response to determining that the third video frame is ready to be encoded, encoding the third video frame to generate an encoded instance of the third video frame for inclusion in the second stream for the third frame period; and in response to determining that the third video frame is not ready to be encoded, re-encoding the first video frame or duplicating the first encoded instance of the first video frame to generate a third encoded instance of the first video frame for inclusion in the second stream for the third frame period.

[0040] In some aspects, re-encoding the first video frame to generate the second encoded instance of the first video frame includes re-encoding the first video frame based at least in part on the first encoded instance of the first video frame to generate the second encoded instance of the first video frame with a higher image quality than the first encoded instance. In some embodiments, the method further includes determining the second frame rate based on a setting of the first device and further includes querying the second device over the network to determine the second frame rate. In some aspects, the method further includes executing a video game application on the first device, and rendering the first stream of unencoded video frames includes rendering the first stream in response to instructions executed in the video game application and further in response to user game controller commands received over the network.

[0041] In one aspect, a method includes providing a stream of encoded video frames at a specified frame rate for transmission from a first device to a second device over a network. Providing the stream of encoded video frames includes encoding a first video frame to generate a first encoded video frame for inclusion in the stream for a first frame period corresponding to the specified frame rate, and in response to determining that a second video frame is not ready for encoding for a second frame period following the first frame period, re-encoding the first rendered video frame to generate a second encoded video frame for inclusion in the stream for the second frame period. In some aspects, the method further includes in response to determining that the second video frame is ready for encoding for the second frame period, encoding the second video frame to generate a third encoded video frame for inclusion in the stream for the second frame period. In some embodiments, the method further includes, in response to determining that a third video frame is not ready for encoding for a third frame period following the second frame period, re-encoding the first video frame to generate a third encoded video frame for inclusion in the stream for the third frame period. The method may also include determining the specified frame rate based on at least one of a default setting of the first device, a user-specified setting, a query from the first device to the second device over the network, and an analysis of the network. Determining that the second video frame is not ready may include determining that a frame buffer of the first device does not contain a completed rendering of the second video frame upon expiration of a timer corresponding to a start of the second frame period.The method may include rendering, at the first device, a sequence of video frames at a frame rate lower than the specified frame rate for buffering in the frame buffer, the sequence including the first video frames and the second video frames. In some aspects, re-encoding the first video frames to generate second encoded video frames includes re-encoding the first video frames based in part on the first encoded video frames to generate the second encoded video frames at a higher image quality than the first encoded video frames.

[0042] According to another aspect, a system includes a first device, the first device including a network interface configured to couple to a network, and an encoder coupled to the network interface and configured to encode a first stream of rendered video frames having a first frame rate to generate a second stream of encoded video frames for transmission over the network via the network interface, the second stream having a second frame rate greater than the first frame rate, the encoder configured to selectively encode multiple instances of at least one video frame of the first stream for inclusion in the second stream to compensate for a difference between the first frame rate and the second frame rate. The system may also include a second device coupled to the network and configured to receive and decode the second stream of encoded video frames for playback on a display of a second device, the second device may include at least one of a keyboard or a game controller to provide user input to the first device over the network, the first device including a server configured to execute a video game application to render the first stream of rendered video frames based on the user input.

[0043] In some embodiments, the encoder is configured to selectively provide a plurality of encoded instances of at least one video frame by: encoding a first video frame of the first stream to generate a first encoded instance of the first video frame for inclusion in the second stream for a first frame period; determining whether a second video frame of the first stream is ready to be encoded for a second frame period following the first frame period, the first frame period and the second frame period being based on the second frame rate; in response to determining that the second video frame is ready to be encoded, encoding the second video frame to generate a encoded instance of the second video frame for inclusion in the second stream for the second frame period; and in response to determining that the second video frame is not ready to be encoded, re-encoding the first video frame or replicating the first encoded instance of the first video frame to generate a second encoded instance of the first video frame for inclusion in the second stream for the second frame period. The encoder can be configured to render each video frame of the first stream in sequence in a frame buffer at the first frame rate, and the encoder can be configured to determine whether the second video frame of the first stream is ready to be encoded for the second frame period by determining whether rendering of the second video frame in the frame buffer is completed in response to expiration of a timer.

[0044] In some aspects, the encoder is configured to selectively provide a plurality of encoded instances of at least one video frame by: determining whether a third video frame of the first stream is ready to be encoded for a third frame period following the second frame period; in response to determining that the third video frame is ready to be encoded, encoding the third video frame to generate an encoded instance of the third video frame for inclusion in the second stream for the third frame period; and in response to determining that the third video frame is not ready to be encoded, re-encoding the first video frame to generate a third encoded instance of the first video frame for inclusion in the second stream for the third frame period. The encoder can be configured to re-encode the first video frame based at least in part on the first encoded instance of the first video frame to generate the second encoded instance of the first video frame with a higher image quality than the first encoded instance of the first video frame. In some aspects, the encoder is further configured to determine the second frame rate based on at least one of a default setting, a user-specified setting, an analysis of the network, and a query to a second device coupled to the network and configured to receive and decode a second stream of the encoded video frames for playback on a display of the second device.

[0045] In yet another aspect, a non-transitory computer-readable medium embodies a set of executable instructions that operate at least one processor to perform a method or combination of the methods described above.

[0046] In some embodiments, some aspects of the techniques described above are realized by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, caches, random access memories (RAMs), or other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other formats of instructions that are interpreted or executable by one or more processors.

[0047] A computer-readable storage medium includes any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray® disks), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS)-based storage media. A computer-readable storage medium can be embedded in a computing system (e.g., system RAM or ROM), fixed to a computing system (e.g., magnetic hard drive), removably attached to a computing system (e.g., optical disks or universal serial bus (USB)-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0048] It should be noted that not all of the operations or elements described above in the general description are required. Parts of a particular operation or device may not be required. In addition to those described above, one or more additional operations may be performed or one or more elements may be included. Furthermore, the order in which the operations are listed is not necessarily the order in which these operations are performed. Also, the concepts are described with reference to specific embodiments. However, those skilled in the art will appreciate that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the following claims. Thus, the specification and drawings should be regarded as illustrative rather than restrictive, and all modifications are intended to be included within the scope of the present disclosure.

[0049] Benefits, other advantages, and solutions to problems have been described above with reference to certain embodiments. However, the benefits, advantages, solutions to problems, and any features that may give rise to or make more prominent any benefit, advantage, or solution should not be construed as key, necessary, or essential features of any or all claims. Moreover, the specific embodiments disclosed above are illustrative only. The disclosed subject matter may be modified and practiced in different but equivalent manners, as will be apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design shown herein, other than as set forth in the following claims. It is therefore apparent that the specific embodiments disclosed above may be altered or modified, with all variations being considered within the scope of the disclosed subject matter. The protection sought herein is therefore set forth in the following claims.

Claims

1. 1. A method comprising: Rendering, at a first device, a first stream of unencoded video frames at a first frame rate; and encoding, at the first device, video frames of the first stream to generate a second stream of encoded video frames, the second stream having a second frame rate, the second frame rate being a target frame rate for transmission of the second stream, wherein encoding the video frames of the first stream includes, based on the first frame rate effectively falling below the second frame rate, selectively providing a plurality of encoded instances of at least one video frame of the first stream for inclusion in the second stream to compensate for a difference between the first frame rate and the second frame rate to increase an effective frame rate of the second stream to the second frame rate, the method further comprising: providing the second stream for transmission over a network to a second device.

2. Selectively providing a plurality of encoded instances of the at least one video frame includes: encoding a first video frame of the first stream to generate a first encoded instance of the first video frame for inclusion in the second stream for a first frame period; determining whether a second video frame of the first stream is ready to be encoded for a second frame period following the first frame period, the first frame period and the second frame period being based on the second frame rate; moreover, In response to determining that the second video frame is ready to be encoded, encode the second video frame and generate an encoded instance of the second video frame for inclusion in the second stream for the second frame period. Generating, 2. The method of claim 1, comprising: in response to determining that the second video frame is not ready to be encoded, re-encoding the first video frame or replicating the first encoded instance of the first video frame to generate a second encoded instance of the first video frame for inclusion in the second stream for the second frame period.

3. Rendering the first stream includes rendering each video frame of the first stream in sequence in a frame buffer at the first frame rate; Determining whether the second video frame of the first stream is ready to be encoded for the second frame period comprises:

3. The method of claim 2, further comprising determining whether rendering of the second video frame in the frame buffer is complete in response to expiration of a timer representing an end of the first frame period.

4. Selectively providing a plurality of encoded instances of at least one video frame includes: determining whether a third video frame of the first stream is ready to be encoded for a third frame period following the second frame period; in response to determining that the third video frame is ready to be encoded, encoding the third video frame to generate an encoded instance of the third video frame for inclusion in the second stream for the third frame period; 4. The method of claim 2 or 3, comprising: in response to determining that the third video frame is not ready to be encoded, re-encoding the first video frame or replicating the first encoded instance of the first video frame to generate a third encoded instance of the first video frame for inclusion in the second stream for the third frame period.

5. Re-encoding the first video frame to generate the second encoded instance of the first video frame includes:

5. The method of claim 2, further comprising: re-encoding the first video frame based at least in part on the first encoded instance of the first video frame to generate the second encoded instance of the first video frame with a higher image quality than the first encoded instance.

6. moreover, The method of claim 1 , comprising determining the second frame rate based on a setting of the first device.

7. moreover, The method of claim 6 , further comprising querying the second device over the network to determine the second frame rate.

8. moreover, executing a video game application on the first device; 8. The method of claim 1, wherein rendering the first stream of unencoded video frames comprises rendering the first stream in response to instructions executed in the video game application and further in response to user game controller commands received over the network.

9. 1. A method comprising: providing a stream of encoded video frames at a specified frame rate for transmission from a first device to a second device over a network, the providing of the stream of encoded video frames comprising: encoding a first video frame to generate a first encoded video frame for inclusion in the stream for a first frame period corresponding to the specified frame rate; in response to determining that rendering of a second video frame has been skipped or delayed and that the second video frame is not ready for encoding for a second frame period following the first frame period, re-encoding the rendered first video frame to generate a second encoded video frame for inclusion in the stream for the second frame period.

10. moreover, 10. The method of claim 9, comprising, in response to determining that the second video frame is ready for encoding for the second frame period, encoding the second video frame to generate a third encoded video frame for inclusion in the stream for the second frame period.

11. moreover, 11. The method of claim 9 or 10, comprising, in response to determining that a third video frame is not ready for encoding for a third frame period following the second frame period, re-encoding the first video frame to generate a third encoded video frame for inclusion in the stream for the third frame period.

12. moreover, 12. The method of claim 9, further comprising determining the specified frame rate based on at least one of a default setting of the first device, a user-specified setting, a query from the first device to the second device over the network, and an analysis of the network.

13. Determining that the second video frame is not ready includes:

13. A method according to any one of claims 9 to 12, comprising determining, upon expiration of a timer corresponding to a start of the second frame period, that a frame buffer of the first device does not contain a completed rendering of the second video frame.

14. moreover, 14. The method of claim 13, comprising rendering, at the first device, a sequence of video frames at a frame rate lower than the specified frame rate for buffering in the frame buffer, the sequence including the first video frame and the second video frame.

15. Re-encoding the first video frame to generate a second encoded video frame includes:

15. The method of any one of claims 9 to 14, comprising re-encoding the first encoded video frame based in part on the first encoded video frame to generate the second encoded video frame at a higher image quality than the first encoded video frame.

16. 1. A system comprising: A first device, the first device comprising: a network interface configured to couple to a network; and an encoder coupled to the network interface and configured to encode a first stream of rendered video frames having a first frame rate to generate a second stream of encoded video frames for transmission over the network via the network interface, the second stream having a second frame rate, the second frame rate being a target frame rate for transmission of the second stream, the encoder configured to, based on the first frame rate effectively falling below the second frame rate, selectively encode multiple instances of at least one video frame of the first stream for inclusion in the second stream to compensate for a difference between the first frame rate and the second frame rate to increase an effective frame rate of the second stream to the second frame rate.

17. 17. The system of claim 16, further comprising a second device coupled to the network and configured to receive and decode a second stream of the encoded video frames for playback on a display of a second device.

18. the second device includes at least one of a keyboard or a game controller for providing user input to the first device over the network; 20. The system of claim 17, wherein the first device comprises a server configured to execute a video game application to render the first stream of rendered video frames based on the user input.

19. The encoder is configured to selectively provide a plurality of encoded instances of at least one video frame by: encoding a first video frame of the first stream to generate a first encoded instance of the first video frame for inclusion in the second stream for a first frame period; and determining whether a second video frame of the first stream is ready to be encoded for a second frame period following the first frame period, the first frame period and the second frame period being based on the second frame rate, the following configuration further comprising: in response to determining that the second video frame is ready to be encoded, encoding the second video frame to generate an encoded instance of the second video frame for inclusion in the second stream for the second frame period; 19. The system of claim 16, further comprising: in response to determining that the second video frame is not ready to be encoded, re-encoding the first video frame or replicating the first encoded instance of the first video frame to generate a second encoded instance of the first video frame for inclusion in the second stream for the second frame period.

20. the encoder is configured to render each video frame of the first stream in sequence in a frame buffer at the first frame rate; 20. The system of claim 19, wherein the encoder is configured to determine whether the second video frame of the first stream is ready to be encoded for the second frame period by determining whether rendering of the second video frame in the frame buffer is complete in response to expiration of a timer.

21. The encoder is configured to selectively provide a plurality of encoded instances of at least one video frame by: determining whether a third video frame of the first stream is ready to be encoded for a third frame period following the second frame period; in response to determining that the third video frame is ready to be encoded, encoding the third video frame to generate an encoded instance of the third video frame for inclusion in the second stream for the third frame period; and in response to determining that the third video frame is not ready to be encoded, re-encoding the first video frame to generate a third encoded instance of the first video frame for inclusion in the second stream, for the third frame period.

22. 20. The system of claim 19, wherein the encoder is configured to re-encode the first video frame based at least in part on the first encoded instance of the first video frame to generate the second encoded instance of the first video frame with higher image quality than the first encoded instance of the first video frame.

23. The encoder further comprises:

23. The system of claim 16, configured to determine the second frame rate based on at least one of a default setting, a user-specified setting, analysis of the network, and a query to a second device, the second device coupled to the network and configured to receive and decode the second stream of encoded video frames for playback on a display of the second device.

24. A program for causing at least one processor to carry out the method according to any one of claims 1 to 15.

Citation Information

Patent Citations

  • Network communication system

    JP1993216800A

  • Image signal decoding method, decoder, image signal coding method, coder and image signal processing method and processing system

    JP1999252566A

  • multiplayer video game system

    JP2008535622A

  • Method, apparatus and system used for multimedia signal encoding

    JP2009500951A

  • Video signal dividing device, video transfer transmission apparatus, video transfer reception apparatus and video transfer system

    JP2014086770A