Remote video stream frame processing

Abstract

A method of processing a video stream, comprising: decoding the video stream into a plurality of frames, each frame comprising video information and a timestamp; storing the frame in a buffer; determining a display frame rate associated with the video display; determining a future display time value from the display frame rate and the current time; a first frame of the plurality of frames is processed based on a future display time value, a timestamp associated with the first frame, and a timestamp of an additional frame stored in the buffer, where the processing includes one of: (a) sending the first frame to the video display and removing the first frame from the buffer, (b) refraining from transmitting the first frame to the video display and refraining from removing the first frame from the buffer, or (c) refraining from transmitting the first frame to the video display and removing the first frame from the buffer.

Description

BACKGROUND

[0001] Video and other multimedia content is often streamed between a host and client device(s) in a remote desktop environment. Given two independent devices without electronic interlocking, potentially asymmetric polling intervals, refresh rates, clock skew / drift, phase difference, compression latency, and network queuing, there can be desynchronization between capture at the host side and rendering of remote frames of the video stream at the client side. Under these conditions, for example, video content containing smooth motion or other gentle panning actions can stutter on the client side. It would be helpful for the system to analyze video stream frame metadata and make decisions about whether to send or discard incoming frames from the host device to minimize inter-frame rendering jitter on the client side. BRIEF DESCRIPTION OF DRAWINGS

[0002] Various features will now be described with reference to the following figures. Common reference numerals can be used throughout the figures to refer to like elements. The figures are provided to illustrate examples described herein and are not intended to limit the scope of the disclosure.

[0003] Figure 1 is a block diagram illustrating an example remote video stream processing system for providing a video stream to a display; Figure 2 is a block diagram illustrating an example client device in a remote video stream processing system; Figure 3 illustrates an example frame processing analysis performed by a client device; Figure 4 is a flow diagram depicting an example routine for processing a video stream; Figure 5 is a flow diagram depicting an additional example routine for processing a video stream by sending frames of the video stream to a display; and Figure 6 is a flow diagram depicting an additional example routine for processing a video stream by withholding sending frames of the video stream to a display. DETAILED DESCRIPTION

[0004] A remote video stream processing system can be configured to provide smooth and accurate reproduction of content playback on the client side of a remote desktop environment. In a common application, the system can assign timestamps to frames of an input video stream stored in a buffer. The system can further determine, on a frame-by-frame basis, whether to send a frame of the video stream to a client-side display based on timestamps of buffered frames and potential errors between streaming content at the host and playback of the display.

[0005] Aspects of the disclosure relate to a method of processing a video stream, comprising: decoding the video stream into a plurality of frames, each frame comprising video information and a timestamp; storing the plurality of frames in a buffer; determining a display frame rate associated with a video display; determining a future display time value from the display frame rate and a current time; processing a first frame of the plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in the buffer; wherein the processing comprises one of: (a) sending the first frame to the video display and removing the first frame from the buffer, (b) refraining from sending the first frame to the video display and refraining from removing the first frame from the buffer, or (c) refraining from sending the first frame to the video display and removing the first frame from the buffer.

[0006] In some aspects, the techniques described herein relate to a method of processing a video stream, wherein the audio stream is uncompressed.

[0007] In some aspects, the techniques described herein relate to a method of processing a video stream, wherein the additional frames stored in the buffer comprise all frames stored in the buffer except the first frame.

[0008] In some aspects, the techniques described herein relate to a method of processing a video stream, further comprising receiving an indication of a number of frames to include as the additional frames stored in the buffer.

[0009] In some aspects, the techniques described herein relate to a method of processing a video stream, further comprising receiving a latency tolerance value, and wherein the processing the first frame of the plurality of frames is further based on the latency tolerance value.

[0010] In some aspects, the techniques described herein relate to a method of processing a video stream, further comprising determining an error value associated with displaying the first frame and not displaying the first frame, and wherein the processing is further based on the error value.

[0011] In some aspects, the techniques described herein relate to a method of processing a video stream, wherein the error value corresponds to a latency value associated with a next frame to be displayed.

[0012] In some aspects, the techniques described herein relate to a method of processing a video stream, wherein the processing comprises refraining from sending the first frame to the video display and removing the first frame from the buffer when the timestamp associated with the first frame is earlier than the display time.

[0013] In some aspects, the techniques described herein relate to a method of processing a video stream, wherein the processing comprises sending the first frame to the video display and removing the first frame from the buffer when the timestamp of the first frame is more than an acceptable latency value from the display time.

[0014] In some aspects, the technology described herein relates to a method of processing a video stream, wherein the acceptable latency value is 150 ms.

[0015] Aspects of the disclosure relate to a method of processing a video stream, comprising: storing a plurality of frames of a video stream in a buffer, each frame comprising video information and a timestamp; determining a display frame rate associated with a video display; determining a future display time value from the display frame rate and a current time; and processing a first frame of the plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in the buffer, wherein the processing comprises sending the first frame to the video display and removing the first frame from the buffer when the timestamp of the first frame is less than the acceptable latency value from the display time.

[0016] In some aspects, the technology described herein relates to a method of processing a video stream, wherein the additional frames stored in the buffer comprise all frames stored in the buffer except the first frame.

[0017] In some aspects, the technology described herein relates to a method of processing a video stream, further comprising receiving an indication of a number of frames to include as the additional frames stored in the buffer.

[0018] In some aspects, the technology described herein relates to a method of processing a video stream, further comprising receiving a latency tolerance value, and wherein the processing the first frame of the plurality of frames is further based on the latency tolerance value.

[0019] In some aspects, the technology described herein relates to a method of processing a video stream, further comprising determining an error value associated with displaying the first frame and not displaying the first frame, and wherein the processing is further based on the error value.

[0020] In some aspects, the technology described herein relates to a method of processing a video stream, wherein the error value corresponds to a latency value associated with a next frame to be displayed.

[0021] Aspects of the disclosure relate to a method of processing a video stream, comprising: storing a plurality of frames of a video stream in a buffer, each frame comprising video information and a timestamp; determining a display frame rate associated with a video display; determining a future display time value from the display frame rate and a current time; processing a first frame of the plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in the buffer; wherein the processing comprises refraining from sending the first frame to the video display and removing the first frame from the buffer when the timestamp associated with the first frame is earlier than the display time.

[0022] In some aspects, the techniques described herein relate to a method of processing a video stream, wherein the additional frames stored in the buffer include all frames stored in the buffer except the first frame.

[0023] In some aspects, the techniques described herein relate to a method of processing a video stream, further comprising receiving an indication of a number of frames to include as additional frames stored in the buffer.

[0024] In some aspects, the techniques described herein relate to a method of processing a video stream, further comprising receiving a latency tolerance value, and wherein said processing the first frame of the plurality of frames is further based on the latency tolerance value.

[0025] In some aspects, the techniques described herein relate to a method of processing a video stream, further comprising determining an error value associated with displaying the first frame and not displaying the first frame, and wherein said processing is further based on the error value, wherein the error value corresponds to a latency value associated with a next frame to be displayed.

[0026] Figure 1 An example of a remote video stream processing system 100 for providing a video stream to a display is illustrated. The remote video stream processing system 100 includes a host device 102, a client device 104, and a display 106.

[0027] As Figure 1 illustrated in FIG. 1, the host device 102 can be any device capable of sending a video stream. The host device 102 can include a personal computing device, a laptop computing device, a tablet computing device, an e-reader device, a wearable computing device, a mobile device (e.g., cellular and other mobile phones, smart phones, media players, handheld gaming devices, etc.), a streaming media device, and various other electronic devices and appliances. In some examples, the host device 102 can be a virtual machine. The host device 102 can be communicatively coupled to the client device 104. For example, the host device 102 and the client device 104 can connect via a network 150, wireless communication (such as Bluetooth® ), according to IEEE 802 standards, etc.

[0028] The client device 104 can be any device capable of receiving and transmitting a video stream to a display. The client device 104 can include a personal computing device, a laptop computing device, a tablet computing device, an e-reader device, a wearable computing device, a mobile device (e.g., cellular and other mobile phones, smart phones, media players, handheld gaming devices, etc.), a streaming media device, and various other electronic devices and appliances. In some examples, the client device 104 can be configured to receive video stream content from the host device 102 in the form of video information (e.g., frames) and timestamps. In some examples, the client device 104 can make determinations of whether to transmit or discard particular frames of the video stream to the display 106.

[0029] The display 106 can be any device capable of displaying a video stream. As illustrated, the display 106 can be physically connected to the client device 104. For example, the display device 106 can be connected to the client device 104 using a wired connection. In some examples, the display 106 can be integrated within the hardware of the client device 104 and refer to the screen of the client device 104. In some examples, the two devices can be electronically synchronized due to the physical connection between the client device 104 and the display 106.

[0030] Figure 2 is a block diagram illustrating an example client device 104 in the remote video stream processing system 100. As shown, the client device 104 can be communicatively coupled to the host device 102 and the display 106. In one example, the host device 102 can transmit a video stream to the client device 104 for display on the display 106. Each component of the client device 104 will be described in turn.

[0031] As Figure 2 As illustrated in FIG. 2, the client device 104 includes a processor(s) 202, a network interface 204, such as a network interface card (“NIC”), a computer-readable medium 206, such as a high-density disk (“HDD”), a solid-state drive (“SDD”), a flash drive, and / or other persistent non-transitory computer-readable medium, a graphics processing unit (“GPU”) 208, a storage 210, such as random access memory (“RAM”) and / or other volatile non-transitory computer-readable medium, a buffer 212, and a frame processing module 214.

[0032] The client device 104 includes a processor(s) 202, such as a physical central processing unit (“CPU”), and / or the like. The processor(s) 202 can decode a video stream as it is being received from the host device 102. For example, the processor(s) 202 can decode the video stream into a plurality of frames (e.g., video information and corresponding timestamps).

[0033] In some examples, the processor(s) 202 will determine a timestamp for each decoded frame of the video stream. In some examples, the timestamp for each frame is high-granularity such that the frame rate of the display 106 can be resolved. In some examples, if the timestamp from the host device 102 is not available, the processor(s) 202 can inspect the video stream at the packet level and estimate the timestamp between frames that are continuously transmitted from the host device 102. In some examples, the time increment between consecutive frames of the decoded video stream can correspond to the time increment of the original streamed video on the host side.

[0034] In some examples, the processor(s) 202 can be connected to other components within the client device 104. For example, the processor(s) 202 can send decoded frames of the video stream to the buffer 212.

[0035] The buffer 212 can be an area within the memory 210 that is configured to store frames of the video stream and can include both the video information and the corresponding timestamp for each frame. In some examples, the buffer 212 can store any number of frames at a time. For example, the buffer 212 can store 1, 2, 5, 10... N frames at a time. In some examples, the amount of frames stored at a time by the buffer 212 can depend on the speed at which the processor(s) 202 receive and decode the video stream, which can further depend on the network conditions, connection conditions, etc. between the host device 102 and the client device 104.

[0036] The GPU 208 can include graphics processing cores with specialized circuitry and logic to process images / video and / or other digital signal-based data. In some examples, the GPU 208 is located within the client device 104. In some examples, the GPU 208 is located within the display 106 and is in communication with the client device 104. In some examples where the display 106 is integrated within the client device 104, the GPU 208 is located within the client device 104. In some examples, the GPU 208 can manage the retrieval and display of frames according to a frame rate associated with the display 106. As understood in the art, the frame rate corresponds to the frequency or rate at which consecutive images or frames are displayed (e.g., frames per second).

[0037] The frame processing module 214 can communicate with components of the client device 104 in order to determine whether certain frames of the video stream are to be sent to the display 106. In some examples, the frame processing module 214 can be implemented as software. In some examples, given a number of frames in the buffer 212, the frame processing module 214 can make a determination as to whether to send a first buffered frame to the display 106 based on the timestamps of all buffered frames and a future display time value. In some examples, the future display time value corresponds to a frame rate of the display 106 and a current time.

[0038] In some examples, the frame processing module 214 will make a determination as to whether to send a frame to the display 106 in order to minimize inter-frame presentation jitter. For example, at the occurrence of a display refresh event (as determined by the frame rate of the display’s GPU), the frame processing module 214 will either push a frame to the display, wait to push a frame, or discard a frame. In some examples, the frame processing module 214 will make a decision in order to minimize timing error, i.e., to minimize latency.

[0039] Figure 3 A frame processing analysis performed by the frame processing module 214 is illustrated. Figure 3 A frame rate of the display 106 is illustrated, shown by vertical dashed lines, spaced equidistantly apart by ΔFR. Figure 3 A buffer 212, which can contain a number of frames, is also illustrated, as shown by frames Fl, F2, F3, F4, and F5. In some examples, the buffer 212 can contain more or fewer frames. In one example, frames Fl-F5 can be all of the frames stored in the buffer 212 at a current time T=0.

[0040] In some examples, the frame processing module 214 can decide at T=0 whether to send, hold, or discard the first frame Fl. This decision can be illustrated by a send scenario 302, a hold scenario 304, and a discard scenario 306. In some examples, the decision as to how to dispose of Fl can be based on the total buffered frames Fl-F5. For example, the analysis performed by the frame processing module 214 can include calculating a total absolute error associated with the potential scenarios. As used herein, an absolute error can refer to a time between a buffered frame and a refresh period of the display’s frame rate, as shown by ΔE. As used herein, a total absolute error can refer to a sum of the ΔE for a scenario.

[0041] In some examples, in calculating the absolute error, the frame processing module 214 can pair each buffered frame with a future display refresh period. Note that “future” can refer to a time greater than T=0. In some examples, the frame processing module 214 can calculate the ΔE between each frame and its corresponding refresh period. In some examples, the frame processing module 214 can pair a single frame with a single future display refresh period.

[0042] For example, in transmission scenario 302, if the first frame F1 is sent to the display at the current time T=0, the frame processing module 214 can calculate the error. In this example, ΔE1 can refer to the error between the F1 timestamp and the future refresh cycle. Similarly, ΔE2-ΔE5 can refer to the error between the F2-F5 timestamps and the paired future refresh cycles. The total absolute error associated with transmission scenario 302 can be represented by the sum of ΔE.

[0043] In hold scenario 304, if the first frame F1 is held until the upcoming display refresh cycle, the frame processing module 214 can calculate the total error. In this example, F1 is sent to the display in the future display refresh cycle, and ΔE1 is 0. However, ΔE2-ΔE5 may refer to the error between the F2-F5 timestamps and the paired future refresh cycles. The total absolute error associated with send scenario 302 can be represented by the sum of ΔE.

[0044] In discard scenario 406, frame processing module 214 may discard the first frame, for example, by not sending it to the display and removing it from the buffer. In this example, F1 is removed from the buffer. In this example, ΔE2-ΔE5 may refer to the error between the F2-F5 timestamp and the paired future refresh cycle. The total absolute error associated with sending scenario 302 can be represented by the sum of ΔE.

[0045] In one example, frame processing module 214 may decide to execute the scene with the lowest absolute error.

[0046] Figure 4 This is a flowchart depicting an example routine 400 for processing video streams. Routine 400 can be implemented by a client device 104 within a remote video stream processing system 100.

[0047] At box 402, client device 104 decodes the video stream into frames. In one example, client device 104 decodes the video stream into multiple frames. In one example, each frame includes video information and a timestamp.

[0048] At box 404, client device 104 stores frames in a buffer. In one example, client device 104 stores multiple frames in the buffer. In one example, additional frames stored in the buffer include all frames stored in the buffer except for the first frame. In one example, client device 104 may receive an indication of the number of frames to be included as additional frames stored in the buffer.

[0049] At box 406, client device 104 determines the display frame rate. In one example, client device 104 determines the display frame rate associated with the video display.

[0050] At box 408, client device 104 determines the future display time value. In one example, client device 104 determines the future display time value from the display frame rate and the current time.

[0051] At box 410, client device 104 processes the first frame based on the display frame rate and a future display time value. In one example, client device 104 processes the first frame of a plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in a buffer. In one example, the processing includes one of the following: (a) sending the first frame to a video display and removing the first frame from the buffer, (b) suppressing the sending of the first frame to the video display and suppressing the removal of the first frame from the buffer, or (c) suppressing the sending of the first frame to the video display and removing the first frame from the buffer.

[0052] In one example, client device 104 receives a delay tolerance value, and the first frame of the processing of multiple frames is further based on the delay tolerance value.

[0053] In one example, client device 104 determines an error value associated with displaying the first frame and not displaying the first frame, and the processing is further based on the error value. In one example, the error value corresponds to a delay value associated with the next frame to be displayed.

[0054] In one example, the processing includes suppressing the sending of the first frame to the video display and removing the first frame from the buffer when the timestamp associated with the first frame is earlier than the display time.

[0055] In one example, the process involves sending the first frame to the video display and removing it from the buffer when the timestamp of the first frame exceeds the display time but is less than an acceptable delay value. In one example, the acceptable delay value is 150ms.

[0056] Figure 5 This is a flowchart depicting an example routine 500 for processing a video stream by sending frames of the video stream to a display. Routine 500 can be implemented by a client device 104 within a remote video stream processing system 100.

[0057] At box 502, client device 104 stores frames in a buffer. In one example, client device 104 stores multiple frames of a video stream in the buffer. In one example, each frame includes video information and a timestamp. In one example, additional frames stored in the buffer include all frames stored in the buffer except for the first frame. In one example, client device 104 receives an indication of the number of frames to be included as additional frames stored in the buffer.

[0058] At box 504, client device 104 determines the display frame rate. In one example, client device 104 determines the display frame rate associated with the video display.

[0059] At box 506, client device 104 determines the future display time value. In one example, client device 104 determines the future display time value from the display frame rate and the current time.

[0060] At box 508, client device 104 processes the first frame based on a future display time value and a timestamp. In one example, client device 104 processes the first frame of a set of multiple frames based on a future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in a buffer.

[0061] In one example, the client device receives a delay tolerance value, and the first frame of the processing of multiple frames is further based on the delay tolerance value.

[0062] In one example, client device 104 determines an error value associated with displaying the first frame and not displaying the first frame, and the processing is further based on the error value. In one example, the error value corresponds to a delay value associated with the next frame to be displayed.

[0063] At box 510, client device 104 sends the first frame to the display. In one example, when the timestamp of the first frame exceeds the display time but is less than an acceptable delay value, client device 104 removes the first frame from the buffer.

[0064] Figure 6 This is a flowchart depicting an example routine 600 for processing a video stream by suppressing frames sent to the display. Routine 600 can be implemented by a client device 104 within a remote video stream processing system 100.

[0065] At box 602, client device 104 stores frames in a buffer. In one example, client device 104 stores multiple frames of the video stream in the buffer. In one example, each frame includes video information and a timestamp.

[0066] At box 604, client device 104 determines the display frame rate. In one example, client device 104 determines the display frame rate associated with the video display.

[0067] At box 606, client device 104 determines the future display time value. In one example, client device 104 determines the future display time value from the display frame rate and the current time.

[0068] At box 608, client device 104 processes the first frame based on a future display time value and a timestamp. In one example, client device 104 processes the first frame of a set of multiple frames based on a future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in a buffer.

[0069] At box 610, client device 104 suppresses the transmission of the first frame to the display. In one example, client device 104 removes the first frame from the buffer when the timestamp associated with the first frame is earlier than the display time.

[0070] It should be understood that not all objectives or advantages may be achieved according to any particular embodiment described herein. Therefore, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one or more advantages as taught herein, without necessarily achieving other objectives or advantages that may be taught or suggested herein.

[0071] All processes described herein can be embodied in and fully automated via software code modules, which include one or more specific computer-executable instructions executed by a computing system. The computing system may include one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

[0072] Many other variations besides those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain actions, events, or functions of any algorithm described herein may be performed in a different order, may be added, combined, or omitted together (e.g., not all described actions or events are necessary for the practice of the algorithm). Furthermore, in some embodiments, actions or events may be performed concurrently, for example, through multithreaded processing, interrupt handling, or multiple processors or processor cores simultaneously, or on other parallel architectures, rather than sequentially. Additionally, different tasks or processes may be performed by different machines and / or computing systems that can operate together.

[0073] The various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein can be implemented or executed by a machine such as a processing unit or processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may be a microprocessor, but alternatively, it may be a controller, a microcontroller, or a state machine, a combination thereof, etc. A processor may include circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other combination of such configurations. Although the description herein primarily concerns digital technologies, a processor may also primarily comprise analog components. A computing environment may include any type of computer system, including, but not limited to, microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within appliances, to name just a few.

[0074] Among other things, conditional language such as “can”, “could”, or “might”, unless specifically stated otherwise, is understood in the context to generally convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are required in any way for use in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether such features, elements, and / or steps are included in any particular embodiment or whether they are to be performed in any particular embodiment, with or without user input or prompting.

[0075] Unless otherwise specifically stated, disjunctive language such as the phrase “at least one of X, Y, or Z” is otherwise understood as being generally used to represent a context in which items, terms, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0076] Any process description, element, or block depicted in the flowcharts described herein and / or in the accompanying drawings should be understood as potentially representing modules, code segments, or code portions that include one or more executable instructions for implementing specific logical functions or elements within a process. Alternative implementations are included within the scope of the embodiments described herein, wherein elements or functions may be omitted, performed out of the order shown or discussed (including substantially simultaneous or in reverse order, depending on the functions involved), as will be understood by those skilled in the art.

[0077] Unless otherwise expressly stated, articles such as “a” or “an” should generally be interpreted as including one or more of the described items. Therefore, phrases such as “the device is configured to” are intended to include one or more of the described devices. Thus, one or more of the described devices can also be configured together to perform the descriptions. For example, “the processor is configured to perform descriptions A, B, and C” could include a first processor configured to perform description A working in conjunction with a second processor configured to perform descriptions B and C.

Claims

1. A method of processing a video stream, comprising: decoding a video stream into a plurality of frames, each frame including video information and a timestamp; storing the plurality of frames in a buffer; determining a display frame rate associated with a video display; determining a future display time value from the display frame rate and a current time; processing a first frame of the plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in the buffer; wherein the processing includes one of (a) sending the first frame to the video display and removing the first frame from the buffer, (b) refraining from sending the first frame to the video display and refraining from removing the first frame from the buffer, or (c) refraining from sending the first frame to the video display and removing the first frame from the buffer.

2. The method of claim 1, wherein the additional frames stored in the buffer include all frames stored in the buffer except the first frame.

3. The method of claim 1, further comprising receiving an indication of a number of frames to include as the additional frames stored in the buffer.

4. The method of claim 1, further comprising receiving a latency tolerance value, and wherein the processing the first frame of the plurality of frames is further based on the latency tolerance value.

5. The method of claim 1, further comprising determining an error value associated with displaying the first frame and not displaying the first frame, and wherein the processing is further based on the error value.

6. The method of claim 5, wherein the error value corresponds to a latency value associated with a next frame to be displayed.

7. The method of claim 1, wherein the processing includes refraining from sending the first frame to the video display and removing the first frame from the buffer when the timestamp associated with the first frame is earlier than the display time.

8. The method of claim 1, wherein the processing includes sending the first frame to the video display and removing the first frame from the buffer when the timestamp of the first frame exceeds the display time by less than an acceptable latency value.

9. The method of claim 8, wherein the acceptable latency value is 150 ms.

10. A method of processing a video stream, comprising: storing a plurality of frames of a video stream in a buffer, each frame including video information and a timestamp; determining a display frame rate associated with a video display; determining a future display time value from the display frame rate and a current time; and processing a first frame of the plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in the buffer, wherein the processing includes sending the first frame to the video display and removing the first frame from the buffer when the timestamp of the first frame exceeds the display time by less than an acceptable latency value.

11. The method of claim 10, wherein the additional frames stored in the buffer include all frames stored in the buffer except the first frame.

12. The method of claim 10, further comprising receiving an indication of a number of frames to include as the additional frames stored in the buffer. ​ 13. The method of claim 10, further comprising receiving a latency tolerance value, and wherein said processing the first one of the plurality of frames is further based on the latency tolerance value.

14. The method of claim 10, further comprising determining an error value associated with displaying the first frame and not displaying the first frame, and wherein said processing is further based on the error value.

15. The method of claim 14, wherein the error value corresponds to a latency value associated with a next frame to be displayed.

16. A method of processing a video stream, comprising: storing a plurality of frames of a video stream in a buffer, each frame including video information and a timestamp; determining a display frame rate associated with a video display; determining a future display time value from the display frame rate and a current time; processing a first one of the plurality of frames based on the future display time value, a timestamp associated with the first frame, and timestamps of additional frames stored in the buffer; wherein said processing includes refraining from sending the first frame to the video display and removing the first frame from the buffer when the timestamp associated with the first frame is earlier than the display time.

17. The method of claim 16, wherein the additional frames stored in the buffer include all frames stored in the buffer except the first frame.

18. The method of claim 16, further comprising receiving an indication of a number of frames to include as the additional frames stored in the buffer.

19. The method of claim 16, further comprising receiving a latency tolerance value, and wherein said processing the first one of the plurality of frames is further based on the latency tolerance value.

20. The method of claim 16, further comprising determining an error value associated with displaying the first frame and not displaying the first frame, and wherein said processing is further based on the error value, wherein the error value corresponds to a latency value associated with a next frame to be displayed.

Images