Low latency video stream-modified frame rate

Abstract

The invention relates to a low latency video stream-modified frame rate. In an example, an apparatus may include logic to receive a plurality of video frames from a video source, logic to process the plurality of video frames, logic to provide at least some of the plurality of video frames to a video receiver and logic to reduce a display latency of each of the video frames provided to the video receiver by changing a frame rate of some or all of the frames.

Description

[0001] Cross-reference of related applications

[0002] This application is related to the following applications filed on the same date by their respective inventors: U.S. Patent Application No. 18 / 932,328 entitled "Low Latency Video Streaming – Reducing Frame Buffers" (Attorney-in-charge No. 5009.220197US01); and U.S. Patent Application No. 18 / 932,361 entitled "Low Latency Video Streaming – Partial Frames" (Attorney-in-charge No. 5009.220197US02). The relevant disclosure of each of these applications is incorporated herein by reference for all purposes. Technical Field

[0003] This document generally relates to video streaming, and more specifically, to low-latency video streaming by reducing the display latency of video frames provided to a video receiver. Background Technology

[0004] Traditional video streaming, whether broadcast or IP-based, has historically relied on complex network infrastructure to deliver a smooth video experience to end users. This typically involves a video player decoding each video and audio frame and sending it to the TV in real time. Such traditional methods depend on complex network architectures and extensive buffering to ensure consistent frame delivery. This usually results in higher costs due to increased bandwidth and storage requirements. To address these challenges, various video coding standards (such as MPEG2, AVC, HEVC, VP9, ​​and AV1) have been developed for efficient compression, but this typically comes at the cost of increased computational complexity.

[0005] In addition, to ensure smooth video playback, buffering mechanisms have been implemented at different levels of the video pipeline, including cloud servers, networks, and video players (such as set-top boxes or OTT clients). For example, popular streaming services like YouTube and Netflix typically buffer 10 to 40 seconds of video frames.

[0006] Emerging applications such as cloud gaming, video conferencing, and virtual reality require low latency performance. These applications are driving the development of new network, encoder, and system standards. With increasing internet speeds, cloud gaming has become increasingly popular. In this model, the actual game server resides in the cloud, while the local game controller sends commands to the cloud server. The game is presented on the cloud server, and the encoded video is transmitted via video pipeline to the user's device (such as a TV or set-top box) for display.

[0007] Furthermore, video conferencing applications have become indispensable for remote work, online education, and telemedicine. Low latency is crucial for a seamless experience. Many users are turning to OTT devices or set-top boxes for larger screen displays rather than traditional conferencing equipment. Additionally, virtual reality experiences typically require high-resolution video and low latency. Cloud-based rendering provides the necessary processing power, while local devices can focus on displaying the rendered content.

[0008] For applications such as gaming, video conferencing, and virtual reality and the like, end-to-end latency is more important than smooth video. Traditional set-top boxes and OTT devices designed for smooth streaming video typically rely on fixed frame rates and buffering at multiple stages of the video pipeline.

[0009] In other words, a video pipeline typically contains frame buffers at different stages, such as encoders, decoders, video processing, HDMI inputs, and within the TV itself. This buffering ensures smooth playback but also introduces latency because each stage buffers multiple frames to ensure that the full frame data is available at the input before being fed into the output stage. Traditional pipelines typically need to transmit the entire frame, regardless of the number of pixels that have changed. This approach also introduces significant latency, which is undesirable for latency-sensitive applications such as cloud gaming, video conferencing, and virtual reality.

[0010] In other instances, at HDMI interfaces, the frame rate is typically fixed and cannot be dynamically adjusted during playback. For example, in a 60 FPS configuration, a full frame must be sent every 1 / 60th of a second. If the next frame is not ready in time (i.e., underflow), the previous frame is repeated, potentially causing visual artifacts. This fixed frame rate requirement limits the ability to reduce latency in applications requiring real-time response. Summary of the Invention

[0011] On one hand, this disclosure provides an apparatus comprising: logic for receiving a plurality of video frames from a video source at a fixed frame rate, the plurality of received video frames including video frames comprising a first plurality of subframes; logic for processing at least some of the plurality of video frames to generate a plurality of processed video frames, wherein: processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes; and processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames; and logic for providing at least some of the plurality of processed video frames, comprising at least some of the processed first plurality of subframes, to a video receiver at the flexible frame rate.

[0012] On the other hand, this disclosure provides a method comprising: receiving a plurality of video frames from a video source at a fixed frame rate, the plurality of received video frames including video frames comprising a first plurality of subframes; processing at least some of the plurality of video frames to generate a plurality of processed video frames, wherein: processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes; and processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames; and providing at least some of the plurality of processed video frames, comprising at least some of the processed first plurality of subframes, to a video receiver at the flexible frame rate.

[0013] On the other hand, this disclosure provides a set-top box comprising: an input interface configured to receive a plurality of video frames from a video source at a fixed frame rate, the plurality of received video frames including video frames comprising a first plurality of subframes; a processor configured to process at least some of the plurality of video frames to generate a plurality of processed video frames, wherein: processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes; and processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames; and an output interface configured to provide at least some of the processed video frames, comprising at least some of the plurality of processed subframes, to a video receiver at the flexible frame rate. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating components of an apparatus for reducing latency in a video pipeline according to some embodiments.

[0015] Figure 2 This is a functional block diagram illustrating an apparatus having a video pipeline according to some embodiments.

[0016] Figure 3 It is a frame timing diagram of a video pipeline according to some embodiments.

[0017] Figure 4 This is a flowchart illustrating an exemplary method for reducing latency in a video pipeline according to some embodiments.

[0018] Figure 5 This is a flowchart illustrating an exemplary method for reducing latency in a video pipeline according to some embodiments.

[0019] Figure 6 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0020] Figure 7This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0021] Figure 8 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0022] Figure 9 This is a flowchart illustrating an exemplary method for reducing latency in a video pipeline according to some embodiments.

[0023] Figure 10 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0024] Figure 11 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0025] Figure 12 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0026] Figure 13 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0027] Figure 14 This describes a technique for generating video frames from a reference frame according to some embodiments.

[0028] Figure 15 This is a flowchart illustrating an exemplary method for reducing latency in a video pipeline according to some embodiments.

[0029] Figure 16 This is a flowchart illustrating an exemplary method for reducing latency in a video pipeline according to some embodiments.

[0030] Figure 17 This describes a frame comprising multiple subframes according to some embodiments.

[0031] Figure 18 This describes frames that have both decoded and displayed images.

[0032] Figure 19 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0033] Figure 20 This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0034] Figure 21This is a functional block diagram illustrating an apparatus including a video pipeline with reduced latency according to some embodiments.

[0035] Figure 22 This is a block diagram illustrating example components of a device according to some embodiments. Detailed Implementation

[0036] Some embodiments may employ techniques described in more detail below to provide low-latency video streaming. Such techniques may be used individually and / or in combination in various embodiments. For example only, some embodiments avoid or reduce frame buffering in the video pipeline. Some embodiments may employ partial frame processing (e.g., encoding and / or decoding), which can reduce processing time per frame and reduce and / or minimize latency. Some embodiments provide flexible output frame rates by allowing the video pipeline to decouple the output frame rate from the input frame rate and / or dynamically adjust the frame rate (including by using subframes); this can optimize performance and reduce and / or minimize latency.

[0037] Exemplary Examples

[0038] Certain exemplary embodiments are described below. As those skilled in the art will understand, each of the described embodiments can be implemented alone or in any combination. Therefore, a single embodiment or combination of embodiments should not be considered limiting.

[0039] Figure 1 This illustration includes an example block diagram of device 100, visually representing exemplary components of device 100 and their interactions to reduce display latency of video frames. In this example, device 100 is a set-top box (STB) or a component of a set-top box. As used herein, the term "set-top box" can include any device that is attached to or communicates with a television or other display device (e.g., via an HDMI connection, etc.) and provides video or other media to the display device; this can include STBs provided by cable or satellite television providers and other streaming media devices such as Apple TV. TM Roku TM Device, Amazon TM Firestick TM and / or similar.

[0040] In certain embodiments, device 100 may be an integrated component, such as a system-on-a-chip (SOC), which, by way of example only, may be integrated with or incorporated into an STB, display device, television, etc. Several embodiments use examples of STBs to describe the functionality of this device 100; however, the reader should understand that the described functionality is not limited to STB implementations, and different embodiments may include different devices 100 performing such functions as described herein as individual units or integrated units, including encoders and / or decoders (which may be used by media servers, video conferencing servers, game servers, etc.), STBs, televisions, computing devices, mobile devices, and / or the like.

[0041] Device 100 includes an input interface 102, which may contain any means or components capable of or required to receive media from video source 120, including (but not limited to) one or more physical interfaces, such as wired (local area network (LAN)) or wireless network interfaces (wireless local area network (WLAN), wireless wide area network (WWAN, cellular)), such as interfaces described in more detail below, wired and / or satellite television interfaces, multimedia (e.g., HDMI) interfaces, and / or the like. Input interface 102 may also contain any hardware, firmware, or software (collectively, "logic") required to receive and / or convert media received via such physical interfaces (including (but not limited to) receiving cable or satellite television signals, over-the-air ("OTT") television signals, and / or the like). As described in more detail below, the term "logic" is used broadly herein to describe any hardware circuitry, firmware instructions, software instructions, and / or processors implementing such instructions for providing the functionality described herein.

[0042] Video source 120 can vary depending on the specific application and the technology used. For example, in cloud gaming, video source 120 could be a cloud server that generates video frames and transmits them to device 100. In virtual reality, video source 120 could be a 3D rendering engine that creates and transmits virtual environments. In video conferencing, video source 120 is a user's webcam. Each participant's webcam captures their video and transmits it to the conferencing platform, which then transmits the video to device 100. In such instances, video source 120 could involve the generation or capture of video content, which is then transmitted to device 100 for display.

[0043] Device 100 also includes one or more output interfaces 104, which may include any means or components capable of or required to provide media output to video receiver 230. This may include (but is not limited to) one or more physical interfaces, such as wired or wireless network interfaces (e.g., interfaces described in more detail below), wired and / or satellite television interfaces, multimedia (e.g., HDMI) interfaces, and / or the like. Output interface 104 may also include any hardware, firmware, or software (collectively, logic) required to convert and / or transmit media received via such physical interfaces.

[0044] In some embodiments, device 100 includes memory 106, which can be used to store logic (e.g., instructions) that can be executed by one or more processors 108 to perform the various functions described herein. Examples of such memory 106 and processors 108 will be described in more detail below. In some embodiments, this logic may include an application program 110 (alternatively and / or additionally, which may be implemented as and generally described as logic 114, which is described in more detail herein), which can cause device 100 (and / or its components) to perform operations as described in more detail herein. Application program 110 can be used to capture and / or process video, as described in more detail herein.

[0045] In an example, logic 114 may include logic for receiving multiple encoded video frames from video source 120. In some embodiments, memory 106 may be used to store one or more frame buffers, including (but not limited to) a reference buffer, as described in more detail below. As used herein, the term “frame buffer” means an area of ​​memory (e.g., memory 106) that stores single-frame video data (i.e., pixel data (and / or other data, such as metadata) of a frame of a video sequence). A video pipeline may include several different types of buffers. One example is a capture buffer or network buffer, which holds frames received by device 100 before they are processed. Another example is a display buffer, which holds frames currently being displayed by a display device or being received by a display device or prepared for display, and is conventionally a double or triple buffer that operates in a ping-pong manner, as described in more detail below. Yet another example is a video processing buffer, which may store frame data of frames processed by various pipeline stages (e.g., decoder 114) before proceeding to the next stage of the pipeline or the output of device 100.

[0046] As used herein, the term “frame” means data representing an image (usually, but not necessarily, the data value of the entire display screen). Frames are typically characterized by a two-dimensional pixel measurement represented as width × height (resolution or frame), such as 1080 × 720 (High Definition or HD), 1920 × 1080 (Full High Definition or FHD), 2560 × 1440 (Quad High Definition or QHD), 3480 × 2160 (4K Ultra High Definition or UHD), 5120 × 2880 (5K UHD), 7680 × 4320 (8K UHD) and the like, but embodiments are not limited to frames of any particular size or resolution. As used herein, the term “partial frame” means any portion of a video frame smaller than the entire frame, while “full frame” is used to refer to the entire frame at the video resolution. The term “subframe” is used herein to refer to a specific type of partial frame: one of a group of partial frames that together form the full frame, such as those formed by… Figure 17 As explained and described below.

[0047] Multiple frames are collectively referred to herein as a “stream,” and a continuously viewed frame stream provides a series of “moving pictures” or videos. Typically, the frame stream is provided sequentially and continuously by the video source 120, but this is not necessary in every embodiment, and in some embodiments, the apparatus may reorder the frames of the received stream (e.g., by discarding one or more frames of the received stream) while processing the stream to provide one or more output streams of video frames to the video receiver 130. In such cases, the processed frames are referred to as “out of order” relative to the ordered stream of frames received by the apparatus 100.

[0048] Typically, video frames are encoded for transmission. As used herein, “encoding” means any operation or process that compresses or converts raw video data into a digital format that can be stored, transmitted, and played on various devices. Raw video data is typically represented as a sequence of pixel values ​​for each pixel in a frame matrix (e.g., 1080 columns × 1900 rows of pixels in an FHD frame), where pixel values ​​represent color and / or intensity (e.g., RGB, YCbCr, etc.), optionally with control or clock signals. An encoded frame of video is a raw video frame that has been encoded using one or more encoding operations or processes.

[0049] Encoding is typically performed by an encoder (or the encoding pipeline of a CODEC), which may contain logic (e.g., dedicated hardware, firmware, and / or software) for performing such processes and / or operations, including: compression to reduce the amount of data required to represent the video; format conversion to convert the raw video data into a specific format for playback on various devices, such as H.26X, MPEG, etc.; image quality adjustment; resolution and / or frame rate adjustment; audio encoding to perform similar processing on the audio accompanying the video stream; error correction; and / or adding metadata to the frames or stream itself. This metadata may typically include things such as subtitles, closed captions, chapter markers, and the like, and may also include metadata types discussed in more detail below. As is known in the art, encoding may include lossy compression, lossless compression, transform coding, and / or predictive coding. In some aspects, multiple encoded video frames may be encoded at a specific frame rate (typically measured in frames per second or (FPS)). In many cases, the video stream may be encoded and / or transmitted to device 100 at a fixed (constant) frame rate. In some respects, a fixed frame rate can impose an order on video frames, as mentioned above.

[0050] Logic 112 may also include logic for processing multiple video frames, for example, as described in more detail below. In an example, the logic for processing multiple video frames may include logic for decoding (e.g., a decoder represented by the decoding pipeline of CODEC 114) at least some of the multiple encoded video frames to produce multiple decoded video frames. CODEC 114 may include logic for encoding and / or decoding any suitable media format, including media encoded according to any of various standards (e.g., the MPEG family of standards, the H.26x standard, etc.). In some cases, the CODEC may be implemented as a separate encoder and decoder, while in others, the CODEC may include integrated logic for both encoding and decoding. As described in more detail below, some embodiments include a CODEC with parallel encoding and / or decoding pipelines, which may allow simultaneous encoding / decoding of multiple frames (or subframes, as described in more detail below). In some embodiments, an encoder may not be necessary, and CODEC 114 may be replaced by a decoder-only module. For example, if device 100 receives encoded video, then processing video frames may include decoding the received frames (or a portion of a frame, such as a partial frame or subframe). In other cases, processing video frames may include encoding or re-encoding decoded frames (or a portion of a frame) for transmission to a video receiver (e.g., via a network, etc.).

[0051] As used herein, the term "pipeline" is used broadly to refer to any series of operations or stages that video and / or audio content goes through from creation to delivery. In some embodiments, a video pipeline may encompass the entire workflow, from capturing or producing video to editing, processing, and finally distributing it to the desired platform or audience. In other embodiments, a pipeline may contain only a subset of these processes or stages. For example only, regarding... Figure 1 A video pipeline 150 may only include operations occurring within device 100 (e.g., operations ranging from receiving or capturing media (e.g., video frames) at input interface 102 to providing processed media (e.g., video frames) from output interface 104). As another example, a second video pipeline 150b may also cover operations or processes occurring at video source 120 and / or video receiver 130. Specific examples of video pipelines according to various embodiments will be described in more detail below.

[0052] In some embodiments, logic 114 may include logic for providing at least some of a plurality of video frames to video receiver 130 (e.g., via one or more of output interfaces 104). For example, if device 100 receives X video frames from video source 120, then device 100 may provide Y video frames to video receiver 130. In some cases, X and Y may be the same qualitatively and quantitatively. However, in other cases, X and Y may be different quantitatively. By way of example only, using the various techniques described below, some embodiments may receive X frames from video source 120 and provide Y frames to video receiver 130; in such cases, Y may be less than X, and in this way, some embodiments may particularly reduce frame latency. In other embodiments, device may receive frames with resolution X from video source 120 and provide frames with resolution Y to video receiver 130. In this way, some embodiments may particularly reduce frame latency. Based on these examples and the more fully described below, those skilled in the art will understand that various embodiments may reduce latency in video pipeline 150 in a variety of ways.

[0053] By way of example only, in some embodiments, the logic for providing at least some of a plurality of decoded video frames to video receiver 130 may include logic for providing at least some of a plurality of processed (e.g., decoded and possibly re-encoded) video frames to video receiver 130 via a High Definition Multimedia Interface (HDMI) and for providing at least some of a plurality of video frames via alternative paths separate from the HDMI connection (e.g., a second HDMI connection, a data network, a LAN interface, a WLAN interface, etc.). For example, if some frames or portions of frames are transmitted as different streams (examples of which will be described in more detail below), then some of these streams may be transmitted via HDMI, while others may be transmitted via LAN / WLAN, etc., and / or the video receiver may have the ability to combine or otherwise process these individual streams. According to various embodiments, logic 114 may include logic for reducing the display latency of each of the video frames provided to video receiver 130 by using one or more of various techniques (including, but not limited to, techniques for reducing display latency of video frames described in conjunction with the operation of the methods disclosed below) to reduce the number of frames in video pipeline 150. As used herein, the term “display delay” (or simply “delay”) means the amount of time required to display a video frame on a display device (monitor, TV, etc.), for example, as measured from the time it takes to display the previous frame of a video stream received by the display device (or, in the case of device 100 transmitting multiple streams, the time it takes to display the last frame of any stream provided by the device).

[0054] For example, Figure 2 The video pipeline 250 within the device 200 is described; the device 200 may be similar to the device 100 described above. However, Figure 2 This describes certain functional blocks arranged within pipeline 250, while Figure 1 The various components of the apparatus 100 are described for ease of description. These functional blocks typically correspond to... Figure 1 The various components of the device 100 are as described above. Figure 2 The right side of the diagram illustrates an exemplary functional block of pipeline 250, while the left side illustrates the process of multiple frames F1 to F10 passing through the pipeline. Specifically, the left side of the diagram shows frames F3 to F8n stored in the frame buffer between processing stages of pipeline 250, while F1 and F2 have exited pipeline 250 and been output to video receiver 230, and F9 and F10 have not yet entered pipeline 250.

[0055] Video pipeline 250 includes operations performed by device 100 on a stream of video frames (F1 to F10) from which they are received by device 200 from video source 220 and delivered to video receiver 230. Pipeline 250 operates in a conventional manner with the goal of smooth video. In the illustrated embodiment, as video frames (F) are received by device 200 from video source 220, the video frames (F) are ordered from bottom to top. Thus, in this example, frames F1 and F2 have passed through pipeline 250 and have been provided to video receiver 230 by HDMI output 204, wherein frames F1 and F2 are stored in a display buffer in video receiver 230 and / or displayed at video receiver 230. In such applications, the video receiver 230 will typically have two buffers (or a double buffer), shown here storing frames F1 and F2 in a "ping-pong" manner, where one buffer is displayed while the other is filled; for example, the first buffer may be filled with F1 (from the output 204 of device 200), and when F1 is displayed, the second buffer is filled with F2. Then, when F2 is displayed, the first buffer is filled with a third frame (which in this example would be F3). The same process typically occurs in the frame buffers within device 200 (shown as storing F3 and F4). It should be noted that in some cases, the display device may have multiple processing buffers (e.g., a triple buffer) for processing frames before displaying them.

[0056] As mentioned, frames F9 and F10 have not yet reached pipeline 250 because F9 is being transmitted from video source 220 and F10 is being encoded at video source 220 (or is otherwise waiting to be transmitted). As mentioned above, STB and OTT devices are designed to support smooth streaming video and anticipate: a fixed number of frames per second (FPS) video stream input, a fixed FPS for display at output, and a frame buffer at each pipeline stage. While the entire frame buffer is stored at each pipeline stage within the video pipeline (e.g., at encoding, decoding, video processing, HDMI input, and within video receiver 230), each stage waits until the full frame data is available at the input before being fed into the output stage. Although Figure 2 (and the discussion below) Figures 6 to 8 This describes the video pipeline, but it should be understood that the audio pipeline in device 200 will operate in a similar manner.

[0057] exist Figure 2In pipeline 250, device 100 stores several frames in a frame buffer to support smooth video playback, but this adds unnecessary latency, which is not optimal for low-latency applications. All data frames pass through this pipeline 250 regardless of the number of pixels that change between frames. At the HDMI interface 804, when playing games or participating in video conferencing, the FPS cannot change in real time with each frame. For example, for a 60 FPS configuration, the data value of one frame is sent every 1 / 60th of a second. If the next frame is not ready in time (underflow), the previous frame is repeated because the video receiver 230 (e.g., a TV) needs to obtain the data value of a frame from the HDMI output 204 every 1 / 60th of a second. Therefore, the pipeline buffers many frames (F3 to F8), thus increasing frame display latency.

[0058] For example, in Figure 2 As mentioned above, F3 and F4 are stored in the video processing frame buffer after processing stage 216 (as mentioned above, these buffers can be operated in a ping-pong manner, such that the first buffer is filled with F3 after processing stage and then provided to video receiver 230 through output interface 204, while the second buffer is filled with F4 and then provided through output 204, while the first buffer is filled with F5 (after processing F5 at processing stage 216).

[0059] exist Figure 2 In this process, frames F5 and F6 to F6n have been decoded by decoder 212 and stored in the frame buffer awaiting processing by stage 216, while frames F7 to F8n are stored in the compressed frame buffer in the queue of decoder 212 after being received by application 210. Frames F8 to F8n (and frames F6 to F6n) may represent multiple frames that are processed and / or buffered by application 210 and decoder 212 respectively to account for network latency (as further described below).

[0060] Therefore, in Figure 2 In this example, there is a 10-frame delay between F1 displayed at the video receiver and F10 ready for transmission (not counting any additional buffering of frames F6a to F6n and F8a to F8n by application 212 or decoder 212). Figure 3 Display frames F1 to F10 (and) Figure 2 The timeline 300 shows the two additional frames F11 and F12 that are not depicted above. Figure 3 This describes the frames processed by each module and the storage in each frame buffer during the duration shown by the time slice on the X-axis (i.e., 16.66 milliseconds at 60 FPS). Video receivers (e.g., TVs) require the full frame time to display each time slice, and the bottom line indicates the time slice in which a frame is active on the display. For example, F1 is displayed until 16.66 milliseconds, F2 is displayed from 16.66 to 33.33 milliseconds, and so on. Figure 3 As illustrated above with respect to the frame size relative to the time slice, various pipeline stages (e.g., processing modules) can operate faster than this fixed frame rate, so the processing time for each frame does not occupy the entire 16.66 millisecond time slice. At each time slice, each module in the pipeline begins processing the next frame; for example, from 0 to 16.66 milliseconds, the decoder decodes F6, and at 16.66 milliseconds, the decoder begins decoding F7. For simplicity, Figure 3 This ignores some real-world factors, such as network jitter and application processing time.

[0061] As mentioned above and by Figure 3 The demonstration showed a latency of approximately 10 frames (166.66 ms) from the video source to the display. Return to Figure 2 In practical implementations, to mitigate the effects of jitter and support smooth video, application 210 typically stores multiple frame buffers, which... Figure 2 The data is shown as F8, F8a…F8n (which can be frame data of 30 seconds or longer). Similarly, decoder 212 can have multiple frame stores in the pipeline, as shown by F6, F6a…F6n. This latency helps ensure smooth video, but does not help with low-latency applications. Various embodiments provide different techniques to reduce this latency, as described in more detail below.

[0062] For example, Figure 4 A method 400 for reducing latency in providing video frames to a video receiver is described. In some embodiments, method 400 includes receiving multiple video frames from a video source, for example, using a device 100 with an STB, a video encoder at a content provider, etc. (block 402). Receiving multiple video frames may include various operations, some of which will be described in more detail below. However, in general, "receiving" one or more video frames includes, for example, obtaining, receiving (in the conventional sense), accepting, ingesting, or otherwise making video frames available to device 100 for processing (e.g., described in more detail below) and / or distributing to video source 130. As mentioned above, some embodiments may receive video frames from any of a variety of video sources.

[0063] In some embodiments, method 400 includes processing a plurality of video frames (block 404). In one aspect, processing at least some of the plurality of video frames may produce a plurality of processed video frames, for example, as described in more detail below.

[0064] Several operations (including, but not limited to, those described in more detail below) may be considered as processing one or more video frames. Generally, in the context of video frames, the term "processing" is used herein to describe any operation that encodes, decodes, creates, or otherwise modifies a video frame and / or removes, adds, modifies, or otherwise alters one or more video frames in a video frame stream. As used herein, the term "processing multiple video frames" may include processing one or more of a plurality of video frames and does not require processing every single frame of the plurality, unless the context clearly indicates otherwise.

[0065] In certain aspects of some embodiments, processing multiple video frames (e.g., one or more of a plurality of video frames) can reduce the display latency of one video frame, some video frames, or all of the plurality of video frames (box 406), for example, by using any combination of one or more techniques described in more detail below. By way of example only, in some embodiments, processing multiple video frames can reduce the display latency of each of the video frames provided to a video receiver, for example, by reducing the number of frames in the video pipeline including means (e.g., means 100).

[0066] The following describes several techniques for reducing display latency. In some embodiments, any or all such techniques may be selectively disabled based on configuration settings (e.g., settings that specify when latency should be reduced and which techniques to use based on resource availability, device load, content type, etc.) and / or user controls (e.g., a user configuring the device’s behavior at any given time based on the user’s preferences). For example, some features may be disabled depending on the nature of the application associated with the video stream (e.g., when watching video content that prioritizes smooth video playback).

[0067] In some embodiments, method 400 includes providing at least some of a plurality of video frames to video source 130 (block 410). Various technologies and / or interfaces may be used to provide video frames to video source 130, including (but not limited to) those described in more detail below. By way of example only, providing video frames to video source 130 may include transmitting video frames via one or more output interfaces 104 of device 100, such as dedicated multimedia interfaces (e.g., HDMI interfaces), alternative paths (e.g., LAN (or WLAN) interfaces), and / or the like. In some cases, multiple technologies or interfaces may be used to provide multiple video frames to video source 130; for example, some of the multiple video frames (some video frames in the frame stream) may be provided via an HDMI interface, while other video frames in the stream may be provided via a LAN / WLAN interface.

[0068] Figure 5Method 500 is described, which includes several techniques for reducing display latency, such as reducing the number of frames in the display pipeline (e.g., within the device of, for example, device 100), which reduces the latency of video frames provided to the video receiver. For example only, see reference to... Figure 2 Receiving multiple video frames may include, for example, receiving multiple encoded video frames F1 to F10 from video source 220 at device 200. In some embodiments, the multiple encoded video frames received from video source 220 may be fixed frame rate encoded, and in some cases, an order is imposed on the received video frames. For example, return to reference Figure 3 Frames F1 to F10 are based on... Figure 3 The fixed frame rate described is received by device 200 in this order (F1, F2…F10), and they typically want to display the video in this order according to the fixed frame rate to provide smooth video, such as… Figure 2 As described in the context.

[0069] In some embodiments, method 500 includes decoding at least some of a plurality of encoded video frames to generate a plurality of decoded video frames (block 502). For example, as mentioned above, exemplary device 100 may include CODEC 112 (which can act as...). Figure 2 The decoder 212); if the received video frames are encoded (e.g., encoded as MPEG or H.26x video streams), then the device 100 can decode these video frames using CODEC 112, which allows for further processing of the video frames, for example, as described in more detail below. Figure 3 As explained in the context, many operations can be performed by the device 200 at a rate faster than required by a fixed frame rate, and one example of such operations is the decoding stage; in some embodiments, multiple encoded video frames can be decoded at a rate faster than a fixed frame rate.

[0070] In some embodiments, processing video frames may include reducing the number of frames stored in one or more buffers in the video pipeline (box 504). By way of example only, as mentioned above, for smooth video playback, it is desirable to play each frame, and the decoder 212 can process frames faster than the frame rate of the input frames (i.e., in real time). Therefore, most of the time, the video pipeline 250 is full, and multiple frames are buffered in transit from the video source 220 (e.g., a server, cable head, etc.) to the video receiver 230 (e.g., a TV screen). On the other hand, for low-latency use cases, it is desirable to play the latest video frame from the video source 220 as quickly as possible. Therefore, whenever multiple frames are available in the pipeline, some embodiments display the latest available frame and discard earlier decoded frames. For example, Figure 6 Description of device 600 with video pipeline 650. Comparison. Figure 2 and Figure 6Such embodiments may discard multiple frames (including (but not limited to) any frames (F6, F6a, F8, F8a, F8b, etc.)) already buffered by application 210 and / or decoder 220 to mitigate network jitter. As described in more detail below, additional frames can be discarded in some embodiments. In this way, frames can be processed by pipeline 250 and served to video receiver 230 more quickly.

[0071] For example, in the case of video streaming, frame buffers are typically needed to store frames for video processing and / or enhancement (e.g., scaling, cropping, etc.) because the same video stream is sent (broadcast) to multiple devices (e.g., 200). On the other hand, for low-latency use cases, each client (e.g., device 200) typically receives its own stream from video source 220 (e.g., a game server, video conferencing server, etc.) and can satisfy all scaling or cropping requirements at video source 220 before encoding the stream. Therefore, partial or complete local post-processing can be eliminated, and latency of more than two frames can be avoided. This is due to… Figure 6 This indicates that the post-processing buffer has been removed from video pipeline 650, and frames F3 and F4 have been discarded. Therefore, latency has been reduced by two frames (which can be appended to any frame for network jitter buffering, as mentioned above).

[0072] Therefore, in some embodiments, method 500 may include determining that a plurality of available decoded video frames (e.g., F3 to F6n) exist in video pipeline 250 (box 506) and / or identifying the most recently available decoded video frame (box 508). Figure 6 In this case, the most recently decoded frame available is F5, because F6 is still being used to fill the ping-pong buffer in device 600. The remaining video frames in the pipeline (box 510) can be discarded. In this case, these frames are F3 and F4, because F6, F6a… were not buffered by the decoder, as mentioned above. Reference Figure 4 Providing at least some video frames (box 410) may therefore include providing the most recently available decoded video frame. In one aspect, this frame may be provided out of order at any fixed frame rate that encodes multiple frames, for example, providing F5 without providing F3 and F4 first. Therefore, discarding a buffered frame before the most recently available frame may cause this frame to be delivered at a time earlier than the time specified by the fixed frame rate of the encoded stream received by device 600, thereby reducing the display latency of this frame. Similarly, providing at least some video frames may include providing multiple decoded video frames to a video receiver without storing any of the multiple decoded video frames in a video processing frame buffer, as provided by [other device / receiver]. Figure 6 As shown. In some embodiments, at least some of the plurality of decoded video frames may be provided to video receiver 630 via HDMI.

[0073] In a further embodiment, providing at least some video frames may include using a single-frame display buffer to provide multiple decoded video frames. For low-latency applications, the IP* frame structure (where an I-frame is followed by a specified number of P-frames, such as IPPP or IPPPPPPPPPP) is typically used for compression, and the decoding and display order is the same. A low-latency B structure (IB*BBBB) that also includes B-frames can also be used, while maintaining the same decoding and display order, but the prediction block may reference two previous frames. In such applications, as mentioned above, the video receiver 230 (e.g., a display device) will typically have two buffers (e.g., as described above and in...) Figure 2 The buffer contains storage for F1 and F2, which operate in a "ping-pong" manner. According to some embodiments, this ping-pong structure can be eliminated. For example, for low-latency applications, it is acceptable for a video receiver to display the first X lines from frame Fm and the remaining lines from frame F(m+1) or F(m-1) without using two buffers. Figure 7 The device 700 is described, in which the ping-pong buffer has been eliminated, thereby allowing frame F5 to be discarded to save latency for another frame. In some embodiments, it is also as follows... Figure 7 The display also shows that the input buffer can be eliminated in the video receiver 730 and frame F2 can be discarded.

[0074] It should be understood that eliminating one of the display buffers at the video receiver 730 can result in frame tearing, where a portion of the displayed image appears to be horizontally split from another portion of the displayed image. In some use cases (e.g., in games), this is acceptable, and method 500 therefore includes, in some embodiments, allowing frame tearing when displaying multiple decoded video frames (box 512). In one aspect of some embodiments, device 700 may selectively allow frame tearing at the video receiver 730 based on user input and / or characteristics of the application (e.g., a specific game where frame tearing is not destructive), and the method according to such embodiments may therefore include determining whether frame tearing should be allowed based on such factors.

[0075] In many cases, a video decoder may need to store multiple frames as references to future frames; in such cases, it may be important to avoid data from subsequent frames overwriting the data of the reference frames in the frame buffer. Traditionally, the same frame buffer is often used to feed frames into the output display queue and as a reference buffer. For display pipelines (e.g., those with a single video processing frame buffer at the output of decoder 712, as in pipeline 750) or without a video processing frame buffer (as in pipeline 850, discussed in more detail below), this approach may be necessary. Figure 7 and 8In some embodiments, the corresponding pipelines 750 and 850 may provide the ability to store reference frames via a separate reference buffer 708. In such embodiments, the decoder 712 may maintain separate frame queues for internal reference and display. For example, each decoded frame data (e.g., F1, F6n) may be written simultaneously to both the separate reference queue 708 and the display pipeline (e.g., the frame buffer at the decoder's output). Figure 7 (This is illustrated as a stored frame F6n). Therefore, in some embodiments, method 500 includes storing a plurality of reference frames in a reference buffer located outside the video pipeline and separate from the display buffer of the video pipeline (box 512). Furthermore, in some embodiments, method 500 includes generating a video frame from one or more of the reference frames stored in the reference buffer (box 514). This operation can be performed using techniques known in the art (e.g., generating full frames from I-frames and P-frames) but using one or more frames stored in the reference buffer instead of frames stored in the frame buffer, which is part of the video pipeline 750.

[0076] In a further embodiment, such as by Figure 8 As demonstrated, the video processing frame buffer can be completely eliminated from the video pipeline 850 while still providing processed video frames via the HDMI interface 804. It should be understood that the HDMI output to the video receiver 830 typically needs to have a fixed data rate; for example, the data value of a single frame can be sent per time slice via the HDMI output 804 (e.g., as combined with...). Figure 3(As described above). As previously mentioned, in some embodiments, decoder 812 may process frames at a faster rate than this. In such embodiments, as the decoder begins decoding a frame, the frame may be routed to HDMI output interface 704, which may control the pull rate from decoder 812 to match the transmission rate of the output interface (e.g., a time slice). Therefore, in some embodiments, method 500 may include providing a decoded portion of the encoded video frame to HDMI output interface 804 (block 516) before the entire encoded video frame is fully decoded. For example, because the decoder operates on macroblock units (e.g., according to MPEG-2 and H.264 standards) or other encoding units of various sizes (e.g., 16×16 pixels, 128×128 pixels, etc.) (e.g., according to H.265 and H.266 standards), decoder 812 may decode between approximately 16 and 128 lines simultaneously. Therefore, up to approximately 128 lines can be obtained from decoder 812 before HDMI output 804 can begin feeding frames into video receiver 830, in which case frame buffer latency can be reduced to a maximum of 128 lines from decoder 812. Thus, in some embodiments, method 500 includes providing a decoded portion of the encoded video frame to an output (e.g., the HDMI output of the device) before the entire encoded video frame is fully decoded (block 516). The decoded portion of the encoded video frame includes one or more macroblock units or one or more coding units.

[0077] In some use cases (e.g., gaming, virtual reality, etc.), the entire displayed image in a video stream does not constantly change; instead, most of the displayed image can be largely static, and only a portion of the image needs to be added, such as adding or moving small features, like an enemy player in a game, some text, pop-up messages, etc. In such cases, latency can be significantly reduced by refreshing a small portion of one or more video frames instead of the entire frame. In such embodiments, once the correct location of a particular area is determined, the apparatus according to some embodiments can refresh that area without refreshing the entire frame.

[0078] In some embodiments, an encoder (which may be located at video source 120 or within device 100) may encode smaller resolution frames (referred to herein as “partial video frames” or “partial frames”) and transmit the encoded partial frames to another device, video receiver, etc., faster than encoding and transmitting the entire frame. In some cases, a partial frame may contain several pixels extracted from the entire video frame. In some cases, a partial frame may contain (or be accompanied by) metadata, such as location information indicating the position (e.g., position coordinates) of the partial frame to be updated within the full frame, the identifier of the full frame, timing and / or synchronization information, etc.

[0079] Similarly, at the decoder, partial frames can typically be decoded faster than full frames and therefore processed and served to the video receiver more quickly. For example, if a partial frame is 1 / 20th the size of the full frame, it can be decoded in roughly proportionally less time (e.g., 1 ms instead of 16 ms). Therefore, in some embodiments, the partial frame is generated by, for example, a device 100 and / or served to the video receiver (e.g., in…) Figure 4 The processed video frames (at frame 410) may include one or more processed partial video frames, and in such embodiments, such partial frames may be used to reduce the display latency of at least some video frames (e.g., in...). Figure 4 (at frame 406). Figure 9 The description includes a method 900 for various techniques, such as using partial video frames, to reduce display latency. In some embodiments, these techniques may be implemented as complementary to and / or separate from, for example, the techniques described above.

[0080] In some embodiments, processing the received frames may include, for example, decoding one or more received video frames using a decoder or CODEC (box 902). In some cases, the encoded frames received from the video source may include one or more partial frames. Examples of this use case are provided by Figure 10 The device 1000 supports a video pipeline 1050 (shown in a simplified form) including a decoder 1012. A video source 1020 encodes and transmits a first video frame F1, which is a full video frame (e.g., 1080p, 4K, etc.). The decoder 1012 decodes F1 and provides it to a video receiver 1030 (e.g., via HDMI), where F1 is displayed. The next frame F2 is largely the same image as F1, except that a portion of the frame, F2p, has been changed. The video source 1020 encodes only F2p and provides it to the device 1000, where the decoder 1012 decodes F2p and provides the decoded portion of the frame F2p to the video receiver (box 906). This arrangement reduces the display latency of F1+2p compared to a configuration where the entire F2 frame is decoded by the device 1000 and provided to the video receiver 1030. In some embodiments, such as by Figure 11As depicted, method 900 may include creating (e.g., via device 1100 or its components, such as logic 1114) one or more partial video frames (box 904) from a full frame received from video source 1120. This may be performed, for example, by storing F1 in a reference buffer after decoding and then comparing the decoded F2 frame with F1 to determine that only a portion of the frame has changed. In either case, when sending a partial frame to a video receiver, the device may provide, for example, the metadata discussed above (e.g., using Supplemental Enhancement Information (SEI) messages) indicating that F2p is a partial frame update of F1 (i.e., updating F1 to F2), the position of F2p within F2, and / or synchronization data to synchronize the timing of the partial frame update (F2p) with the full frame (F1). This metadata may be received from video source 1120 (e.g., via a full frame received from video source 1120, such as by device 1100 or its components, such as logic 1114). Figure 10 The illustrated embodiments utilize F2p) and / or can be created by the device 1100 itself (e.g., in the case of...). Figure 11 (As illustrated in the embodiments).

[0081] In some cases, the device's output interface (e.g., an HDMI output interface) may not support providing partial frame updates, or the video receiver may be unable to display partial frame updates. In these cases, method 900 may include generating a composite frame from one or more full video frames and / or partial video frames; this composite frame can be provided to the video receiver, for example, as a full frame update. Figure 12 This illustrates an example of this embodiment. For example, in Figure 12 In the process, video source 1220 first encodes and transmits F1, and device 1200 decodes F1 and provides it to video receiver 1230. In some embodiments, device 1200 also stores F1 in a reference buffer (described above, but for simplicity...) Figure 12 (not shown in block 908). Video source 1220 then encodes and transmits F2p decoded by transmission device 1200. In some embodiments, the method includes receiving metadata from the video source (containing, for example, positional information about the relative position of F2p within F1) (block 910). In other embodiments, the device may include logic for calculating the position of F2p within F1 (e.g., based on metadata). Logic 1214 obtains F1 from a reference buffer and updates F1 with F2p based at least in part on this metadata to produce a composite frame (F2) (block 912), which device 1200 then provides to video receiver 1230.

[0082] In some embodiments, the device may receive partial frame data via a separate graphics path. Figure 13Examples of such embodiments are illustrated, wherein the video source encodes F2p into graphics and provides F2p via a path independent of F1 and / or a graphics path that is not part of the same stream as F1 (e.g., a separate network connection, route, etc.). Logic 1314 (e.g., a compositer) in device 1300 mixes the graphics (F2p) with the video frames, as mentioned above, which can be provided to the video receiver ( Figure 13 (Not shown above) and stored in the reference buffer ( Figure 13 (Not shown above) to generate composite frames, which are also provided to the video receiver. In some embodiments, text and / or graphic elements may be pre-generated and stored in the device 1300. In this case, the video source may only send metadata, such as updates on the position and / or timing information of the text and / or graphics, which may allow logic 1314 to generate composite frames at appropriate locations in the video stream from previous frames, stored graphic / text elements, and position information.

[0083] In some embodiments, adaptive resolution coding tools can be used to encode portions of frames, such as by resampling from reference frames, for example, those supported by H.265. Using such tools, consecutive frames of a video sequence do not need to have the same size, and prediction between reference frames of different sizes is feasible. Therefore, in the case of, for example… Figures 10 to 13 In the exemplary embodiment, F1 can be the reference frame, while F2p can be coded at a smaller resolution and predicted from the corresponding (smaller) region of the previous frame F1 (e.g., using inter-frame coding / motion compensation). Conversely, if F2 changes significantly relative to F1, then intra-frame coding can be used instead (without any prediction from F1).

[0084] In some embodiments, partial frame updates can be performed sequentially in the manner described above (e.g., as by...). Figure 14 As illustrated in F3p, F4p, etc., a full frame update may be appropriate until any delay-critical events have passed. By using multiple reference frames and storing at least one full-size frame (frame F1) in the decoder's reference frame storage, inter-frame / motion compensation can be used to efficiently encode subsequent full frames (e.g., F5), and this frame F5 can be decoded using F1 to produce a second decoded full frame without having to completely restart the decoder (e.g., without having to start from an I-frame again). As mentioned above, in some embodiments, metadata (e.g., SEI messages) may be sent with each of these frames to indicate that this is in progress and to indicate where the decoder should display / overlay this smaller frame.

[0085] Figure 15Method 1500, whose operation can be performed by means such as the apparatus 100 described above, includes several techniques for reducing display latency, such as decoupling the frame rate of video frames received from video source 120 from the frame rate of video frames provided to video receiver 130. By way of example only, receiving multiple video frames may include receiving multiple subframes from video source 120. In some cases, apparatus 100 may receive each of the subframes as a separate stream. In some cases, apparatus 100 may receive multiple subframes as a single stream. In some embodiments, receiving multiple video frames may include receiving the entire video frame.

[0086] Method 1500 may include receiving a plurality of video frames at a fixed frame rate at a device such as device 100 (block 1502). In some aspects, the video frames may be received from a video source such as video source 120. In block 1504, method 1500 includes processing at least some of the plurality of video frames to produce a plurality of processed video frames. In some embodiments, processing the video frames may decouple the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames, for example using the following combination. Figure 16 And various techniques described elsewhere herein. As used herein, the term “decoupling” may include any technique that changes the frame rate of the video stream provided to the video receiver from the frame rate of the video stream received from the video source; the frame rate of the received stream may be fixed or variable, and the video stream provided to the video receiver may also be fixed or variable. In block 1506, method 1500 includes providing each of a plurality of processed video frames to the video receiver at a flexible frame rate. In some embodiments, providing video frames to the video receiver may include providing a first set of a plurality of decoded video frames to the video receiver via a dedicated multimedia interface and a second set of a plurality of decoded video frames to the video receiver via a local area network.

[0087] According to various embodiments, processing a received video frame may include several operations, some of which are combined. Figure 16 Description. It should be understood that many of these operations can be performed independently of each other, but they can also be performed as part of the same process, such as in a single video pipeline.

[0088] For example, Figure 16Method 1600 for processing video frames is described, which can be used to decouple a flexible frame rate of a processed video frame from a fixed frame rate of a received frame. In some cases, the received video frame may not be encoded. Therefore, in block 1602, method 1600 may include encoding at least some of a plurality of video frames with a CODEC (e.g., MPEG CODEC, H.26x CODEC, etc.) for transmission over a network. In some embodiments, the CODEC encodes the frame at a flexible frame rate. In other embodiments, the received video frame may have been encoded at a fixed frame rate by a video CODEC. Method 1600 may therefore include decoding at least some of the plurality of video frames to produce a plurality of decoded video frames (block 1604).

[0089] More specifically, in some cases, one or more of the received video frames may include multiple subframes. In certain aspects, Figure 17 The description includes video frame F1 comprising multiple subframes SF1 to SF16, which will be used in the discussion of various examples below. It should be understood that, according to various embodiments, the full frame can be divided into any number of subframes (e.g., 2 subframes, 4 subframes, 8 subframes, 32 or 64 subframes, etc.) and the subframes may (but are not required to) be equal (or substantially equal) in terms of data size, resolution, aspect ratio, etc. In some embodiments, the apparatus may receive the full frame and divide it into subframes for processing (e.g., decoding, etc.). In other cases, the apparatus may receive each subframe individually, for example, as part of a single frame stream or as part of multiple streams. This is merely an example. Figure 17 Subframes SF1 to SF16 of frame F1 may be received in a single stream as a group of consecutive subframes in this order (or another order), or as separate streams each comprising a subset of subframes (e.g., a first stream with SF1, SF5, SF9, SF13…, a second stream with SF2, SF6, SF10, SF14… etc.), and / or as part of a separate stream (e.g., a first stream with SF1, a second stream with SF2… a fifth stream with SF5… etc.). In any case, method 1600 may include processing at least some subframes of the encoded video frame (box 1606), and such processing operations may include any video pipeline operations described herein (including, but not limited to, decoding subframes, for example, as discussed in more detail below).

[0090] In some cases, the apparatus may process not all subframes of the encoded video frame. For example, apparatus 100 may process only SF1, SF2, SF5, and SF6, ignoring (or discarding) the remaining subframes. (In some aspects, this technique can be used to decode partial video frames, for example, as described above.) In such cases, apparatus 100 may provide only the processed subframes or even a subset of one or more processed subframes to the video receiver. In certain embodiments, apparatus 100 may provide each of the processed subframes as part of a different video stream and / or may provide all (or a subset of) the processed subframes as part of the same video stream. This provides considerable flexibility in delivering video frames to video receiver 130.

[0091] In some cases, the entire frame can be considered a decoded frame because the entire frame is received and decoded by device 100. A portion of a frame can be considered a displayed frame because that portion of the frame is the content provided to the display receiver. In some embodiments, the displayed frame can move between frames along with the decoded frame. Although this embodiment is not limited to the subframe context, Figure 17 Useful explanations are provided for discussing this feature. For example, the display of a frame may include SF1, SF2, SF5, and SF6, and for this frame, only these subframes will be transmitted to the video receiver 130. However, in the next frame, the display may include SF3, SF4, SF7, and SF8, and only these subframes will be transmitted to the video receiver 130. (Of course, the display may remain unchanged for subsequent frames.)

[0092] Figure 18 A similar principle is illustrated, wherein the first frame F1 includes a decoded screen 1805 and a display screen 1810a, with the display screen 1810a occupying a portion of the decoded screen. The display screen 1810a may contain a portion of frame 1815a, which may be a part of the display screen 1810a. In the next frame F2, the display screen 1810b has moved and now occupies a second portion of the decoded screen, and a portion of frame 1815b may be present (or may not be present), which may occupy the same or different portions of the display screen 1810b. Similarly, in frame F3, the display screen 1810c may occupy the same or, as described, different portions of the decoded screen 1805, and a portion of frame 1815c (if present) may occupy the same or different portions of the display screen 1810c. (In some embodiments, as in...) Figure 17 In this context, subframe groups can be considered as displayed frames, and subframe subgroups or even individual subframes can be considered as partial frames.

[0093] In some cases, this allows the entire frame (decoded frame) to be transmitted by the video source 120 at a relatively slow frame rate (possibly 10 to 20 fps), while the display frame is processed and delivered to the video receiver at a much faster rate (possibly 60 to 120 fps). In some embodiments, partial frames can be processed and delivered to the receiver at an even faster rate (e.g., the fastest rate supported by the hardware). This is particularly useful when some parts of the decoded frame are more dynamic than others (i.e., have more inter-frame variations), allowing processing and transmission resources to be focused where they are most helpful. And as different parts of the decoded frame become more dynamic, the display frame (and / or partial frames) x can move with the decoded frame to provide a higher frame rate for the more dynamic parts of the decoded frame at any given point in the stream.

[0094] Different streams can be delivered in different ways; for example, one stream may be provided by HDMI and another by, for example, a LAN / WLAN connection. To illustrate some examples of subframe delivery across various stream combinations, Figure 19 Explanation of device 1900, wherein the subframes of the first frame F1 (labeled F1-SF1 to F1-SF16) are in video pipeline 1950 (for illustrative purposes, they are in...). Figure 18 The process is simplified, but may include various other processing levels, including (but not limited to) the processing levels described elsewhere herein, where four decoders 1912 process the subframes in parallel. As shown, the first decoder 1912a decodes subframes F1-SF1, F1-SF5, F1-SF9, and F1-SF13, while the other decoders 1912b to 1912d process the other subframes of F1 in parallel with decoder 1912a in a similar manner as shown. Figure 19 In this embodiment, device 1900 (and video pipeline 1950) also includes two multiplexers / demultiplexers 1960, which can be used to distribute subframes of F1 to decoder 1912 and aggregate decoded subframes for output, for example, via HDMI interface 1904, to deliver the subframes as part of a single stream to video receiver 1930. When the next frame F2 is received, subframes F2-SF1 to F2-SF16 can be processed in a similar manner. (The multiplexer / demultiplexer 1960 is actually illustrative for purposes of illustration, and any suitable type of hardware, firmware, and / or software logic can be used to perform the distribution / aggregation functionality.) Thus, a single stream provided by device 1900 may include F1-SF1, F1-SF2…F2-SF15, F2-SF16… Such multiplexing and / or demultiplexing operations can be considered as aspects of processing video frames, as this term is used herein.

[0095] Furthermore, if one or more subframes (e.g., SF9) remain unchanged between F1 and F2, then device 1900 may not decode, process, and / or transmit F2-SF9 to video receiver 1930, thereby allowing video receiver 1930 to continue displaying F1-SF2, while simultaneously displaying F2-SF1 to 8 and F2-SF10 to 16. Similarly, if SF9 is the only frame that changes between F1 and F2, then it can be the only subframe in F2 transmitted to and displayed by video receiver 1930. This technique can be used to limit the processing and / or transmission of any subset of SF1 to SF16 in each frame of the stream. This can reduce latency by reducing processing time in device 1900 and / or video receiver 1930. This technique can be used to re-display any or all subframes that remain unchanged between consecutive frames (or non-consecutive frames, such as where buffered frames are discarded). In this way, each subframe can have a different frame rate, which depends on when and / or how often each subframe changes in each frame of the stream transmitted to the video receiver 1930.

[0096] Figure 20 The exemplary device 2000 operates in a similar manner, except that subframes output from each decoder are transmitted as separate streams to the video receiver 2030; in other words, device 2000 will output four separate streams, each having subframes of consecutive (or non-consecutive) frames, such as a first stream with F1-SF1, F1-SF5…, a second stream with F2-SF9, F2-SF13…, etc. The streams can be output using any suitable technology or combination of technologies, such as one or more HDMI connections, LAN / WLAN connections, etc. For example, one stream may be provided to the video receiver 130 via a first HDMI connection, while the second stream is provided by a second HDMI connection, and the third and fourth streams are transmitted via WLAN. Those skilled in the art will understand based on this disclosure that any combination of transmission technologies may be employed in different embodiments. In some cases, a suitable multimedia interface (e.g., a multi-stream HDMI connection) may be used.

[0097] In some embodiments, each subframe may be delivered as a separate stream. Figure 21Another exemplary apparatus 2100 is described. In this apparatus 2100, four decoders 2112 provide their outputs to a multiplexer / demultiplexer 2160b, which transmits 16 streams (corresponding to F1-SF1 to F1-SF16) as portions of 16 separate streams to a video receiver 2130, such as F1-SF1, F2-SF1, etc. Those skilled in the art will understand from these examples that, depending on the different embodiments, any number of decoders can be used to provide parallel decoding. In some cases, a single decoder is capable of decoding multiple subframes in parallel. Similarly, while these examples depict frames with 16 subframes, it should be understood that different embodiments can support any number of subframes. Likewise, various embodiments can provide any number of streams as outputs to the video receiver.

[0098] In some cases, the method may include generating composite frames (e.g. Figure 17 F1), which includes some or all of the subframes of this frame (e.g. Figure 17 (SF1 to SF16) (box 1608), regardless of whether the device 100 receives the frame as a set of subframes or receives the frame as a full frame and subsequently divides the frame into subframes. In some embodiments, the device 100 may include dedicated logic 114 and / or application 110 to divide the frame into subframes and / or generate a composite full frame from some or all of the subframes. This division and compositing operation can be considered as an aspect of processing frames, as used herein. In some embodiments, the composite frame is not a full frame, but may be a partial frame that can be used to update a previous full frame, for example, as described in more detail above.

[0099] As mentioned above, in some embodiments, device 100 can process frames faster than a typical fixed display frame rate. This is particularly true for embodiments that decode and / or otherwise process subframes in parallel. Thus, if frames are received at a fixed frame rate of 60 fps, device 100 can decode / process subframes within a fraction of that time (possibly 600 fps), where a parallel pipeline is used for decoding and / or otherwise processing the subframes. These frames can be processed at their rate and delivered to the video receiver (e.g., using any of the techniques described herein), in some cases without any post-processing buffer (e.g., using any of the techniques described herein).

[0100] While this disclosure provides numerous examples of devices (e.g., 100, 200, 600…2100) and video pipelines (e.g., 150, 250, 650…2150), it should be understood that various embodiments may combine any or all of the features of these devices, and in some aspects, all devices and pipelines may be considered to illustrate different functionalities of the same embodiment. Similarly, while many operations are described in conjunction with methods 400, 500, 900, 1500, and 1600, any and / or these operations may be combined in various embodiments, and in some aspects, in particular embodiments, all these operations may be considered to be combined into a single method.

[0101] Demonstration device

[0102] Figure 22 This is a block diagram illustrating an example of device 2200, which, according to an embodiment, can be used as an STB or any other device described above (including (but not limited to) devices 100, 200, 600…2100) and / or can be used to perform some or all of the operations described herein. Figure 22 The components shown should not be considered necessary or required for every embodiment. For example, many embodiments may not include a processor and / or may be fully implemented in the hardware or firmware circuitry. Similarly, many embodiments may not include input devices, output devices, or network interfaces.

[0103] After the prelude, as Figure 22 As shown, device 2200 may include bus 2205. Bus 2205 may include one or more components that enable wired and / or wireless communication between components of device 2200. Bus 2205 can... Figure 22 Two or more components are coupled together, for example via operational coupling, communication coupling, electronic coupling, and / or electrical coupling. Such components may include a processor 2210, a non-volatile storage device 2215, working memory (e.g., system dynamic random access memory (DRAM)) 2220, and / or circuitry 2225. In some cases, system 2200 may include a human-machine interface component 2230 and / or a communication interface 2235.

[0104] Although these components are shown as being integrated within device 2200, some components may be located outside device 2200. Thus, in addition to or apart from the components themselves, device 2200 may also include facilities for communicating with such external devices, which in some embodiments may therefore be considered part of device 2200.

[0105] By way of example only, non-volatile storage device 2215 may include hard disk drives (HDDs), solid-state drives (SSDs), and / or any other form of persistent storage device (i.e., storage devices that do not require power to maintain the state of stored data). While this storage device is typically incorporated within device 2200 itself, it may be external to device 2200 and may include external HDDs, SSDs, flash drives, or the like, as well as network storage devices (e.g., shared storage devices on a file server), storage area networks (SANs), cloud storage devices, and / or the like. Unless the context otherwise indicates, any such storage device may be considered part of device 2200 according to various embodiments. In one aspect, storage device 2215 may be non-transitory.

[0106] Similarly, the human-machine interface 2230 may include input components 2240 and / or output components 2245, which may be located within, outside, and / or combinations thereof in the device 2200. Input components 2240 enable the device 2200 to receive input, such as user input and / or sensed input. For example, input components 2240 may include a touchscreen, keyboard, keypad, mouse, buttons, microphone, switch, sensor, GPS sensor, accelerometer, gyroscope, and / or actuator. In some cases, such components may be external to the device 2200 and / or communicate with components internal to the device 2200, such as input jacks, USB ports, Bluetooth radios, and / or the like. Similarly, output components 2245 enable the device 2200 to provide output, such as via a display, printer, speaker, and / or the like, any of which may be internal to the device 2200 and / or external to the device but communicate with internal components, such as USB ports, Bluetooth radios, video ports, and / or the like. Similarly, unless the context otherwise indicates, any such component may be considered part of the device 2200 according to various embodiments.

[0107] These examples should illustrate that various embodiments can support various arrangements of external and / or internal components, all of which can be considered part of the device 2200.

[0108] In one aspect, the non-volatile storage device 2215 may be considered a non-transitory computer-readable medium. In some embodiments, the non-volatile storage device 2215 may be used to store software and / or data for use by the device 2200. This software / data may include an operating system 2250a, data 2255a, and / or instructions 2260a. Depending on the nature of the device 2200, the operating system may contain instructions for managing the basic operations of the device 2200 and may include various personal computer or server operating systems, embedded operating systems, and / or the like. Data 2255a may contain any of various types of data used or generated by the device 2200 (and / or its operation), such as media content, databases, documents, and / or the like. Instructions 2260a may contain software code for programming the processor 2210 to perform operations according to various embodiments, such as application programs, object code, assembly, binary, etc. In one aspect, in some embodiments, the operating system 2250a may be considered part of the instructions 2260a. Copies of the operating system 2250b, data 2255b, and / or instructions 2260b may be stored in the working memory 2220 and / or executed by one or more processors 2210.

[0109] Processor 2210 may include one or more of the following: a central processing unit, a graphics processing unit, a microprocessor, a controller, a microcontroller, a digital signal processor (DSP), programmable logic (e.g., a field-programmable gate array (FPGA), an erasable programmable logic device (EPLD), or the like), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), and / or another type of processing component. Each of processor 2210 may be implemented in hardware, firmware, or a combination of hardware, firmware, and / or software. In some embodiments, processor 2210 includes one or more processors capable of being programmed to perform one or more operations or processes described elsewhere herein.

[0110] For example, in some embodiments, device 2200 may include logic 2265. In a particular embodiment, logic 2265 may include logic 2265a, which provides some functionality of the overall wireless 2200. In other cases, various other components may themselves include logic, for example, as described in more detail below.

[0111] This logic can be any kind of code, instructions, circuitry, and / or the like (e.g., for performing some or all of the processes and / or operations described herein) that can cause device 2200 (or its various subsystems or interfaces) to operate according to the embodiments herein. By way of example only, logic 2265 may include instructions 2260 that may be stored on non-volatile storage device 2215, loaded into working memory 2220, and / or executed by processor 2210 to perform operations and methods according to various embodiments. In one aspect, these instructions 2260 may be considered as programming processor 2210 to operate according to such embodiments. Similarly, operating system 2250 (which is to some extent separate from instructions 2260) may be stored on non-volatile storage device 2215, loaded into working memory 2220, and / or executed by processor 2210.

[0112] Alternatively and / or additionally, the logic may include circuitry 2225 (e.g., hardware or firmware) which may operate independently of or in cooperation with any processor 2210 that the device 2200 may or may not have. (As mentioned above, in some cases, circuitry 2225 itself may be considered processor 2210.) Circuitry 2225 may be embodied by a chip, SoC, ASIC, programmable logic device (FPGA, EPLD, etc.) and / or the like. Thus, some or all of the logic that implements or causes some or all of the operations described herein to be performed may be encoded in a hardware or firmware circuitry (e.g., circuitry 2225) and executed directly by such circuitry or dedicated or embedded processor 2210, rather than as software instructions 2260 loaded into working memory 2220. (In some cases, logic 2265 may include hardware instructions or various functionalities of a dedicated circuitry and / or device 2200 that can be performed by executing the hardware instructions or dedicated circuitry.) Therefore, unless the context otherwise indicates, the embodiments described herein are not limited to any particular combination of hardware, firmware, and / or software.

[0113] Device 2200 may also include a communication interface 2235, which may include (but is not limited to): one or more wireless interfaces 2270, enabling the device to communicate wirelessly and / or via radio frequency (RF) with other devices; and / or one or more wired interfaces 2275, enabling the device 2200 to communicate with other devices via wired (e.g., electrical and / or optical) connections. The wireless interface 2270 may include (but is not limited to) a Bluetooth interface, a Wi-Fi and / or WLAN interface, a 5G or cellular interface, a satellite interface, etc. Such wireless interfaces 2270 and wired interfaces 2275 may respectively include logic 2265b and 2265c, including (but not limited to) logic similar to or coexisting with logic 2265 and / or a processor similar to the aforementioned processor 2210.

[0114] In certain embodiments, for example, wireless interface 2270 or wired interface 2275 may include logic corresponding to various layers of the Open Systems Interconnection (OSI) model. For example, the logic of the wireless interface may include PHY and MAC segments, radio, any necessary modems, antennas, ports, etc., and / or logic implementing any higher layer of the OSI model, to some extent, without any such layer being implemented in the logic of device 2200 itself. In some embodiments, this logic 2265b, 2265c or interfaces 2270, 2275 themselves may be combined and implemented as a discrete chip, SoC, and / or the like. Depending on the nature of device 2200, communication interface 2235 (and / or wireless and wired interfaces 2270, 2275) may include any standard or proprietary components to allow communication as described according to various embodiments.

[0115] Various aspects of some embodiments

[0116] In addition to the exemplary embodiments described above, some embodiments may include any combination or sub-combination of the aspects discussed in the examples below. Furthermore, within the scope of the various embodiments, some or all aspects of the embodiments described below may be combined with and / or implemented therein with the examples described above. No single embodiment requires any particular combination of these aspects; however, for the same reason, the aspects described in different contexts should not necessarily be considered as distinct kinds or embodiments.

[0117] One set of embodiments includes a set-top box.

[0118] In one aspect of some embodiments, an exemplary set-top box includes an input interface for receiving a plurality of video frames from a video source. In one aspect of some embodiments, the set-top box includes a decoder for decoding the plurality of video frames to produce a plurality of decoded video frames. In one aspect of some embodiments, the set-top box includes an interface for providing the plurality of decoded video frames to a video receiver. In one aspect of some embodiments, the set-top box includes logic for reducing the display latency of each of the decoded video frames provided to the video receiver by reducing the number of video frames stored in one or more frame buffers in the set-top box's video pipeline.

[0119] In one aspect of some embodiments, another exemplary set-top box includes an input interface for receiving a plurality of video frames from a video source. In one aspect of some embodiments, the set-top box includes a decoder for decoding the plurality of video frames to generate a plurality of decoded video frames, the plurality of decoded video frames including one or more decoded partial video frames. In one aspect of some embodiments, the set-top box includes an output interface for providing at least some of the plurality of decoded video frames to a video receiver. In one aspect of some embodiments, the set-top box includes a processor for using the one or more decoded partial video frames to reduce display latency of the at least some of the plurality of decoded video frames provided to the video receiver.

[0120] In one aspect of some embodiments, another set-top box includes an input interface configured to receive a plurality of video frames from a video source at a fixed frame rate, the plurality of received video frames including video frames comprising a first plurality of subframes. In one aspect of some embodiments, the set-top box includes a processor configured to process at least some of the plurality of video frames to generate a plurality of processed video frames. In one aspect of some embodiments, processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes. In one aspect of some embodiments, processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames. In one aspect of some embodiments, the set-top box includes an output interface configured to provide at least some of the processed video frames, comprising at least some of the plurality of processed subframes, to a video receiver at the flexible frame rate.

[0121] Another set of embodiments provides an apparatus.

[0122] In one aspect of some embodiments, an apparatus includes logic for receiving a plurality of encoded video frames from a video source. In one aspect of some embodiments, the apparatus includes logic for processing at least some of the plurality of encoded video frames. In one aspect of some embodiments, the apparatus includes logic for providing at least some of the processed video frames to a video receiver. In one aspect of some embodiments, processing the plurality of video frames reduces the display latency of each of the video frames provided to the video receiver by reducing the number of video frames stored in one or more frame buffers in a video pipeline associated with the apparatus. In one aspect of some embodiments, the apparatus further includes logic for selectively disabling the logic for reducing the display latency of the video frames provided to the video receiver based on: applications, configuration settings, and / or user controls associated with the video frames.

[0123] In one aspect of some embodiments, the video pipeline includes a plurality of video processing stages and a plurality of frame buffers within the device, the plurality of processing stages including decoders, and the plurality of frame buffers including video processing frame buffers and display frame buffers.

[0124] In one aspect of some embodiments, processing the plurality of video frames includes decoding at least some of the plurality of encoded video frames to generate a plurality of decoded video frames. In one aspect of some embodiments, the apparatus includes providing at least some of the processed video frames to the video receiver, which includes providing at least some of the decoded video frames to the video receiver.

[0125] In one aspect of some embodiments, reducing the number of frames stored in one or more frame buffers in the video pipeline includes identifying a plurality of available decoded video frames in the video pipeline. In one aspect of some embodiments, reducing the number of frames stored in one or more frame buffers in the video pipeline includes identifying the most recently available decoded video frame. In one aspect of some embodiments, reducing the number of frames stored in one or more frame buffers in the video pipeline includes providing the most recently available decoded video frame to the video receiver. In one aspect of some embodiments, reducing the number of frames stored in one or more frame buffers in the video pipeline includes discarding the remainder of the plurality of available decoded video frames.

[0126] In one aspect of some embodiments, the received plurality of video frames are encoded at a fixed frame rate that imposes an order on the video frames. In one aspect of some embodiments, providing the most recently available decoded video frame to the video receiver includes providing the most recently available decoded video frames not in order from the fixed frame rate.

[0127] In one aspect of some embodiments, reducing the number of frames stored in one or more buffers in the video pipeline includes providing the plurality of decoded video frames to the video receiver without storing any of the plurality of decoded video frames in the video processing frame buffer.

[0128] In one aspect of some embodiments, reducing the number of frames stored in one or more buffers in the video pipeline includes providing the plurality of decoded video frames to the video receiver while storing no more than a single frame in the frame display buffer.

[0129] In one aspect of some embodiments, providing the plurality of decoded video to the video receiver includes providing at least some of the plurality of decoded video frames to the video receiver via a High Definition Multimedia Interface (HDMI) connection.

[0130] In one aspect of some embodiments, providing the plurality of decoded frames to the video receiver further includes providing at least some of the plurality of decoded video frames to the video receiver via an alternative connection separate from the HDMI connection.

[0131] In one aspect of some embodiments, reducing the number of frames stored in one or more buffers in the video pipeline further includes storing a plurality of reference frames in a reference buffer located outside the video pipeline and separate from the display buffer of the video pipeline.

[0132] In one aspect of some embodiments, the apparatus includes logic for generating video frames from one or more of the reference frames stored in the reference buffer.

[0133] In one aspect of some embodiments, reducing the number of frames stored in one or more buffers in the video pipeline further includes allowing frame tearing when displaying the plurality of decoded video frames. In one aspect of some embodiments, allowing frame tearing when displaying the plurality of decoded video frames includes selectively allowing frame tearing based on user input or characteristics of the application displaying the plurality of video frames.

[0134] In one aspect of some embodiments, providing at least some of the processed video frames to the video receiver includes decoding at least some of the plurality of encoded video frames at a rate faster than the frame rate of the plurality of encoded video frames. In one aspect of some embodiments, providing at least some of the processed video frames to the video receiver includes providing a decoded portion of an encoded video frame to the output of the device before the entire encoded video frame is fully decoded.

[0135] In one aspect of some embodiments, the decoded portion of the encoded video frame includes one or more macroblock units or one or more coding units. In one aspect of some embodiments, providing the decoded portion of the encoded video frame to the output of the device reduces the display delay of the decoded portion of the encoded video frame to the number of lines specified by the one or more macroblock units or one or more coding units.

[0136] In one aspect of some embodiments, the device is a set-top box, a component of a set-top box, or a system-on-a-chip (SoC). In one aspect of some embodiments, the device is a television.

[0137] In one aspect of some embodiments, an exemplary apparatus includes logic for receiving a plurality of video frames from a video source. In one aspect of some embodiments, the apparatus includes logic for processing the plurality of video frames to generate a plurality of processed video frames, the plurality of processed video frames including one or more processed partial video frames. In one aspect of some embodiments, the apparatus includes logic for providing at least some of the plurality of processed video frames to a video receiver. In one aspect of some embodiments, the one or more processed partial frames reduce the display latency of one or more of the video frames provided to the video receiver. In one aspect of some embodiments, the apparatus includes logic for selectively disabling the logic for processing the plurality of video frames based on an application, configuration setting, or user control associated with the video frame.

[0138] In one aspect of some embodiments, receiving the plurality of video frames includes receiving one or more encoded partial video frames. In another aspect of some embodiments, processing the plurality of video frames includes creating the one or more partial video frames from one of the plurality of received video frames.

[0139] In one aspect of some embodiments, the plurality of video frames received from the video source include a plurality of encoded video frames. In one aspect of some embodiments, processing the plurality of video frames includes decoding at least some of the plurality of encoded video frames to generate a plurality of decoded video frames, the plurality of decoded video frames including one or more decoded partial video frames. In one aspect of some embodiments, providing at least some of the plurality of processed video frames to the video receiver includes providing at least some of the plurality of decoded video frames to the video receiver. In one aspect of some embodiments, reducing the display latency of each of the decoded video frames provided to the video receiver further includes providing a decoded partial video frame to the video receiver. In one aspect of some embodiments, providing at least some of the plurality of decoded video frames to the video receiver includes providing each of the plurality of decoded video frames, including the decoded partial video frames, to the video receiver via a High Definition Multimedia Interface (HDMI) connection.

[0140] In one aspect of some embodiments, providing at least some of the plurality of decoded video frames to the video receiver includes providing at least some of the plurality of decoded video frames to the video receiver via a High Definition Multimedia Interface (HDMI) connection. In one aspect of some embodiments, providing at least some of the plurality of decoded video frames to the video receiver includes providing the decoded partial video frames to the video receiver via an alternative path separate from the HDMI connection. In one aspect of some embodiments, the apparatus includes logic for providing metadata to the video receiver to enable the video receiver to use the decoded partial video frames to create a decoded full video frame.

[0141] In one aspect of some embodiments, the plurality of encoded video frames includes one or more encoded partial video frames encoded at adaptive resolution. In one aspect of some embodiments, decoding the one or more encoded partial video frames includes decoding the one or more encoded partial video frames encoded at adaptive resolution.

[0142] In one aspect of some embodiments, the plurality of encoded video frames includes a first encoded full video frame encoded at a first resolution. In one aspect of some embodiments, the plurality of encoded video frames includes encoded partial video frames following the encoded full video frame, the encoded partial video frames being encoded at a second resolution. In one aspect of some embodiments, decoding at least some of the plurality of encoded video frames includes decoding the encoded full video frame to produce a decoded full video frame. In one aspect of some embodiments, decoding at least some of the plurality of encoded video frames includes decoding the encoded partial video frames to produce a decoded partial video frame.

[0143] In one aspect of some embodiments, providing at least some of the plurality of decoded video frames to the video receiver includes providing the decoded portion of the video frames to the video receiver.

[0144] In one aspect of some embodiments, processing the plurality of video frames includes generating a second decoded full video frame from the first decoded full video frame and the decoded partial video frames. In one aspect of some embodiments, providing at least some of the plurality of decoded video frames to the video receiver includes providing the second decoded full video frame to the video receiver.

[0145] In one aspect of some embodiments, the coded partial video frame includes content predicted from a portion of the coded full video frame corresponding to the coded partial video frame using inter-frame coding.

[0146] In one aspect of some embodiments, processing the plurality of video frames includes storing the first decoded full video frame in a reference buffer. In one aspect of some embodiments, processing the plurality of video frames includes receiving a second encoded full video frame after the encoded partial video frame, the second encoded full video frame being encoded from the first encoded full video frame using inter-frame coding. In one aspect of some embodiments, processing the plurality of video frames includes using the first decoded full video frame to decode the second encoded full video frame to generate a second decoded full video frame. In one aspect of some embodiments, providing at least some of the plurality of decoded video frames to the video receiver includes providing the second decoded full video frame to the video receiver.

[0147] In one aspect of some embodiments, the apparatus includes logic for receiving information from the video source to indicate the position of the encoded partial video frame within the entire frame.

[0148] In one aspect of some embodiments, processing the plurality of video frames includes calculating the position of the decoded partial video frame within the entire frame. In another aspect of some embodiments, processing the plurality of video frames includes providing information to the video receiver to indicate the position of the decoded partial video frame within the entire frame.

[0149] In one aspect of some embodiments, the device is a set-top box, a component of a set-top box, or a system-on-a-chip (SoC). In one aspect of some embodiments, the device is a television.

[0150] In one aspect of some embodiments, an apparatus includes logic for receiving a plurality of video frames from a video source at a fixed frame rate, the plurality of received video frames including video frames comprising a first plurality of subframes. In one aspect of some embodiments, the apparatus includes logic for processing at least some of the plurality of video frames to generate a plurality of processed video frames. In one aspect of some embodiments, processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes. In one aspect of some embodiments, processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames. In one aspect of some embodiments, the apparatus includes logic for providing at least some of the plurality of processed video frames, comprising at least some of the processed first plurality of subframes, to a video receiver at the flexible frame rate. In one aspect of some embodiments, the apparatus includes logic for selectively disabling the decoupling of the fixed frame rate from the flexible frame rate and providing the processed video frames at the flexible frame rate based on applications, configuration settings, and / or user controls associated with the video frames.

[0151] In one aspect of some embodiments, processing at least some of the plurality of video frames includes encoding at least some of the plurality of video frames with a video encoder / decoder (CODEC) for transmission over a network. In one aspect of some embodiments, the received plurality of video frames includes a plurality of encoded video frames encoded with a video CODEC. In one aspect of some embodiments, processing at least some of the plurality of video frames includes decoding at least some of the plurality of video frames to generate a plurality of decoded video frames.

[0152] In one aspect of some embodiments, providing each of the plurality of processed frames at the flexible frame rate includes providing a first set of the plurality of decoded video frames to the video receiver via a dedicated multimedia interface. In one aspect of some embodiments, providing each of the plurality of processed frames at the flexible frame rate includes providing a second set of the plurality of decoded video frames to the video receiver via a local area network.

[0153] In one aspect of some embodiments, providing at least some of the processed first plurality of subframes to the video receiver includes providing not all of the processed first plurality of subframes to the video receiver. In one aspect of some embodiments, processing at least some of the first plurality of subframes includes processing not all of the first plurality of subframes. In one aspect of some embodiments, providing at least some of the processed first plurality of subframes to the video receiver includes providing at least some of the processed subframes as a portion of a separate stream. In one aspect of some embodiments, providing at least some of the processed first plurality of subframes to the video receiver includes providing the processed subframes as a portion of a single stream.

[0154] In one aspect of some embodiments, providing the plurality of processed video frames to the video receiver includes generating the processed video frames from at least some of the processed first plurality of subframes and previously processed video frames. In another aspect of some embodiments, providing the plurality of processed video frames to the video receiver includes providing the processed video frames to the video receiver.

[0155] In one aspect of some embodiments, decoding at least some of the plurality of encoded video frames to produce a plurality of decoded video frames includes decoding two or more of the first plurality of subframes in parallel. In one aspect of some embodiments, receiving the plurality of encoded video frames from the video source includes receiving at least some of the first plurality of subframes as a separate stream. In one aspect of some embodiments, receiving the first plurality of encoded video frames from the video source includes receiving the first plurality of subframes as a single stream.

[0156] In one aspect of some embodiments, receiving the plurality of video frames from the video source includes receiving the entire video frame. In one aspect of some embodiments, processing at least some of the plurality of video frames includes dividing the entire video frame into a second plurality of subframes. In one aspect of some embodiments, providing each of the plurality of processed video frames to the video receiver at the flexible frame rate includes providing at least some of the second plurality of subframes to the video receiver.

[0157] In one aspect of some embodiments, processing at least some of the plurality of received video frames includes processing at least some of the plurality of received video frames at a rate higher than the fixed frame rate. In one aspect of some embodiments, providing each of the plurality of processed video frames to the video receiver at the flexible frame rate includes providing each of the plurality of processed video frames to the video receiver at the rate at which each corresponding frame has been processed without post-processing buffering.

[0158] In one aspect of some embodiments, each of the received plurality of frames includes a decoded frame having a first frame size. In one aspect of some embodiments, each of the processed video frames includes a display frame having a second frame size smaller than the first frame size, the display frame including a portion of the decoded frame. In one aspect of some embodiments, the portion of the decoded frame included in the display frame includes variations between subsequent frames.

[0159] In one aspect of some embodiments, a method includes receiving a plurality of video frames from a video source by a device. In one aspect of some embodiments, the method includes processing the plurality of video frames. In one aspect of some embodiments, the method includes providing at least some of the plurality of video frames to a video receiver. In one aspect of some embodiments, processing the plurality of video frames reduces the display latency of each of the video frames provided to the video receiver by reducing the number of video frames in the video pipeline associated with the device.

[0160] In one aspect of some embodiments, another exemplary method includes receiving a plurality of video frames from a video source. In one aspect of some embodiments, the method includes processing the plurality of video frames to generate a plurality of processed video frames, the plurality of processed video frames including one or more processed partial video frames. In one aspect of some embodiments, the method includes providing at least some of the plurality of processed video frames to a video receiver. In one aspect of some embodiments, the one or more processed partial frames reduce the display latency of one or more of the video frames provided to the video receiver.

[0161] In one aspect of some embodiments, another method includes receiving a plurality of video frames from a video source at a fixed frame rate, the plurality of received video frames including video frames comprising a first plurality of subframes. In one aspect of some embodiments, the method includes processing at least some of the plurality of video frames to generate a plurality of processed video frames. In one aspect of some embodiments, processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes. In one aspect of some embodiments, processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames. In one aspect of some embodiments, the method includes providing at least some of the plurality of processed video frames, comprising at least some of the first plurality of processed subframes, to a video receiver at the flexible frame rate.

[0162] in conclusion

[0163] In the foregoing description, numerous details have been set forth for illustrative purposes to provide a full understanding of the described embodiments. However, those skilled in the art will understand that other embodiments can be practiced without some of these details. In other instances, structures and apparatuses are shown in block diagram form for clarity, without exhaustive detail. Several embodiments are described herein, and while various features pertain to different embodiments, it should be understood that features described with respect to one embodiment may also be incorporated into other embodiments. However, for the same reason, any single or multiple features describing an embodiment should not be considered essential to every embodiment of the invention, as such features may be omitted in other embodiments of the invention.

[0164] Therefore, the above description provides an illustration and description of some features and aspects of various embodiments, but is not intended to be exhaustive or to substantially limit the embodiments to the precise forms disclosed. Those skilled in the art will recognize that modifications can be made in light of the above disclosure or modifications can be obtained from the practice of the embodiments, all of which fall within the scope of the various embodiments. For example, as mentioned above, the methods and processes described herein can be implemented using software components, firmware and / or hardware components (including (but not limited to) processors, other hardware circuit systems, custom integrated circuits (ICs), programmable logic, etc.) and / or any combination thereof.

[0165] Furthermore, while for ease of description, the various methods and processes described herein may be described with respect to specific structures and / or functional components, the methods provided by the various embodiments are not limited to any particular structural and / or functional architecture, but can be implemented in any suitable hardware configuration. Similarly, while some functionality may be attributed to one or more system components, unless the context otherwise indicates, this functionality may be distributed across a variety of other system components according to several embodiments.

[0166] Similarly, although the procedures of the methods and processes described herein are presented in a specific order for ease of description, various procedures may be reordered, added, and / or omitted according to various embodiments unless the context otherwise indicates. Furthermore, the procedures described with respect to one method or process may be incorporated into other described methods or processes; similarly, system components described with respect to a particular architecture and / or system may be organized in an alternative architecture and / or incorporated into other described systems, depending on the specific architecture. Therefore, although various embodiments with or without certain features are described for ease of description and illustration of aspects of these embodiments, various components and / or features described herein with respect to particular embodiments may be replaced, added, and / or omitted by other described embodiments unless the context otherwise indicates.

[0167] As used herein, the term "component" is intended to be interpreted broadly as hardware, firmware, software, or any combination thereof. It will be understood that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, and / or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit any embodiment unless specifically stated in the appended claims. Therefore, when the operation and behavior of systems and / or methods are described herein without reference to specific software code, those skilled in the art will understand that software and hardware can be used to implement the systems and / or methods based on the description herein.

[0168] In this disclosure, when an element is referred to herein as “connected” or “coupled” to another element, it should be understood that an element may be directly connected to another element or that an intervening element may exist between the elements. Conversely, when an element is referred to as “directly connected” or “directly coupled” to another element, it should be understood that there is no intervening element in a “direct” connection between the elements. However, the presence of a direct connection does not preclude the presence of other connections in which intervening elements may exist. Similarly, although the methods and processes described herein may be described in a particular order for ease of description, it should be understood that unless the context otherwise indicates, the intervention process may occur before and / or after any part of the described process, and as mentioned above, the described procedures may be reordered, added, and / or omitted according to various embodiments.

[0169] In this application, the use of the singular includes the plural unless otherwise specified, and the use of the term “and” means “and / or” unless otherwise indicated. Furthermore, as used herein, the term “or” when used in a series implies inclusion and may be used interchangeably with “and / or” unless otherwise expressly stated (e.g., if used in combination with “(either of)” or “only one of…”). Furthermore, the use of the term “comprising” and other forms such as “include” and “included” should be considered non-exclusive. Additionally, the terms “element” or “component” cover both elements and components comprising one unit and elements and components comprising more than one unit, unless otherwise specified. As used herein, the phrase “at least one of…” preceding a series of items and the terms “and” or “or” used to separate any item modify the entire list, not each member of the list (i.e., each item). The phrase “at least one of…” does not require selection of at least one for each item; rather, the phrase allows for the meaning of at least one of any item and / or at least one of any combination of items. For example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C and / or any combination of A, B, and C. In cases where it is desired to select “at least one of each of A, B, and C” or, alternatively, “at least one A, at least one B, and at least one C”, it is explicitly described as such.

[0170] Unless otherwise indicated, all figures used herein to indicate quantity, size, etc., should be understood to be modified by the term “about” in all instances. As used herein, the article “a” is intended to include one or more items and is interchangeable with “one or more”. Similarly, as used herein, the article “described” is intended to include one or more items referenced in conjunction with the article “described” and is interchangeable with “described one or more”. As used herein, the term “group” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or their equivalents) and is interchangeable with “one or more”. If only one item is desired, the phrase “only one” or similar language is used. As used herein, the term “has” or its equivalents are intended to be open-ended terms. Furthermore, the phrase “based on” is intended to mean “at least partially based on”, unless explicitly stated otherwise. In the above descriptions, depending on the context, satisfying a threshold can refer to a value greater than a threshold, greater than or equal to a threshold, less than a threshold, less than or equal to a threshold, equal to a threshold, or its equivalents, depending on the context.

[0171] Although there are specific combinations of features described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of the various embodiments. In fact, many of these features can be combined in ways not specifically described in the claims and / or disclosed in the specification. Therefore, while each appended dependent claim may be directly relied upon by only one claim, the disclosure of the various embodiments includes a combination of each dependent claim with each other claim in the claim set. No element, action, or instruction used herein should be construed as critical or essential unless expressly described as such.

Claims

1. An apparatus comprising: Logic for receiving multiple video frames from a video source at a fixed frame rate, wherein the multiple received video frames include video frames containing a first plurality of subframes; Logic for processing at least some of the plurality of video frames to generate a plurality of processed video frames, wherein: Processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes; and Processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames; and Logic for providing at least some of the plurality of processed video frames, containing at least some of the processed first plurality of subframes, to a video receiver at the flexible frame rate.

2. The apparatus according to claim 1, wherein: Processing at least some of the plurality of video frames includes encoding the at least some of the plurality of video frames with a video encoder / decoder (CODEC) for transmission over a network.

3. The apparatus according to claim 1, wherein: The received multiple video frames include multiple encoded video frames encoded with a video CODEC. and Processing at least some of the plurality of video frames includes: Decode at least some of the plurality of video frames to generate a plurality of decoded video frames.

4. The apparatus according to claim 3, wherein: Providing each of the plurality of processed frames at the flexible frame rate includes: The first group of the plurality of decoded video frames is provided to the video receiver via a dedicated multimedia interface; and The second set of the plurality of decoded video frames is provided to the video receiver via a local area network.

5. The apparatus according to claim 4, wherein: Providing at least some of the processed first plurality of subframes to the video receiver includes: Not all of the processed first plurality of subframes are provided to the video receiver.

6. The apparatus according to claim 1, wherein: Processing at least some of the first plurality of subframes includes: Process not all of the first plurality of subframes.

7. The apparatus according to claim 1, wherein: Providing at least some of the processed first plurality of subframes to the video receiver includes: Provide at least some of the processed subframes as parts of a separate stream.

8. The apparatus according to claim 1, wherein: Providing at least some of the processed first plurality of subframes to the video receiver includes: The processed subframe is provided as part of a single stream.

9. The apparatus according to claim 1, wherein: Providing the plurality of processed video frames to the video receiver includes: A processed video frame is generated from at least some of the processed first plurality of subframes and previously processed video frames; and The processed video frame is provided to the video receiver.

10. The apparatus according to claim 1, wherein: Decoding at least some of the plurality of encoded video frames to generate a plurality of decoded video frames includes: Parallel decoding of two or more of the first plurality of subframes.

11. The apparatus according to claim 1, wherein: Receiving the plurality of encoded video frames from the video source includes: Receive at least some of the first plurality of subframes as separate streams.

12. The apparatus according to claim 1, wherein: Receiving the first plurality of encoded video frames from the video source includes: Receive the first plurality of subframes as a single stream.

13. The apparatus according to claim 1, wherein: Receiving the plurality of video frames from the video source includes: Receive all video frames; Processing at least some of the plurality of video frames includes: The entire video frame is divided into a second set of sub-frames; and Providing each of the plurality of processed video frames to the video receiver at the flexible frame rate includes: At least some of the second plurality of subframes are provided to the video receiver.

14. The apparatus according to claim 1, wherein: Processing at least some of the plurality of received video frames includes: At least some of the received video frames are processed at a rate higher than the fixed frame rate; and Providing each of the plurality of processed video frames to the video receiver at the flexible frame rate includes: Each of the plurality of processed video frames is provided to the video receiver at the rate at which each corresponding frame has been processed, without the need for post-processing buffering.

15. The apparatus of claim 1, wherein the apparatus comprises a set-top box, a component of a set-top box, or a system-on-a-chip (SoC).

16. The apparatus of claim 1, wherein the apparatus is a television.

17. The apparatus according to claim 1, wherein: Each of the received multiple frames contains a decoded frame with a first frame size; Each of the processed video frames includes a display frame with a second frame size smaller than the first frame size, the display frame including a portion of the decoded frame; and The portion of the decoded screen included in the displayed screen includes changes between subsequent frames.

18. The apparatus of claim 1, further comprising: Logic for selectively disabling the decoupling of the fixed frame rate and the flexible frame rate, and for providing the processed video frames at the flexible frame rate, based on the following: Applications associated with the video frame; Configuration settings; or User control.

19. A method comprising: Receive multiple video frames from a video source at a fixed frame rate, wherein the multiple received video frames include video frames containing a first plurality of subframes; Process at least some of the plurality of video frames to generate a plurality of processed video frames, wherein: Processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes; and Processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames; and At least some of the plurality of processed video frames, including at least some of the processed first plurality of subframes, are provided to the video receiver at the flexible frame rate.

20. A set-top box, comprising: An input interface configured to receive multiple video frames from a video source at a fixed frame rate, the multiple received video frames including video frames containing a first plurality of subframes; A processor configured to process at least some of the plurality of video frames to generate a plurality of processed video frames, wherein: Processing at least some of the plurality of video frames includes processing at least some of the first plurality of subframes; and Processing at least some of the plurality of video frames decouples the fixed frame rate of the received video frames from the flexible frame rate of the processed video frames; An output interface configured to provide at least some of the processed video frames, comprising at least some of the plurality of processed subframes, to a video receiver at the flexible frame rate.

Images