Temporally interpolated picture prediction using frame level motion vectors

Abstract

A frame-level motion vector is used for motion compensated prediction of a generated (e.g., temporally interpolated picture) reference frame to improve prediction quality of the generated reference frame when the generated reference frame is used as a display frame in direct mode. For a current frame encoded to a bitstream, a reference frame is generated using a backward reference frame of the current frame and a forward reference frame of the current frame. It is determined that the generated reference frame is to be used as a display frame in place of the current frame. Based on the determination, a prediction of the generated reference frame is generated using a frame-level motion vector signaled within the bitstream. A frame reconstruction is generated based on the prediction, and the frame reconstruction is output within an output video stream.

Description

BACKGROUND

[0001] Digital video streams can use sequences of frames or still images to represent video. Digital video can be used for a variety of applications, including, for example, video conferencing, high-definition video entertainment, video advertising, or sharing of user-generated video. Digital video streams can contain large amounts of data and consume a substantial amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various ways have been proposed to reduce the amount of data in a video stream, including encoding or decoding techniques. SUMMARY

[0002] Systems and techniques for picture prediction using temporally interpolated frame-level motion vectors during encoding are disclosed herein, among other things.

[0003] A method for prediction using frame-level motion vectors according to implementations of the present disclosure includes generating, for a current frame encoded into a bitstream, a reference frame using a backward reference frame of the current frame and a forward reference frame of the current frame, determining that the generated reference frame is to be used as a display frame in place of the current frame, generating, based on the determination, a prediction of the generated reference frame using a frame-level motion vector signaled within the bitstream, producing a frame reconstruction based on the prediction, and outputting the frame reconstruction within an output video stream.

[0004] In some implementations of the method, generating the reference frame using the backward reference frame and the forward reference frame includes generating a motion field for the current frame using motion vectors of the backward reference frame and the forward reference frame, using the motion field to fetch reference blocks from the backward reference frame and the forward reference frame, and generating the reference frame using the reference blocks.

[0005] In some implementations of the method, generating the reference frame using the reference blocks includes interpolating the reference blocks.

[0006] In some implementations of the method, determining that the generated reference frame is to be used as a display frame in place of the current frame includes determining, based on signaling of a ZEROMV motion vector for the generated reference frame, whether there is a motion mismatch with the generated reference frame.

[0007] In some implementations of the method, the ZEROMV motion vector is the frame-level motion vector.

[0008] In some implementations of the method, generating the prediction of the generated reference frame using the frame-level motion vector signaled within the bitstream includes performing additional residual code processing for the generated reference frame to prepare contents of the generated reference for display within the output video stream.

[0009] In some implementations of the method, the additional residual code processing includes motion compensation performed for the generated reference frame using the frame-level motion vector.

[0010] In some implementations of the method, generating the frame reconstruction based on the prediction includes interpolating motion indicated by the frame-level motion vector using an interpolation filter.

[0011] In some implementations of the method, the interpolation filter is based on data signaled within the bitstream according to an encoder search.

[0012] A non-transitory computer-readable medium having stored thereon an encoded bitstream, wherein the encoded bitstream is configured for decoding using a predicted operation of a frame-level motion vector, the operation comprising: determining to use a generated reference frame associated with a current frame encoded to the encoded bitstream as a display frame in place of the current frame; based on the determination, generating a prediction of the generated reference frame using the frame-level motion vector; and outputting a frame reconstruction produced based on the prediction within an output video stream.

[0013] In some implementations of the non-transitory computer-readable medium, the operation comprises generating the generated reference frame using motion vectors of a backward reference frame of the current frame and a forward reference frame of the current frame.

[0014] In some implementations of the non-transitory computer-readable medium, determining to use the generated reference frame as a display frame in place of the current frame comprises determining to signal the frame-level motion vector within the encoded bitstream for the generated reference frame.

[0015] In some implementations of the non-transitory computer-readable medium, generating the prediction of the generated reference frame using the frame-level motion vector comprises performing motion compensation for the generated reference frame using the frame-level motion vector.

[0016] In some implementations of the non-transitory computer-readable medium, the operation comprises generating the frame reconstruction based on the prediction; and preparing the frame reconstruction for output as a final display frame using an interpolation filter by interpolating motion indicated by the frame-level motion vector.

[0017] An apparatus for using a predicted operation of a frame-level motion vector comprises: a memory; and a processor configured to execute instructions stored in the memory to: generate a reference frame for a current frame to be decoded; determine to replace the current frame with the generated reference frame within an output video stream; based on the determination, generate a prediction of the generated reference frame using the frame-level motion vector; and output a frame reconstruction produced based on the prediction as a display frame within the output video stream.

[0018] In some implementations of the device, the generated reference frame is generated using a backward reference frame of the current frame and a forward reference frame of the current frame, wherein the backward reference frame, the current frame, and the forward reference frame are signaled within an encoded bitstream and decoded from the encoded bitstream.

[0019] In some implementations of the device, the determination that the current frame is to be replaced within the output video stream with the generated reference frame is based on signaling of frame-level motion vectors within an encoded bitstream.

[0020] In some implementations of the device, the prediction is generated based on additional residual code processing performed for the generated reference frame.

[0021] In some implementations of the device, the additional residual code processing includes motion compensation performed using frame-level motion vectors.

[0022] In some implementations of the device, the frame reconstruction is produced using an interpolation filter.

[0023] These and other aspects of the disclosure are disclosed in the following detailed description of implementations, in the claims, and in the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The description herein makes reference to the accompanying drawings described below, wherein like reference numerals refer to like parts throughout the several views, and

[0025] Figure 1 is a schematic diagram of an example of a video encoding and decoding system.

[0026] Figure 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station.

[0027] Figure 3 is a diagram of an example of a video stream to be encoded and decoded.

[0028] Figure 4 is a block diagram of an example of an encoder.

[0029] Figure 5 is a block diagram of an example of a decoder.

[0030] Figure 6 is an illustration of an example of portions of a video frame.

[0031] Figure 7 is an illustration of a frame used in conjunction with temporally interpolated picture video coding.

[0032] Figure 8 is a flowchart of an example of a technique for temporally interpolated picture prediction using frame-level motion vectors during encoding.

[0033] Figure 9 is a flowchart of an example of a technique for decoded picture prediction using temporally interpolated pictures for frame-level motion vectors during decoding. DETAILED DESCRIPTION

[0034] Video compression schemes can include breaking down respective pictures or frames of a video stream into smaller portions, such as blocks, and generating an encoded bitstream by using encoding techniques that limit the information included by their respective blocks. The bitstream can be decoded to recreate the source frames from the limited information. Video streams can be compressed (i.e., encoded) by a variety of techniques to reduce the bandwidth required to transmit or store the video stream. Similarly, the compressed video stream can be decompressed (i.e., decoded) from the bitstream using a variety of techniques to prepare the video stream for viewing or further processing. Compression of video streams often exploits the spatial and temporal correlation of video signals through spatial and / or motion-compensated prediction. Motion-compensated prediction can also be referred to as inter-frame prediction. Inter-frame prediction uses one or more motion vectors to generate a block (also referred to as a prediction block) that is similar to a current block to be encoded using previously encoded and decoded pixels. By encoding the motion vectors and the difference between the two blocks (i.e., the residual), a decoder receiving the encoded signal can reconstruct the current block by generating the prediction block and adding the pixels of the prediction block to the decoded residual block.

[0035] Each motion vector used to generate a prediction block in an inter-frame prediction process references a reference frame (i.e., a frame other than the current frame that includes the block being predicted). The reference frame can be located before or after the current frame in a sequence of the video stream and can be a frame that was reconstructed prior to being used as a reference frame. In particular, the reference frame can be a forward reference frame (i.e., a frame used for forward prediction relative to the sequence) or a backward reference frame (i.e., a frame used for backward prediction relative to the sequence). One or more forward reference frames and / or backward reference frames can be used to encode or decode a block. In particular, because many conventional video compression and decompression schemes use a pyramid code processing structure to achieve high compression efficiency, many frames are encoded and decoded using bi-prediction, such as using a forward reference frame and a backward reference frame. Bi-prediction using forward and backward reference frames has proven to greatly improve prediction quality and thus greatly improve the overall compression performance of the subject video stream.

[0036] A recent approach for bi-directional prediction uses temporally interpolated picture (TIP) reference frames. A TIP reference frame is a generated reference frame generated by interpolating reference blocks from a forward reference frame and a backward reference frame (e.g., as the most recent future reference frame and the most recent past reference frame relative to a current frame). In particular, a motion field is generated for the current frame using available coded motion vectors in the forward and backward reference frames, and then the motion field is used to obtain reference blocks for generating the TIP reference frame. Thus, TIP video coding refers to an inter prediction mode by which motion of the current frame is predicted using a TIP reference frame. TIP video coding generally involves relatively small motion vectors applied for the TIP reference frame, which are not only cheaper to encode, but also improve prediction detail and quality due to the TIP reference frame leveraging forward and backward reference data. The TIP reference frame is generated independently at each of an encoder and a decoder. In particular, the encoder generates the TIP reference frame using data determined as part of an encoder search process, and the decoder generates the TIP reference frame using bitstream data indicative of the encoder search process. Using this TIP mode for video coding has shown significant coding gains relative to video coding schemes that do not use the TIP mode.

[0037] Current approaches for TIP video coding support a “direct mode” (e.g., “direct TIP mode”) by which a TIP reference frame is used directly (i.e., without additional residual coding) as a display frame to be included in an output video stream for storage or display. This direct mode has proven to improve coding gains by saving computational costs that would otherwise be spent on operations including, for example, partition or frame header processing. However, there are opportunities to further improve the direct mode. In particular, because the TIP frame in the direct mode is used as a display frame without additional residual coding, there is an opportunity to improve the prediction quality of the TIP frame. Implementations of the present disclosure address such issues by using frame-level motion vectors for temporally interpolated picture prediction. In particular, implementations of the present disclosure describe ways for video coding in which frame-level motion vectors that can be used to perform motion compensation for a generated (e.g., TIP) reference frame that is indicated for use in the direct mode are signaled within a bitstream, and used to improve the prediction quality of the generated reference frame.

[0038] While reference is made herein, by way of example, to superblocks, macroblocks, blocks, and the like as are commonly used in video codecs such as VP9, AV1, and AV2 which is currently under development, implementations of the present disclosure can be used in conjunction with other video code processing structures. In one particular, but non-limiting, example, implementations of the present disclosure can be used in conjunction with code processing tree units (CTUs), code processing units (CUs), prediction units (PUs), and the like as are commonly used in video codecs such as H.265, which is referred to as High Efficiency Video Coding, and H.266, which is referred to as Versatile Video Coding. Accordingly, reference herein to particular video code processing structures such as superblocks, macroblocks, blocks, and the like should be taken as an expression of non-limiting example video code processing structures that can be used in conjunction with implementations of the present disclosure.

[0039] Further details of such techniques are described herein initially with reference to systems in which the techniques for using temporally interpolated picture prediction of frame-level motion vectors can be implemented. Figure 1 is an example of a video encoding and decoding system 100. The sending station 102 can be, for example, a computer such as a desktop computer, a laptop computer, a tablet computer, a server computer, a smartphone, a gaming console, a video streaming device, or any other computing device that is capable of encoding and decoding video streams. The sending station 102 can also be a computer having an internal hardware configuration as described below. Figure 2 A computer having an internal hardware configuration as described below is described. However, other implementations of the sending station 102 are possible. For example, processing of the sending station 102 can be distributed among multiple devices.

[0040] The network 104 can connect the sending station 102 and the receiving station 106 for encoding and decoding of video streams. In particular, video streams can be encoded in the sending station 102 and encoded video streams can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of communicating video streams from the sending station 102 to the receiving station 106 (in this example).

[0041] In one example, the receiving station 106 can be a computer such as a desktop computer, a laptop computer, a tablet computer, a server computer, a smartphone, a gaming console, a video streaming device, or any other computing device that is capable of encoding and decoding video streams. Figure 2 A computer having an internal hardware configuration as described below is described. However, other suitable implementations of the receiving station 106 are possible. For example, processing of the receiving station 106 can be distributed among multiple devices.

[0042] Other implementations of the video encoding and decoding system 100 are possible. For example, implementations can omit the network 104. In another implementation, the video stream can be encoded and then stored for transmission to the receiving station 106 or any other device with storage at a later time. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and / or some communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, the Real-time Transport Protocol (RTP) is used to transmit the encoded video over the network 104. In another implementation, a transport protocol other than RTP can be used, such as a Hypertext Transfer Protocol (HTTP) based video streaming protocol.

[0043] When used in a video conferencing system, for example, the sending station 102 and / or the receiving station 106 can include the ability to both encode and decode video streams as described below. For example, the receiving station 106 can be a video conference participant that receives an encoded video bitstream from a video conference server (e.g., the sending station 102) for decoding and viewing, and further encodes and sends his or her own video bitstream to the video conference server for decoding and viewing by other participants.

[0044] In some implementations, the video encoding and decoding system 100 can alternatively be used to encode and decode data other than video data. For example, the video encoding and decoding system 100 can be used to process image data. The image data can include blocks of data from an image. In such implementations, the sending station 102 can be used to encode image data, and the receiving station 106 can be used to decode image data.

[0045] Alternatively, the receiving station 106 can represent a computing device such as storing encoded image data for later use after receiving the encoded or pre-encoded image data from the sending station 102. As a further alternative, the sending station 102 can represent a computing device such as decoding image data before sending the decoded image data to the receiving station 106 for display.

[0046] Figure 2 is a block diagram of an example of a computing device 200 that can implement a sending station or a receiving station. For example, the computing device 200 can implement one or both of the sending station 102 and the receiving station 106 of Figure 1 The computing device 200 can be in the form of a computing system that includes multiple computing devices, or in the form of one computing device (e.g., a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.).

[0047] The processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be any other type of device, or multiple devices, capable of manipulating or processing information now existing or developed in the future. For example, while the disclosed implementations can be practiced with one processor (e.g., processor 202) as shown, advantages in speed and efficiency can be achieved through the use of more than one processor.

[0048] In implementations, the memory 204 in the computing device 200 can be a read-only memory (ROM) device or a random access memory (RAM) device. However, other suitable types of storage devices can be used as the memory 204. The memory 204 can include code and data 206 that are accessed by the processor 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, including at least one program that permits the processor 202 to perform the techniques described herein. For example, the application programs 210 can include applications 1 through N that further include encoding and / or decoding software that performs picture prediction using temporal interpolation of frame-level motion vectors, as described herein, and the like.

[0049] The computing device 200 can also include a secondary storage 214, which can be, for example, a memory card used with a mobile computing device. Because video communication sessions can contain a considerable amount of information, they can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0050] The computing device 200 can also include one or more output devices, such as a display 218. In one example, the display 218 can be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 218 can be coupled to the processor 202 via the bus 212. In addition to, or as an alternative to, the display 218, other output devices can be provided that permit a user to program or otherwise use the computing device 200. When the output device is or includes a display, the display can be implemented in various ways, including through a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0051] The computing device 200 can also include, or be in communication with, an image sensing device 220, such as a camera or any other image sensing device 220, now existing or later developed, that can sense images, such as images of a user operating the computing device 200, or be in communication with such an image sensing device. The image sensing device 220 can be positioned such that it is directed toward a user operating the computing device 200. In an example, the position and optical axis of the image sensing device 220 can be configured such that the field of view includes an area that is directly adjacent to the display 218 and from which the display 218 is visible.

[0052] The computing device 200 can also include, or be in communication with, a sound sensing device 222, such as a microphone or any other sound sensing device, now existing or later developed, that can sense sounds in the vicinity of the computing device 200, or be in communication with such a sound sensing device. The sound sensing device 222 can be positioned such that it is directed toward a user operating the computing device 200 and can be configured to receive sounds, such as speech or other utterances, made by the user as the user operates the computing device 200.

[0053] Although Figure 2 The processor 202 and memory 204 of the computing device 200 are depicted as integrated into one unit, other configurations can be utilized. The operations of the processor 202 can be distributed across multiple machines (where individual machines can have one or more processors) that can be coupled directly or across a local area or other network. The memory 204 can be distributed across multiple machines such as network-based memory or memory in multiple machines performing the operations of the computing device 200.

[0054] While depicted here as one bus, the bus 212 of the computing device 200 can be composed of multiple buses. Further, the secondary storage 214 can be directly coupled to the other components of the computing device 200 or can be accessed via a network, and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, the computing device 200 can be implemented in a multitude of varied configurations.

[0055] Figure 3 is a diagram of an example of a video stream 300 to be encoded and decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a plurality of adjacent video frames 304. Although three frames are depicted as adjacent frames 304, the video sequence 302 can include any number of adjacent frames 304. The adjacent frames 304 can then be further subdivided into individual video frames, e.g., frame 306.

[0056] At a next level, frame 306 can be divided into a series of planes or slices 308. For example, slices 308 can be subsets of a frame that permit parallel processing. Slices 308 can also be subsets of a frame that can separate video data into individual colors. For example, a frame 306 of color video data can include one luminance plane and two chrominance planes. Slices 308 can be sampled at different resolutions.

[0057] Whether or not frame 306 is divided into slices 308, frame 306 can be further subdivided into blocks 310, which can contain data corresponding to, for example, NxM pixels in frame 306, where N and M can refer to the same integer value or different integer values. Blocks 310 can also be arranged to include data from one or more slices 308 of pixel data. Blocks 310 can be any suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger up to a maximum block size, which can be 128x128 pixels or another NxM pixel size.

[0058] Figure 4 FIG. 4 is a block diagram of an example of an encoder 400. As described above, encoder 400 can be implemented in a transmitting station 102, such as by providing a computer software program stored in a memory (e.g., memory 204). The computer software program can include machine instructions that, when executed by a processor such as processor 202, cause transmitting station 102 to encode video data in the manner described in FIG. 3. Encoder 400 can also be implemented as a special-purpose hardware included in, for example, transmitting station 102. In some implementations, encoder 400 is a hardware encoder. Figure 4

[0059] Encoder 400 has the following stages for performing various functions in a forward path (shown by solid connection lines) to produce an encoded or compressed bitstream 420 using video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy encoding stage 408. Encoder 400 can also include a reconstruction path (shown by dashed connection lines) for reconstructing encoded frames for future blocks. In Figure 4

[0060] ​​In some cases, the functions performed by encoder 400 may occur after filtering of video stream 300. That is, before encoder 400 receives video stream 300, video stream 300 may undergo preprocessing according to one or more implementations of this disclosure. Alternatively, encoder 400 itself may continue to perform functions related to... Figure 4 Such preprocessing is performed on video stream 300 before the described functions—such as before processing video stream 300 at intra / inter-frame prediction stage 402.

[0061] When the video stream 300 is submitted for encoding after preprocessing, the corresponding adjacent frames 304, such as frame 306, can be processed in blocks. At the intra-frame / inter-frame prediction stage 402, the corresponding blocks can be encoded using either intra-frame prediction (also known as intra-prediction) or inter-frame prediction (also known as inter-prediction). In either case, prediction blocks can be formed. In the case of intra-frame prediction, prediction blocks can be formed from samples that have already been encoded and reconstructed in the current frame. In the case of inter-frame prediction, prediction blocks can be formed from samples in one or more previously constructed reference frames.

[0062] Next, the predicted block can be subtracted from the current block at the intra / inter-frame prediction stage 402 to produce a residual block (also known as the residual). The transform stage 404 uses a block-based transform to transform the residual into transform coefficients, for example, in the frequency domain. The quantization stage 406 uses a quantizer value or quantization level to convert the transform coefficients into discrete quantum values, referred to as quantized transform coefficients. For example, the transform coefficients can be divided by the quantizer value and truncated.

[0063] The quantized transform coefficients are then entropy encoded by entropy coding stage 408. The entropy-coded coefficients, along with other information for decoding the block (which may include, for example, syntax elements indicating the prediction type, transform type, motion vector, quantizer value, etc.), are then output to the compressed bitstream 420. The compressed bitstream 420 can be formatted using various techniques such as variable-length code processing or arithmetic code processing. The compressed bitstream 420 may also be referred to as an encoded video stream or an encoded video bitstream, and the terms will be used interchangeably herein.

[0064] The reconstruction path (shown by the dashed connection line) can be used to ensure encoder 400 and (see below for details) Figure 5 The decoder 500 (described below) uses the same reference frame to decode the compressed bitstream 420. The reconstruction path is performed in accordance with (see below regarding...)Figure 5 Similar to the encoding process described above, the decoding process includes the following stages: entropy decoding stage 502, inverse quantization stage 504, inverse transform stage 506, intra / inter prediction stage 508, reconstruction stage 510, loop filtering stage 512, and post-filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0065] At the reconstruction stage 414, the predicted block that was predicted at the intra / inter prediction stage 402 can be added to the derived residual to create a reconstructed block. A loop- in filtering stage 416 can apply in-loop filters or other filters to the reconstructed block to reduce distortion, such as blocking artifacts. Examples of filters that can be applied at the loop- in filtering stage 416 include, but are not limited to, a deblocking filter, a directional enhancement filter, and a loop restoration filter.

[0066] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. In some implementations, for certain blocks or frames, a non-transform-based encoder can directly quantize the residual signal without the transform stage 404. In some implementations, an encoder can have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0067] Figure 5 is a block diagram of an example of a decoder 500. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to decode video data in the manner described in the Figure 5 The decoder 500 can also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106. In some implementations, the decoder 500 is a hardware decoder.

[0068] Similar to the reconstruction path of the encoder 400 discussed above, in one example, the decoder 500 includes the following stages for performing various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a post-filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0069] When the compressed bitstream 420 is submitted for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a quantized set of transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantizer value), and the inverse transform stage 506 performs an inverse transform on the dequantized transform coefficients to produce a derived residual, which can be the same as the derived residual created by the inverse transform stage 412 in the encoder 400. Using the header information decoded from the compressed bitstream 420, the decoder 500 can use the intra / inter-frame prediction stage 508 to create a prediction block identical to the prediction block created in the encoder 400 (e.g., at the intra / inter-frame prediction stage 402).

[0070] At reconstruction stage 510, predicted blocks can be added to the derived residuals to create reconstructed blocks. Loop filtering stage 512 can be applied to the reconstructed blocks to reduce blocking artifacts. Examples of filters that can be applied at loop filtering stage 512 include, but are not limited to, deblocking filters, directional enhancement filters, and loop recovery filters. Other filters can be applied to the reconstructed blocks. In this example, post-filtering stage 514 is applied to the reconstructed blocks to reduce blocking distortion, and the result is output as output video stream 516. Output video stream 516 can also be referred to as decoded video stream, and the terms will be used interchangeably herein.

[0071] Other variations of decoder 500 can be used to decode the compressed bitstream 420. In some implementations, decoder 500 can produce output video stream 516 without post-filtering stage 514, or otherwise omit post-filtering stage 514.

[0072] Figure 6 This is an illustration of examples of portions of video frame 600, which may be, for example, [the video frame could be...]. Figure 3The video frame 600 includes a plurality of 64x64 blocks 610, such as four 64x64 blocks 610 in two rows and two columns in a matrix or Cartesian plane, as shown. Each 64x64 block 610 can include up to four 32x32 blocks 620. Each 32x32 block 620 can include up to four 16x16 blocks 630. Each 16x16 block 630 can include up to four 8x8 blocks 640. Each 8x8 block 640 can include up to four 4x4 blocks 950. Each 4x4 block 950 can include 16 pixels, which can be represented in four rows and four columns in each respective block in the Cartesian plane or matrix. In some implementations, the video frame 600 can include blocks larger than 64x64 and / or smaller than 4x4. The video frame 600 can be partitioned into various block arrangements according to features within the video frame 600 and / or other criteria.

[0073] A pixel can include information representative of an image captured in the video frame 600, such as luminance information, color information, and location information. In some implementations, a block, such as the 16x16 pixel block shown, can include a luminance block 660, which can include luminance pixels 662, and two chrominance blocks 670, 680, such as a U or Cb chrominance block 670 and a V or Cr chrominance block 680. The chrominance blocks 670, 680 can include chrominance pixels 690. For example, the luminance block 660 can include 16x16 luminance pixels 662, and each chrominance block 670, 680 can include 8x8 chrominance pixels 690, as shown. Although one arrangement of blocks is shown, any arrangement can be used. Although Figure 6 Although N x N blocks are shown, in some implementations, N x M blocks can be used, where N and M are different numbers. For example, 32x64 blocks, 64x32 blocks, 16x32 blocks, 32x16 blocks, or any other size blocks can be used. In some implementations, N x 2N blocks, 2N x N blocks, or combinations thereof can be used.

[0074] In some implementations, coding the video frame 600 can include in-order block-level coding. In-order block-level coding can include coding the blocks of the video frame 600 in an order, such as a raster scan order, in which the blocks can be identified and processed starting from the block in the upper left corner of the video frame 600 or a portion of the video frame 600 and proceeding along the rows from left to right and from the top row to the bottom row, so that each block is identified in turn for processing. For example, the 64x64 block in the top left column of the video frame 600 can be the first block coded, and the 64x64 block immediately to the right of the first block can be the second block coded. The second row from the top can be the second row coded, so that the 64x64 block in the left column of the second row can be coded after the 64x64 block in the rightmost column of the first row.

[0075] In some implementations, coding the blocks of the video frame 600 can include using quadtree coding that can include coding smaller block units within the blocks in raster scan order. For example, the 64x64 block shown in the lower left corner of the portion of the video frame 600 can be coded using quadtree coding, where the upper left 32x32 block can be coded, then the upper right 32x32 block can be coded, then the lower left 32x32 block can be coded, and then the lower right 32x32 block can be coded. Each 32x32 block can be coded using quadtree coding, where the upper left 16x16 block can be coded, then the upper right 16x16 block can be coded, then the lower left 16x16 block can be coded, and then the lower right 16x16 block can be coded. Each 16x16 block can be coded using quadtree coding, where the upper left 8x8 block can be coded, then the upper right 8x8 block can be coded, then the lower left 8x8 block can be coded, and then the lower right 8x8 block can be coded. Each 8x8 block can be coded using quadtree coding, where the upper left 4x4 block can be coded, then the upper right 4x4 block can be coded, then the lower left 4x4 block can be coded, and then the lower right 4x4 block can be coded. In some implementations, the 8x8 blocks can be omitted for 16x16 blocks, and the 16x16 blocks can be coded using quadtree coding, where the upper left 4x4 block can be coded, then the other 4x4 blocks in the 16x16 block can be coded in raster scan order.

[0076] In some implementations, the code processing of the video frame 600 can include encoding information included in an original version of an image or video frame by, for example, omitting some of the information from the original version of the image or the video frame from a corresponding encoded image or encoded video frame. For example, the code processing can include reducing spectral redundancy, reducing spatial redundancy, or a combination thereof. Reducing spectral redundancy can include using a color model based on a luminance component (Y) and two chrominance components (U and V or Cb and Cr), which can be referred to as a YUV or YCbCr color model or color space. Using a YUV color model can include using a relatively large amount of information to represent a luminance component of a portion of the video frame 600 and using a relatively small amount of information to represent each corresponding chrominance component of the portion of the video frame 600. For example, a portion of the video frame 600 can be represented by a high resolution luminance component that can include a 16x16 block of pixels and two lower resolution chrominance components, where each chrominance component represents the portion of the image as an 8x8 block of pixels. A pixel can indicate a value, such as a value ranging from 0 to 255, and can be stored or transmitted using, for example, eight bits. Although the present disclosure is described with reference to a YUV color model, another color model can be used. Reducing spatial redundancy can include using, for example, a discrete cosine transform to transform a block into a frequency domain. For example, a unit of an encoder can perform a discrete cosine transform using spatial frequency based transform coefficient values.

[0077] Although described herein with reference to a matrix or Cartesian representation of the video frame 600 for clarity, the video frame 600 can be stored, transmitted, processed, or a combination thereof in a data structure such that pixel values can be efficiently represented for the video frame 600. For example, the video frame 600 can be stored, transmitted, processed, or any combination thereof in a two-dimensional data structure such as the illustrated matrix or in a one-dimensional data structure such as a vector array. Furthermore, although described herein as illustrating a chroma subsampled image in which U and V have half the resolution of Y, the video frame 600 can have a different configuration of its color channels. For example, still referring to a YUV color space, full resolution can be used for all color channels of the video frame 600. In another example, a color space other than a YUV color space can be used to represent the resolution of the color channels of the video frame 600.

[0078] Figure 7is an illustration of frames used in conjunction with TIP video coding. The current frame 700 represents a frame that is being predicted (e.g., using the TIP modes described herein) during encoding (e.g., at the intra / inter prediction stage 402) or decoding (e.g., at the intra / inter prediction stage 510), for example. Based on the backward reference frame 704 and the forward reference frame 706, a generated reference frame 702, which can be a TIP reference frame, for example, is generated (e.g., on a streaming basis (i.e., piecemeal)) using a motion field. For example, in the case where the current frame 700 is represented as F i , the backward reference frame 704 can be represented as F i-1 , and the forward reference frame 706 can be represented as F i+1 .

[0079] In general, the backward reference frame 704 and the forward reference frame 706 will be spaced the same distance from the current frame 700 in the display order of the video sequence that includes them. However, in some implementations, the backward reference frame 704 and the forward reference frame 706 can be spaced different distances from the current frame 700 in the display order. The temporal motion vector predictor 708 represents a motion vector predictor that points from the backward reference frame 704 to the forward reference frame 706. The motion vector 710 that points from the current frame 700 to the generated frame 702 represents a motion vector that can be used with the generated frame 702 to predict motion within one or more blocks of the current frame 700.

[0080] In particular, when the reference frames 704 and 706 are identified for use as reference frames for the current frame 700, the reference frames have already been coded. Thus, the motion vectors of the reference frames 704 and 706 are already known and available from the earlier coding of the reference frames 704 and 706. Thus, once the reference frames 704 and 706 are identified, the motion vectors of the reference frames 704 and 706 can be used to determine a motion field for the current frame 700. In particular, the motion field includes motion field motion vectors, each of which points to one of the forward reference frame 706 or the backward reference frame 704. The motion field effectively represents how to project the motion field motion vectors to determine the motion vectors for the current frame 700, since the current frame 700 is between the backward reference frame 704 and the forward reference frame 706. In some cases, a compound motion vector derivation approach can be used.

[0081] The motion field determined for the current frame 700 using the motion vectors of the backward reference frame 704 and the forward reference frame 706 has the same size as the current frame 700. The motion field for the current frame 700 can be determined separately at each of the encoder and the decoder to reduce the bitstream size that would otherwise be used to signal the motion field. Once the motion field has been determined, the motion field motion vectors of the motion field can be stored for later use. For example, the motion field motion vectors can be stored in a memory buffer or cache.

[0082] Further details of techniques for temporally interpolated picture prediction using frame-level motion vectors are now described. Figure 8 FIG. 8 is a flowchart of an example of a technique 800 for temporally interpolated picture prediction using frame-level motion vectors during encoding. Figure 9 FIG. 9 is a flowchart of an example of a technique 900 for temporally interpolated picture prediction using frame-level motion vectors during decoding. For example, the technique 800 can be performed in whole or in part at a prediction stage (e.g., intra / inter prediction stage 402) of an encoder for encoding a video stream, while the technique 900 can be performed in whole or in part at a prediction stage (e.g., intra / inter prediction stage 508) of a decoder for decoding a bitstream.

[0083] The technique 800 and / or the technique 900 can be implemented as, for example, a software program that can be executed by a computing device such as the transmitting station 102 or the receiving station 106. For example, the software program can include machine-readable instructions that can be stored in a memory such as the memory 204 or the secondary storage 214 and, when executed by a processor such as the processor 202, can cause the computing device to perform the technique 800 and / or the technique 900. The technique 800 and / or the technique 900 can be implemented using specialized hardware or firmware. For example, a hardware component such as a hardware code processor can be configured to perform the technique 800 and / or the technique 900. As explained above, some computing devices can have multiple memories or processors and can use multiple processors, memories, or both to distribute the operations described in the technique 800 and / or the technique 900. To facilitate explanation, the technique 800 and / or the technique 900 are each depicted and described herein as a series of steps or operations. However, the steps or operations according to the present disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein can be used. Further, not all illustrated steps or operations can be required to implement a technique according to the disclosed subject matter.

[0084] Reference is first made to Figure 8, showing techniques 800 for temporally interpolated picture prediction using frame-level motion vectors during encoding. At 802, for a current frame to be encoded (e.g., from an input video stream), a reference frame is generated using a backward reference frame for the current frame and a forward reference frame for the current frame. For example, the generated reference frame can be a TIP reference frame. Generating the reference frame using the backward reference frame and the forward reference frame can include generating a motion field for the current frame using motion vectors of the backward reference frame and the forward reference frame, and using the motion field to obtain reference blocks from those reference frames that can be used (e.g., via interpolation) to generate the reference frame.

[0085] At 804, a determination is made that the generated reference frame is to be used as a display frame in place of the current frame. Determining that the generated reference frame is to be used during encoding of the current frame can include using encoder motion search data to determine whether there is a motion mismatch with respect to the generated reference frame and a ZEROMV motion vector determined for the generated reference frame.

[0086] At 806, based on the determination that the generated reference frame is to be used as a display frame, a prediction of the generated reference frame is generated using a frame-level motion vector. The frame-level motion vector may, for example, be a ZEROMV motion vector. The frame-level motion vector is applied using motion compensation to the entire generated reference frame to generate the prediction of the generated reference frame.

[0087] At 808, the prediction and the frame-level motion vector are encoded into a bitstream. For example, the frame-level motion vector can be signaled in conjunction with a frame header associated with the prediction.

[0088] Reference is next made to Figure 9 , showing techniques 900 for temporally interpolated picture prediction using frame-level motion vectors during decoding. At 902, for a current frame to be decoded (e.g., from a bitstream), a reference frame is generated using a backward reference frame for the current frame and a forward reference frame for the current frame. For example, the generated reference frame can be a TIP reference frame. Generating the reference frame using the backward reference frame and the forward reference frame can include generating a motion field for the current frame using motion vectors of the backward reference frame and the forward reference frame, and using the motion field to obtain reference blocks from those reference frames that can be used (e.g., via interpolation) to generate the reference frame.

[0089] At 904, a determination is made that the generated reference frame is to be used as a display frame in place of the current frame. Determining that the generated reference frame is to be used during decoding of the current frame can include determining whether there is a motion mismatch with respect to the generated reference frame based on signaling of a ZEROMV motion vector for the generated reference frame. For example, the ZEROMV motion vector can be a frame-level motion vector determined and encoded into the bitstream for use with the generated reference frame.

[0090] At 906, based on the determination that the generated reference frame is to be used as a display frame, a prediction of the generated reference frame is generated using a frame-level motion vector signaled within the bitstream from which the current frame is being decoded (e.g., a ZEROMV motion vector used to determine a motion mismatch). Generating the prediction includes performing motion compensation for the generated reference frame using the frame-level motion vector.

[0091] At 908, a frame reconstruction is produced based on the prediction. The frame reconstruction represents a final display frame to be used for the generated reference frame and thus in place of the current frame. In some implementations, producing the frame reconstruction can include interpolating the motion indicated by the frame-level motion vector using an interpolation filter. For example, where the motion vector points to a sub-pixel location, an interpolation filter can be used to determine how to interpolate the motion between two or more primary pixels. The particular interpolation filter to be used can be determined based on bitstream data signaled from the encoder search data (e.g., indicating an interpolation filter that produced the least distortion for the current frame in the form in which it was pre-processed).

[0092] At 910, the frame reconstruction is output within an output video stream. For example, the frame reconstruction can be output for display during playback of the output video stream at a computing device.

[0093] The above-described aspects of encoding and decoding illustrate some examples of encoding and decoding techniques. However, it should be understood that when those terms are used in the claims, encoding and decoding can mean compression, decompression, transformation, or any other processing or alteration of data.

[0094] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or as is clear from the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or as is clear from the context to be directed to a singular form. Moreover, use of the term “an implementation” or the terms “one implementation” throughout is not intended to mean the same implementation unless described as such.

[0095] Implementations of the transmitting station 102 and / or the receiving station 106 (and algorithms, methods, instructions, etc. stored thereon and / or executed thereby, including by the encoder 400 and the decoder 500 or another encoder or decoder as disclosed herein) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application- specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors or any other suitable circuit. In the claims, the term "processor" should be understood as encompassing any of the foregoing hardware, either alone or in combination with. The terms "signal" and "data" are used interchangeably. Further, parts of the transmitting station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0096] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 can be implemented using a general purpose computer or a general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms and / or instructions described herein. Additionally or alternatively, for example, a special purpose computer / processor can be utilized that can include other hardware for carrying out any of the methods, algorithms or instructions described herein.

[0097] The transmitting station 102 and the receiving station 106 can be implemented, for example, on computers in a video conferencing system. Alternatively, the transmitting station 102 can be implemented on a server and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communication device. In this example, the transmitting station 102 can encode content into an encoded video signal and transmit the encoded video signal to the communication device. In turn, the communication device can then decode the encoded video signal. Alternatively, the communication device can decode content locally stored on the communication device (e.g., content that was not transmitted by the transmitting station 102). Other suitable transmitting and receiving implementations are available. For example, the receiving station 106 can be a generally stationary personal computer rather than a portable communication device.

[0098] Further, all or a portion of the implementations of the present disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be, for example, any apparatus that can contain, store, communicate, or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or a semiconductor apparatus. Other suitable mediums are available.

[0099] The above-described implementations and other aspects have been described to provide an overall understanding of the disclosure. The above-described implementations and other aspects are not limiting, however, and the scope of the disclosure is defined by the appended claims, which should be given their broadest reasonable interpretation.

Claims

1. A method for prediction using frame-level motion vectors, the method comprising: For the current frame encoded into the bitstream, a reference frame is generated using the backward reference frame of the current frame and the forward reference frame of the current frame; Determine whether to use the generated reference frame as the display frame instead of the current frame; Based on the determination, a prediction of the generated reference frame is generated using the frame-level motion vectors that are signaled within the bitstream. Frame reconstruction is generated based on the prediction; as well as The reconstructed frames are output within the output video stream.

2. The method as described in claim 1, wherein, Generating the reference frame using the backward reference frame and the forward reference frame includes: The motion vectors of the backward reference frame and the forward reference frame are used to generate a motion field for the current frame; The motion field is used to obtain reference blocks from the backward reference frame and the forward reference frame; and The reference block is used to generate the reference frame.

3. The method as described in claim 2, wherein, Using the reference block to generate the reference frame includes: Interpolate the reference block.

4. The method of claim 1, wherein, Determining whether to use the generated reference frame as the display frame instead of the current frame includes: The presence of motion mismatch with respect to the generated reference frame is determined based on the signaling of the ZEROMV motion vectors used for the generated reference frame.

5. The method of claim 4, wherein, The ZEROMV motion vector is the frame-level motion vector.

6. The method according to any one of claims 1 to 5, wherein, The prediction for the generated reference frame is generated using the frame-level motion vectors signaled within the bitstream. Additional residual code processing is performed on the generated reference frame to prepare the content of the generated reference for display in the output video stream.

7. The method of claim 6, wherein, The additional residual code processing includes motion compensation performed on the generated reference frame using the frame-level motion vectors.

8. The method according to any one of claims 1 to 5, wherein, Generating the frame reconstruction based on the prediction includes: Interpolation filters are used to interpolate the motion indicated by the frame-level motion vectors.

9. The method of claim 8, wherein, The interpolation filter is based on data that is signaled within the bitstream according to the encoder search.

10. A non-transitory computer-readable medium storing an encoded bit stream, wherein, The encoded bitstream is configured for decoding by using operations involving the prediction of frame-level motion vectors signaled within the encoded bitstream, the operations including: Determine that the generated reference frame associated with the current frame encoded into the encoded bitstream should be used as the display frame instead of the current frame; Based on the determination, the frame-level motion vectors are used to generate predictions for the generated reference frames; and The reconstructed frames based on the prediction are output within the output video stream.

11. The non-transitory computer-readable medium of claim 10, wherein the operation comprises: The generated reference frame is generated using the motion vectors of the backward reference frame and the forward reference frame of the current frame.

12. The non-transitory computer-readable medium of claim 10, wherein, Determining whether to use the generated reference frame as the display frame instead of the current frame includes: It is determined that the frame-level motion vector is signaled within the encoded bitstream for use in the generated reference frame.

13. The non-transitory computer-readable medium as claimed in any one of claims 10 to 12, wherein, Using the frame-level motion vectors to generate the prediction for the generated reference frame includes: The frame-level motion vectors are used to perform motion compensation for the generated reference frame.

14. The non-transitory computer-readable medium of any one of claims 10 to 12, wherein the operation comprises: The frame reconstruction is generated based on the prediction; as well as Interpolation filters are used to prepare the frame reconstruction for final display frame output by interpolating the motion indicated by the frame-level motion vectors.

15. An apparatus for prediction using frame-level motion vectors, the apparatus comprising: Memory; as well as A processor, configured to execute instructions stored in the memory to: Generate a reference frame for the current frame to be decoded; Determine whether to replace the current frame with the generated reference frame in the output video stream; Based on the determination, frame-level motion vectors are used to generate predictions for the generated reference frames; as well as The reconstructed frames generated based on the prediction are output as display frames within the output video stream.

16. The device as claimed in claim 15, wherein, The generated reference frame is generated using the backward reference frame and the forward reference frame of the current frame, wherein the backward reference frame, the current frame and the forward reference frame are signaled within the encoded bit stream, and the backward reference frame, the current frame and the forward reference frame are decoded from the encoded bit stream.

17. The device as claimed in claim 16, wherein, The determination to replace the current frame with the generated reference frame within the output video stream is based on signaling of the frame-level motion vectors within the encoded bitstream.

18. The device as claimed in any one of claims 15 to 17, wherein, The prediction is generated based on additional residual code processing performed on the generated reference frame.

19. The device as claimed in claim 18, wherein, The additional residual code processing includes motion compensation performed using the frame-level motion vectors.

20. The device as claimed in any one of claims 15 to 17, wherein, The frame reconstruction is generated using an interpolation filter.

Images